Welcome!

Microservices Expo Authors: Elizabeth White, Charles Araujo, Liz McMillan, Pat Romanski, Flint Brenton

Related Topics: Microservices Expo, Java IoT, Microsoft Cloud, Machine Learning , Agile Computing, @CloudExpo

Microservices Expo: Article

Can You See the Storm Coming?

APM solutions enables us to set up alerts against good performance baselines

As much as we try to avoid performance problems, they do happen. It is inevitable. But it is possible to learn to react fast, and in some occasions fast enough that the impact on the end users is negligible. Despite operators' best efforts, 73% of performance issues are reported by users, according to "APM: Getting IT on the C-Level's agenda" report by Aberdeen Group. This number is quite large considering that less than 5% of all users typically bother to complain at all. User Experience has a significant impact on business success. According to the Aberdeen report, poor performance of applications can reduce revenue by 9% and productivity by 64%.

The goal of application performance monitoring is to ensure and improve the quality of applications as perceived by the end users. Getting to the root of the problem quickly is only part of the solution. When we ask various Operation teams how they learn about performance problems they sometimes reply: "Our users tell us." As we already pointed it out we should not wait for the disaster to happen, but rather take appropriate actions as soon as we see the storm coming.

In this article we recount two incidents that happened to our client, ZinMines, a steel and mining company from Zinariya (names changed for commercial reasons). In both cases the Operations team at ZinMines got notified about the problem well in advance of any user reaction. The team members were able to start analyzing and improving the situation by the time users eventually notified them about the problem. If they have waited for users to notify them, the problem would have been solved much later and users would have been much more frustrated.

Case #1: The Maintenance Page
When the Operations team first set up its application performance monitoring solution, the members made sure that the alerts on potential performance problems were set up correctly.

One morning, just before 8 a.m., an alert that monitored total transaction time went off. The Operations team used the APM solution to chart the total time as seen by the end user broken down by time spent on the server and time spent on the network. They saw that significant time was spent on the server (see Figure 1). This could make the whole application slower.

The team started to analyze the problem together with the engineering team. They learned that there was a serious bug and the engineering team would need a few hours to fix it.

Meanwhile, shortly after 9 a.m., they got a call from a user that the services run by ZinMines were particularly slow. The helpdesk informed the users that the problem was already being investigated and a team had been appointed to look into the issue.

Since the delays in processing user requests kept coming in, and in order to avoid further frustration among the end users, the decision was made to enter into maintenance mode. Around 9:30 the Operations team started redirecting part of the traffic to the maintenance page. This took some load off the application, decreased total time and gave the engineering team time to handle the issue. The chart in Figure 1 shows the change in server time and redirect time after the redirect to the maintenance page was enabled, which is indicated on the time line with the blue arrow.

Figure 1 also shows that the traffic was already pretty high for at least one hour before one of the end users notified the Operations team about the problem. The red arrow on the time line with the red arrow in Figure 1 indicates when the problem was first reported.

Figure 1: Reduced server time and total time after activating redirect to the maintenance page

Had the Operations team waited for an end user to report the problem they would not have contacted the engineering team early enough to gain extra time to start resolving the problem. Thanks to properly configured alerts they were able to act in time and shorten the time the users were impacted by poor application performance.

Case #2: 4xx Errors
Sometime later, the Operations team got another alert. They consulted the APM solution and discovered that one of the application servers was generating a lot of 404 errors.

Figure 2 shows the 4xx errors charted around the time of the incident report. The green arrow indicates when the alert was raised.

Figure 2: 4xx errors were happening for almost an hour before the incident was reported by the end user and fixed.

The Operations team performed root cause analysis and discovered that the problem was caused by some caching issues and they decided to restart the server. The blue arrow on the time line in Figure 2 shows when the server was restarted.

Shortly before they initiated the restart procedures they got an incident report submitted by one of the end users from the finance department (see Figure 3).

Figure 3: Incident reported an hour after the performance problems had started.

They could close the issue almost immediately since when they checked the report with HTTP 4xx errors (see Figure 2) the situation was back to normal again. This was again seen as a very positive element by the customer. The team was not only aware of the issues that were troubling the users, before any user would complain, but could also see if the action taken to resolve the issue actually improved the situation.

The red arrow in Figure 2 shows when the report was submitted by the end user. The problem started more than an hour before the user reported the incident, similar to the previous incident where users waited more than an hour. If the Operations team waited for their users they would have lost at least an hour resolving the issue.

Conclusion
Application performance monitoring is more than just following the fault domain isolation workflow to determine the root cause of the problem that is reported by an end user. In many cases waiting for the end users to report problems simply takes too long, while the frustration due to poor performance grows.

APM solutions, such as Compuware dynaTrace Data Center Real User Monitoring (DCRUM), enables us to set up alerts, e.g., against good performance baselines. In many cases these alerts will be triggered long before the end users report an incident so it gives the Operations team more time to react before issues get serious. As shown in the Aberdeen report mentioned above 95% of users will not even bother to report the problem at all.

Second it also enables the team to see if the action taken to rectify the problem actually improves the end-users' experience.

More Stories By Sebastian Kruk

Sebastian Kruk is a Technical Product Strategist, Center of Excellence, at Compuware APM Business Unit.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@MicroservicesExpo Stories
"This all sounds great. But it's just not realistic." This is what a group of five senior IT executives told me during a workshop I held not long ago. We were working through an exercise on the organizational characteristics necessary to successfully execute a digital transformation, and the group was doing their ‘readout.' The executives loved everything we discussed and agreed that if such an environment existed, it would make transformation much easier. They just didn't believe it was reali...
Your homes and cars can be automated and self-serviced. Why can't your storage? From simply asking questions to analyze and troubleshoot your infrastructure, to provisioning storage with snapshots, recovery and replication, your wildest sci-fi dream has come true. In his session at @DevOpsSummit at 20th Cloud Expo, Dan Florea, Director of Product Management at Tintri, provided a ChatOps demo where you can talk to your storage and manage it from anywhere, through Slack and similar services with...
Containers are rapidly finding their way into enterprise data centers, but change is difficult. How do enterprises transform their architecture with technologies like containers without losing the reliable components of their current solutions? In his session at @DevOpsSummit at 21st Cloud Expo, Tony Campbell, Director, Educational Services at CoreOS, will explore the challenges organizations are facing today as they move to containers and go over how Kubernetes applications can deploy with lega...
The “Digital Era” is forcing us to engage with new methods to build, operate and maintain applications. This transformation also implies an evolution to more and more intelligent applications to better engage with the customers, while creating significant market differentiators. In both cases, the cloud has become a key enabler to embrace this digital revolution. So, moving to the cloud is no longer the question; the new questions are HOW and WHEN. To make this equation even more complex, most ...
Learn how to solve the problem of keeping files in sync between multiple Docker containers. In his session at 16th Cloud Expo, Aaron Brongersma, Senior Infrastructure Engineer at Modulus, discussed using rsync, GlusterFS, EBS and Bit Torrent Sync. He broke down the tools that are needed to help create a seamless user experience. In the end, can we have an environment where we can easily move Docker containers, servers, and volumes without impacting our applications? He shared his results so yo...
Don’t go chasing waterfall … development, that is. According to a recent post by Madison Moore on Medium featuring insights from several software delivery industry leaders, waterfall is – while still popular – not the best way to win in the marketplace. With methodologies like Agile, DevOps and Continuous Delivery becoming ever more prominent over the past 15 years or so, waterfall is old news. Or, is it? Moore cites a recent study by Gartner: “According to Gartner’s IT Key Metrics Data report, ...
Enterprise architects are increasingly adopting multi-cloud strategies as they seek to utilize existing data center assets, leverage the advantages of cloud computing and avoid cloud vendor lock-in. This requires a globally aware traffic management strategy that can monitor infrastructure health across data centers and end-user experience globally, while responding to control changes and system specification at the speed of today’s DevOps teams. In his session at 20th Cloud Expo, Josh Gray, Chie...
Kubernetes is a new and revolutionary open-sourced system for managing containers across multiple hosts in a cluster. Ansible is a simple IT automation tool for just about any requirement for reproducible environments. In his session at @DevOpsSummit at 18th Cloud Expo, Patrick Galbraith, a principal engineer at HPE, discussed how to build a fully functional Kubernetes cluster on a number of virtual machines or bare-metal hosts. Also included will be a brief demonstration of running a Galera MyS...
Many organizations are now looking to DevOps maturity models to gauge their DevOps adoption and compare their maturity to their peers. However, as enterprise organizations rush to adopt DevOps, moving past experimentation to embrace it at scale, they are in danger of falling into the trap that they have fallen into time and time again. Unfortunately, we've seen this movie before, and we know how it ends: badly.
Agile has finally jumped the technology shark, expanding outside the software world. Enterprises are now increasingly adopting Agile practices across their organizations in order to successfully navigate the disruptive waters that threaten to drown them. In our quest for establishing change as a core competency in our organizations, this business-centric notion of Agile is an essential component of Agile Digital Transformation. In the years since the publication of the Agile Manifesto, the conn...
"I focus on what we are calling CAST Highlight, which is our SaaS application portfolio analysis tool. It is an extremely lightweight tool that can integrate with pretty much any build process right now," explained Andrew Siegmund, Application Migration Specialist for CAST, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
In IT, we sometimes coin terms for things before we know exactly what they are and how they’ll be used. The resulting terms may capture a common set of aspirations and goals – as “cloud” did broadly for on-demand, self-service, and flexible computing. But such a term can also lump together diverse and even competing practices, technologies, and priorities to the point where important distinctions are glossed over and lost.
"I will be talking about ChatOps and ChatOps as a way to solve some problems in the DevOps space," explained Himanshu Chhetri, CTO of Addteq, in this SYS-CON.tv interview at @DevOpsSummit at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
For organizations that have amassed large sums of software complexity, taking a microservices approach is the first step toward DevOps and continuous improvement / development. Integrating system-level analysis with microservices makes it easier to change and add functionality to applications at any time without the increase of risk. Before you start big transformation projects or a cloud migration, make sure these changes won’t take down your entire organization.
The Jevons Paradox suggests that when technological advances increase efficiency of a resource, it results in an overall increase in consumption. Writing on the increased use of coal as a result of technological improvements, 19th-century economist William Stanley Jevons found that these improvements led to the development of new ways to utilize coal. In his session at 19th Cloud Expo, Mark Thiele, Chief Strategy Officer for Apcera, compared the Jevons Paradox to modern-day enterprise IT, examin...
The taxi industry never saw Uber coming. Startups are a threat to incumbents like never before, and a major enabler for startups is that they are instantly “cloud ready.” If innovation moves at the pace of IT, then your company is in trouble. Why? Because your data center will not keep up with frenetic pace AWS, Microsoft and Google are rolling out new capabilities. In his session at 20th Cloud Expo, Don Browning, VP of Cloud Architecture at Turner, posited that disruption is inevitable for comp...
When you focus on a journey from up-close, you look at your own technical and cultural history and how you changed it for the benefit of the customer. This was our starting point: too many integration issues, 13 SWP days and very long cycles. It was evident that in this fast-paced industry we could no longer afford this reality. We needed something that would take us beyond reducing the development lifecycles, CI and Agile methodologies. We made a fundamental difference, even changed our culture...
In his session at 20th Cloud Expo, Mike Johnston, an infrastructure engineer at Supergiant.io, discussed how to use Kubernetes to set up a SaaS infrastructure for your business. Mike Johnston is an infrastructure engineer at Supergiant.io with over 12 years of experience designing, deploying, and maintaining server and workstation infrastructure at all scales. He has experience with brick and mortar data centers as well as cloud providers like Digital Ocean, Amazon Web Services, and Rackspace. H...
High-velocity engineering teams are applying not only continuous delivery processes, but also lessons in experimentation from established leaders like Amazon, Netflix, and Facebook. These companies have made experimentation a foundation for their release processes, allowing them to try out major feature releases and redesigns within smaller groups before making them broadly available. In his session at 21st Cloud Expo, Brian Lucas, Senior Staff Engineer at Optimizely, discussed how by using ne...
Without a clear strategy for cost control and an architecture designed with cloud services in mind, costs and operational performance can quickly get out of control. To avoid multiple architectural redesigns requires extensive thought and planning. Boundary (now part of BMC) launched a new public-facing multi-tenant high resolution monitoring service on Amazon AWS two years ago, facing challenges and learning best practices in the early days of the new service.