Welcome!

Microservices Expo Authors: Liz McMillan, Pat Romanski, Elizabeth White, Yeshim Deniz, Zakia Bouachraoui

Related Topics: @DevOpsSummit, Microservices Expo, Linux Containers, Containers Expo Blog

@DevOpsSummit: Blog Feed Post

Digital Performance Management | @DevOpsSummit #DevOps #Microservices

The easiest way to run a known transaction end to end from the user device to back-end services

Meet the Three Musketeers of Digital Performance Management: Real, Synthetic and Virtual Users
By James Urquhart

Alexandre Dumas’s classic portrayal of a young man seeking to join the elite guard of his day seems an unlikely source of inspiration for a blog post about digital performance management, but there’s something about groups of three — each balancing the others’ strengths and weaknesses — that makes a great team.

And, while most operators today might view digital systems monitoring in terms of two players — synthetic and real users — there is a third member of the team that turns performance monitoring into performance management: the virtual user.

What does each user agent represent?

To understand how real, synthetic, and virtual user agents complement each other, it’s critical to understand what each represents. Sorting out why one might need both synthetic users and virtual users, for example — or even what the difference between the two might be — can be quite confusing.

Here’s a useful guide to the three user types.

Real users
Monitoring the actual experiences of users running in all of the different network, platform, device and geographical environments that your site or application will experience is absolutely essential to creating a near optimal user experience each and every time.

Real users allow you to measure what the end user actually experiences through their front end stack (browser or app, OS, etc), but at the cost of being able to control the conditions in which the user is running those stacks. Being able to capture anomalies quickly, especially for things like key transactions or usage under exceptional loads requires the ability to collect data with more known initial conditions or baselines.

Synthetic users
The easiest way to run a known transaction end to end from the user device to backend services with known conditions is to use an artificial “user” to drive that transaction. Synthetic user monitoring simplifies asking questions like “is the backend behaving the way it should at this moment for this transaction” or “are users on the east coast of the US experiencing worse performance than users in Europe?”

Unfortunately, because these systems don’t scale well, and these synthetic users can themselves affect performance if they generate too much load, these users don’t scale well at all. So, how do you create a baseline of expected behavior on the backend systems that have to scale to meet customer demand when stressed?

Virtual users
Virtual users aren’t meant to be used as day-to-day real time monitoring of current conditions, but are intended more to give a solid baseline measurement of backend performance under varying amounts of load.
Those baselines can be regularly verified in production, and operators can be warned if virtual user performance slips out of expected ranges.

Virtual users can also be run at incredible scales during scheduled load and stress tests. Their initial conditions can be closely controlled, though the combination of scale and control of conditions comes at the cost of driving load primarily against the backend, without driving actual browsers or mobile applications.

How virtual users complement real and synthetic users
So if those are the three musketeers, and we have a pretty good idea of how synthetic and real user monitoring complement each other let’s explore what virtual users bring to digital performance management.

As our CEO, Tom Lounibos, is wont to say, cloud testing introduced the concept of load testing actual production systems, instead of focusing only on “staging” environments before release to production. This is a powerful concept for today’s rapidly evolving digital applications, but it’s only possible if the load testing system is capable of dynamically adjusting load from an extremely light baseline to a massive peak load without rebuilding the load tests.

The way SOASTA achieves this is by utilizing CloudTest to drive virtual users through the system at a load that can be adjusted throughout a test run as needed. So you can choose to either run “baseline” tests as separate tests from time to time, or continuously run a load test that can be throttled up and down as needed over time.

Low levels of testing give you a baseline of behavior for these test users, which can be compared to behavior when the throttle is turned up. (Low level tests are also great for finding code changes that negatively affect performance or break the load test scripts before a critical load test finds them first.)

This comes at the cost of not using actual browser and device profiles, of course. You won’t know from a virtual user if the experience is better in Chrome or IE, for instance, from a virtual user in a load test. However, you will be able to confirm that your backend servers can handle any anticipated (and all but the most unimaginable unanticipated) loads when they come.

How real and synthetic users complement virtual users
Equally important to the information that virtual users brings to understanding digital performance is the power of utilizing real and synthetic user measurements when designing virtual user load tests. Beyond the simple comparisons of all three types of data — which alone can help greatly with troubleshooting performance and availability problems — there is the power of using real and synthetic data to shape the tests you run.

Example 1: Real user monitoring data will tell you a lot about the situation that cost you the most in terms of whatever your conversion metric might be

Sometimes the situations that cost you the most aren’t obvious. We’ve seen situation in which specific product pages had serious conversion issues, though not all such pages did. Real user data can make that clear, and then load tests can be created to test that condition, at least until you’ve confirmed the problem no longer exists. (Even then you can run the test occasionally as a form of regression testing.)

Example 2: Using the virtual user baselines noted above as additional baselines for real and synthetic user measurements

Does your backend perform differently under varying loads? When is a real or synthetic measurement truly an anomaly vs expected (though not necessarily “acceptable”) behavior? Having a strong, controlled profile of your system performance gives you greater confidence in tagging events as unexpected versus expected.

Takeaway
The most complete protection against performance-related failure needs a three-user team with:

  • synthetic users giving you a sense of user experience in well regulated “experiments” across browsers and devices,
  • real user monitoring telling you about the actual experiences discovered by actual users, and
  • virtual users enabling you to build a feedback loop to test, measure, and verify performance variability and resiliency.

With this combination, you have the makings of a complete picture of where, how, and why performance affects user outcomes.

And understanding user outcomes enables understanding both business and technical outcomes.

Related reading:

More Stories By SOASTA Blog

The SOASTA platform enables digital business owners to gain unprecedented and continuous performance insights into their real user experience on mobile and web devices in real time and at scale.

Microservices Articles
Lori MacVittie is a subject matter expert on emerging technology responsible for outbound evangelism across F5's entire product suite. MacVittie has extensive development and technical architecture experience in both high-tech and enterprise organizations, in addition to network and systems administration expertise. Prior to joining F5, MacVittie was an award-winning technology editor at Network Computing Magazine where she evaluated and tested application-focused technologies including app secu...
Using new techniques of information modeling, indexing, and processing, new cloud-based systems can support cloud-based workloads previously not possible for high-throughput insurance, banking, and case-based applications. In his session at 18th Cloud Expo, John Newton, CTO, Founder and Chairman of Alfresco, described how to scale cloud-based content management repositories to store, manage, and retrieve billions of documents and related information with fast and linear scalability. He addresse...
Adding public cloud resources to an existing application can be a daunting process. The tools that you currently use to manage the software and hardware outside the cloud aren’t always the best tools to efficiently grow into the cloud. All of the major configuration management tools have cloud orchestration plugins that can be leveraged, but there are also cloud-native tools that can dramatically improve the efficiency of managing your application lifecycle. In his session at 18th Cloud Expo, ...
The now mainstream platform changes stemming from the first Internet boom brought many changes but didn’t really change the basic relationship between servers and the applications running on them. In fact, that was sort of the point. In his session at 18th Cloud Expo, Gordon Haff, senior cloud strategy marketing and evangelism manager at Red Hat, will discuss how today’s workloads require a new model and a new platform for development and execution. The platform must handle a wide range of rec...
Containers and Kubernetes allow for code portability across on-premise VMs, bare metal, or multiple cloud provider environments. Yet, despite this portability promise, developers may include configuration and application definitions that constrain or even eliminate application portability. In this session we'll describe best practices for "configuration as code" in a Kubernetes environment. We will demonstrate how a properly constructed containerized app can be deployed to both Amazon and Azure ...
SYS-CON Events announced today that DatacenterDynamics has been named “Media Sponsor” of SYS-CON's 18th International Cloud Expo, which will take place on June 7–9, 2016, at the Javits Center in New York City, NY. DatacenterDynamics is a brand of DCD Group, a global B2B media and publishing company that develops products to help senior professionals in the world's most ICT dependent organizations make risk-based infrastructure and capacity decisions.
Discussions of cloud computing have evolved in recent years from a focus on specific types of cloud, to a world of hybrid cloud, and to a world dominated by the APIs that make today's multi-cloud environments and hybrid clouds possible. In this Power Panel at 17th Cloud Expo, moderated by Conference Chair Roger Strukhoff, panelists addressed the importance of customers being able to use the specific technologies they need, through environments and ecosystems that expose their APIs to make true ...
In his keynote at 19th Cloud Expo, Sheng Liang, co-founder and CEO of Rancher Labs, discussed the technological advances and new business opportunities created by the rapid adoption of containers. With the success of Amazon Web Services (AWS) and various open source technologies used to build private clouds, cloud computing has become an essential component of IT strategy. However, users continue to face challenges in implementing clouds, as older technologies evolve and newer ones like Docker c...
CloudEXPO New York 2018, colocated with DXWorldEXPO New York 2018 will be held November 11-13, 2018, in New York City and will bring together Cloud Computing, FinTech and Blockchain, Digital Transformation, Big Data, Internet of Things, DevOps, AI, Machine Learning and WebRTC to one location.
DevOpsSummit New York 2018, colocated with CloudEXPO | DXWorldEXPO New York 2018 will be held November 11-13, 2018, in New York City. Digital Transformation (DX) is a major focus with the introduction of DXWorldEXPO within the program. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term.