Welcome!

Microservices Expo Authors: Pat Romanski, Elizabeth White, Liz McMillan, Christoph Schell, Matt Brickey

Related Topics: Microservices Expo, Java IoT, Microsoft Cloud, @CloudExpo

Microservices Expo: Article

How to Test Application Throughput: Keep It Real

Rolling out a new release or architectural change relies heavily upon a web application handling a certain TPS

Often we see the workload to a web application measured by throughput. It's a way of quantifying the volume of requests/responses in relation to time. Transactions per second or TPS is the most common ratio used. A performance test plan usually contains certain throughput goals. The "GO or NO GO" decision for rolling out a new release or architectural change relies heavily upon a web application handling a certain TPS. Management wants a "Pass" stamp, but it's your job to make sure that the achieved TPS is indeed realistic - not an illusion of phony numbers. My advice is to "keep it real" by generating workloads that represent all the true characteristics of production. Faking it leads to false positive test result: a certain TPS was met and the Pass stamp was awarded, but the conditions were unrealistic. For example, you could achieve a 460 TPS result by hitting mostly lightweight transactions, or by running a low load of virtual users with little or no think time. In each of these cases, the throughput would be "High" but the workload does not represent what's really happening in production. Not even remotely. What's worse, if you "Pass" a performance test using these unrealistic methods, you have no idea if the deployment is going to withstand the production workload. If the applications falls over....guess who is on the hot seat? This could be an unintentional outcome, so be sure your tests are setup properly and create a realistic workload.

How do I accomplish that? It's actually the Design of the test which is responsible for determining how realistic the throughput is that you generate. There are several key factors here to take into consideration. They are all equally important but the underlying philosophy is (again) "keeping it real". Using a load tool to simulate virtual users executing scripts is a load test, but you also need to emulate accurate activity, conditions, behaviors, usage, etc. Each script executed in a load test contains simple requests and more complex business transactions. When setting up a test of an application, not all transactions are created equal. Throughput is affected by the "weight" of a transaction. For example, lightweight Transaction A can be as simple as serving up a static image. In another example, heavyweight Transaction B can be as complicated as executing a business transaction which involves algorithms to be run on the results of a DB query. The response time of Transaction A is going to be much quicker and will use less resources within the deployment since it is just a webserver response. Conversely, we can expect that Transaction B is going to have a much longer response time and use more resources including the database. When the load tool is executing a script, it waits for the response of a transaction before executing the next transaction. You can see how the response times of transactions affect throughput. The faster the response time of transactions, the higher the throughput. You can easily manipulate a test so you must create conditions which truly mimic expected production activity.

To accurately mimic expected production activity you must first ensure each virtual user represents each real user. If you expect a concurrent load of 2500 users actively using your web application, then you need to have a test which ramps to 2500 virtual users. This is extremely important because every virtual user has a unique footprint on the backend servers in sessions, memory usage, open sockets, etc. Trying to get away with a higher throughput test without an accurate number of virtual users will lead to inexact resource usage on the backend. Only a test which uses the true numbers of users will emulate the right load conditions.

There are typically different "types" of users per web application: shoppers, buyers, admin, etc. When setting up the test, create a population, transaction mixture, which represents the workload during peak usage in production. For example, 50% shoppers, 40% buyers, 10% admin. The most accurate transaction mixes are determined by log reviews or by the business analysis of expected usage in production. The scripts executed by the tool's load generators need to represent "true" user profiles. The scripts need to follow transaction flows: navigations, decisions, inputs, calculations, etc. These transactions need to use dynamic data to be realistic. Dynamic transaction flows include choices of different products, requests of links, submits of forms, or even more complicated: extracting response data to be used in subsequent requests. It is the dynamic scripts that emulate the diverse activity of real users.

To make those robotic Virtual Users act like human beings, pauses/delays need to be incorporated into the scripts. We all need to think, take in information, process it, make decisions, type out forms, etc. These "breathers" contribute to the accurate load characteristics. During these pauses, the servers are still performing housekeeping: closing of ports, garbage collection, gaging timeout, sweeping sessions, etc. all of which takes resources. If you hit the deployment with simultaneous users and simply crank up the throughput, the load characteristics are not realistic.

With today's Rich Internet Applications, there is the requirement to incorporate complex behaviors into the scripts. Representation of asynchronous updates of data being pushed from servers to browsers and vice versa, independent of full page refreshes. The tool needs to "listen" for updates and recreate a script which emulates the activity. This new PUSH challenge is a hurdle in performance testing but also critical in creating the right load characteristics. These rich behaviors affect throughput by usage polling, streaming, and other reactive mechanisms which must be accounted for.

Another factor to consider is the fact that users are connecting to web and mobile applications via all different network speeds. The connection speed affects the download rate. The slower the download, the higher the response time. For your populations of users, choose the bandwidth or bandwidths which really represent the end user connections. For example, you may be testing a LAN application with a known bandwidth restriction. Or there are a group of users connect from another country. A certain percentage may only access a web application via a mobile network.

Taking all these factors into consideration (user profiles, accurate number of users, transaction mixes, dynamic data, think times, bandwidth simulation, behaviors, etc) sets the stage for creating realistic throughput. Review your scripts and make sure your tests really represent expected production. Once you have designed realistic tests, you can execute and properly evaluate whether or not the web application can achieve the set throughput goals.

More Stories By Rebecca Clinard

Rebecca Clinard is a Senior Performance Engineer at Neotys, a provider of load testing software for Web applications. Previously, she worked as a web application performance engineer for Bowstreet, Fidelity Investments, Bottomline Technologies and Timberland companies, industries spanning retail, financial services, insurance and manufacturing. Her expertise lies in creating realistic load tests and performance tuning multi-tier deployments. She has been orchestrating and conducting performance tests since 2001. Clinard graduated from University of New Hampshire with a BS and also holds a UNIX Certificate from Worcester Polytechnic Institute.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@MicroservicesExpo Stories
"DivvyCloud as a company set out to help customers automate solutions to the most common cloud problems," noted Jeremy Snyder, VP of Business Development at DivvyCloud, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
There is a huge demand for responsive, real-time mobile and web experiences, but current architectural patterns do not easily accommodate applications that respond to events in real time. Common solutions using message queues or HTTP long-polling quickly lead to resiliency, scalability and development velocity challenges. In his session at 21st Cloud Expo, Ryland Degnan, a Senior Software Engineer on the Netflix Edge Platform team, will discuss how by leveraging a reactive stream-based protocol,...
DevOps at Cloud Expo, taking place October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 21st Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to w...
From personal care products to groceries and movies on demand, cloud-based subscriptions are fulfilling the needs of consumers across an array of market sectors. Nowhere is this shift to subscription services more evident than in the technology sector. By adopting an Everything-as-a-Service (XaaS) delivery model, companies are able to tailor their computing environments to shape the experiences they want for customers as well as their workforce.
"Outscale was founded in 2010, is based in France, is a strategic partner to Dassault Systémes and has done quite a bit of work with divisions of Dassault," explained Jackie Funk, Digital Marketing exec at Outscale, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
If you read a lot of business and technology publications, you might think public clouds are universally preferred over all other cloud options. To be sure, the numbers posted by Amazon Web Services (AWS) and Microsoft’s Azure platform are nothing short of impressive. Statistics reveal that public clouds are growing faster than private clouds and analysts at IDC predict that public cloud growth will be 3 times that of private clouds by 2019.
"We focus on SAP workloads because they are among the most powerful but somewhat challenging workloads out there to take into public cloud," explained Swen Conrad, CEO of Ocean9, Inc., in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"I think DevOps is now a rambunctious teenager – it’s starting to get a mind of its own, wanting to get its own things but it still needs some adult supervision," explained Thomas Hooker, VP of marketing at CollabNet, in this SYS-CON.tv interview at DevOps Summit at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
For over a decade, Application Programming Interface or APIs have been used to exchange data between multiple platforms. From social media to news and media sites, most websites depend on APIs to provide a dynamic and real-time digital experience. APIs have made its way into almost every device and service available today and it continues to spur innovations in every field of technology. There are multiple programming languages used to build and run applications in the online world. And just li...
If you are thinking about moving applications off a mainframe and over to open systems and the cloud, consider these guidelines to prioritize what to move and what to eliminate. On the surface, mainframe architecture seems relatively simple: A centrally located computer processes data through an input/output subsystem and stores its computations in memory. At the other end of the mainframe are printers and terminals that communicate with the mainframe through protocols. For all of its appare...
Your homes and cars can be automated and self-serviced. Why can't your storage? From simply asking questions to analyze and troubleshoot your infrastructure, to provisioning storage with snapshots, recovery and replication, your wildest sci-fi dream has come true. In his session at @DevOpsSummit at 20th Cloud Expo, Dan Florea, Director of Product Management at Tintri, provided a ChatOps demo where you can talk to your storage and manage it from anywhere, through Slack and similar services with...
"Peak 10 is a hybrid infrastructure provider across the nation. We are in the thick of things when it comes to hybrid IT," explained Michael Fuhrman, Chief Technology Officer at Peak 10, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
Data reduction delivers compelling cost reduction that substantially improves the business case in every cloud deployment model. No matter which cloud approach you choose, the cost savings benefits from data reduction should not be ignored and must be a component of your cloud strategy. IT professionals are finding that the future of IT infrastructure lies in the cloud. Data reduction technologies enable clouds — public, private, and hybrid — to deliver business agility and elasticity at the lo...
"At the keynote this morning we spoke about the value proposition of Nutanix, of having a DevOps culture and a mindset, and the business outcomes of achieving agility and scale, which everybody here is trying to accomplish," noted Mark Lavi, DevOps Solution Architect at Nutanix, in this SYS-CON.tv interview at @DevOpsSummit at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"As we've gone out into the public cloud we've seen that over time we may have lost a few things - we've lost control, we've given up cost to a certain extent, and then security, flexibility," explained Steve Conner, VP of Sales at Cloudistics,in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"I will be talking about ChatOps and ChatOps as a way to solve some problems in the DevOps space," explained Himanshu Chhetri, CTO of Addteq, in this SYS-CON.tv interview at @DevOpsSummit at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
DevOps is often described as a combination of technology and culture. Without both, DevOps isn't complete. However, applying the culture to outdated technology is a recipe for disaster; as response times grow and connections between teams are delayed by technology, the culture will die. A Nutanix Enterprise Cloud has many benefits that provide the needed base for a true DevOps paradigm. In their Day 3 Keynote at 20th Cloud Expo, Chris Brown, a Solutions Marketing Manager at Nutanix, and Mark Lav...
"NetApp's vision is how we help organizations manage data - delivering the right data in the right place, in the right time, to the people who need it, and doing it agnostic to what the platform is," explained Josh Atwell, Developer Advocate for NetApp, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
Five years ago development was seen as a dead-end career, now it’s anything but – with an explosion in mobile and IoT initiatives increasing the demand for skilled engineers. But apart from having a ready supply of great coders, what constitutes true ‘DevOps Royalty’? It’ll be the ability to craft resilient architectures, supportability, security everywhere across the software lifecycle. In his keynote at @DevOpsSummit at 20th Cloud Expo, Jeffrey Scheaffer, GM and SVP, Continuous Delivery Busine...
"We do one of the best file systems in the world. We learned how to deal with Big Data many years ago and we implemented this knowledge into our software," explained Jakub Ratajczak, Business Development Manager at MooseFS, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.