Welcome!

Microservices Expo Authors: Pat Romanski, Elizabeth White, Liz McMillan, Yeshim Deniz, Zakia Bouachraoui

Related Topics: @DevOpsSummit, Microservices Expo, Linux Containers, Containers Expo Blog, @CloudExpo

@DevOpsSummit: Blog Post

What Does an SLA Mean to You? | @DevOpsSummit #DevOps #SLA #ContinuousDelivery

You can factor performance and load testing into your continuous integration process by focusing on performance SLAs

When it comes to testing an application, many project managers and test leads do not fit performance and load testing early on in the development life cycle. Often, performance and load testing are conducted once the application has been completed and after all functional testing is done. In fact, it's frequently a last step - almost an afterthought - before the app is ready to go into production.

The problem with this approach is the classic problem with late-stage testing. When issues are found by the testers, developers have to go back and open up long-finalized code to fix them. There's risk in causing other parts of the application to break. Addressing problems after-the-fact becomes time consuming and expensive. Furthermore, any delay in releasing a new feature or a new app can directly impact revenue.

Even though you are running an agile development process, that doesn't necessarily mean that your performance testing is being conducted in a truly agile way. Saving performance testing for a "final sprint" before release still treats it like a waterfall development step, with all the cost and risk that comes with that. In this post, we will show you how to make load testing happen early and often by putting SLAs on the agile task board.

What Does an SLA Mean to You?
You can factor performance and load testing into your continuous integration process by focusing on performance SLAs, or service level agreements. Traditionally, SLAs are contracts put in place between two parties, such as a service firm and its client. This agreement sets various attributes about how the service will be delivered. For example, a large company that outsources its help desk operations to another company may require that all incoming emails be responded to within one hour. That agreement allows the client to build its business knowing that the desired service will be executed according to its specifications.

As a performance tester, you can think of an SLA in terms of your performance parameters. In your case, it's an implicit contract that you make with your user. It's almost like a standard that you hold your application to on behalf of your users. Your SLA may state that all pages need to load within 2 seconds, or that the first page of results of a search should be displayed within 4 seconds.

Though this isn't a hard-and-fast contract, that's how you want to think about it. Specify your SLA items during the application design process, and you can turn them into requirements that are placed on the task board as part of agile. That way, developers are taking performance into account as they develop the application, and not leaving it as an afterthought.

What Your SLA Should Include
There are all sorts of website metrics you can include in an SLA. The most important thing is to get something specified up front. This is how you are going to get developers coding for performance at the very beginning of the process.

When you are first starting, we recommend that you have enough specified in your performance SLAs to support automated smoke testing. Consider what happens in a smoke test. Once the application is built, it will be deployed and a series of tests will be run to make sure that everything works. This basic test might be as simple as setting up a user and having the user log in. Adding a performance SLA to that smoke test could be as simple as requiring that the login process be completed within a given time window.

Add that requirement to your task board, and suddenly you are automating performance testingright at the beginning of the development process. As more of the app gets developed, you can add more requirements alongside the modules and features being created.

When you create your SLAs, it's a good idea to identify a set of key pages to test regularly (home page, shopping cart, contact form, search results, chat window, etc.), and define requirements for a set of metrics such as:

  • DNS resolution time
  • Time-to-display
  • Time-to-last-byte
  • System uptime

Obviously your application will have its own critical performance metrics as well.

Finally, define a base level of network types (Wi-Fi, 3G) and platforms (operating systems and browsers) that your SLA will be measured from. Requiring zippy load times for someone running IE6 over a dial-up internet connection will only prompt hatred and ire from your developers, but providing realistic expectations that apply to the bulk of your users will keep everyone happy.

How Often to Conduct Load and Performance Testing
You now know how set expectations with SLAs in place, but how often should performance and load tests to be conducted? Remember, testing is happening throughout the agile development process, and at different scales. For many agile shops, your automated build process includes a number of tests that get executed every time a build happens. Then you may have additional testing that is done on a more thorough basis, but less frequently. Sometimes you have test scripts that are run every time code is checked in.

Match your automated performance testing to the process. It's unreasonable to expect a full load test be conducted every time the app is smoke tested, but something simple will match the scope, intent, and turn-around time of the smoke test, fitting in nicely with what's already happening. Use your best judgment and align your SLAs with the work being done, and you should be ok.

The SLA: One Secret to Quick and Efficient Load Testing
As your application grows and your process matures, you'll end up with a rich library of SLA requirements throughout your portfolio of test structures. You'll also have automated tests that can validate those SLAs within the right testing constructs: smoke testing, comprehensive automated tests, unit tests, and other areas. It won't take too many cycles for performance to become a regular expectation of the code that gets produced.

Photo Credit: Enrique Fernández

More Stories By Tim Hinds

Tim Hinds is the Product Marketing Manager for NeoLoad at Neotys. He has a background in Agile software development, Scrum, Kanban, Continuous Integration, Continuous Delivery, and Continuous Testing practices.

Previously, Tim was Product Marketing Manager at AccuRev, a company acquired by Micro Focus, where he worked with software configuration management, issue tracking, Agile project management, continuous integration, workflow automation, and distributed version control systems.

Microservices Articles
Lori MacVittie is a subject matter expert on emerging technology responsible for outbound evangelism across F5's entire product suite. MacVittie has extensive development and technical architecture experience in both high-tech and enterprise organizations, in addition to network and systems administration expertise. Prior to joining F5, MacVittie was an award-winning technology editor at Network Computing Magazine where she evaluated and tested application-focused technologies including app secu...
Using new techniques of information modeling, indexing, and processing, new cloud-based systems can support cloud-based workloads previously not possible for high-throughput insurance, banking, and case-based applications. In his session at 18th Cloud Expo, John Newton, CTO, Founder and Chairman of Alfresco, described how to scale cloud-based content management repositories to store, manage, and retrieve billions of documents and related information with fast and linear scalability. He addresse...
Adding public cloud resources to an existing application can be a daunting process. The tools that you currently use to manage the software and hardware outside the cloud aren’t always the best tools to efficiently grow into the cloud. All of the major configuration management tools have cloud orchestration plugins that can be leveraged, but there are also cloud-native tools that can dramatically improve the efficiency of managing your application lifecycle. In his session at 18th Cloud Expo, ...
The now mainstream platform changes stemming from the first Internet boom brought many changes but didn’t really change the basic relationship between servers and the applications running on them. In fact, that was sort of the point. In his session at 18th Cloud Expo, Gordon Haff, senior cloud strategy marketing and evangelism manager at Red Hat, will discuss how today’s workloads require a new model and a new platform for development and execution. The platform must handle a wide range of rec...
Containers and Kubernetes allow for code portability across on-premise VMs, bare metal, or multiple cloud provider environments. Yet, despite this portability promise, developers may include configuration and application definitions that constrain or even eliminate application portability. In this session we'll describe best practices for "configuration as code" in a Kubernetes environment. We will demonstrate how a properly constructed containerized app can be deployed to both Amazon and Azure ...
SYS-CON Events announced today that DatacenterDynamics has been named “Media Sponsor” of SYS-CON's 18th International Cloud Expo, which will take place on June 7–9, 2016, at the Javits Center in New York City, NY. DatacenterDynamics is a brand of DCD Group, a global B2B media and publishing company that develops products to help senior professionals in the world's most ICT dependent organizations make risk-based infrastructure and capacity decisions.
Discussions of cloud computing have evolved in recent years from a focus on specific types of cloud, to a world of hybrid cloud, and to a world dominated by the APIs that make today's multi-cloud environments and hybrid clouds possible. In this Power Panel at 17th Cloud Expo, moderated by Conference Chair Roger Strukhoff, panelists addressed the importance of customers being able to use the specific technologies they need, through environments and ecosystems that expose their APIs to make true ...
In his keynote at 19th Cloud Expo, Sheng Liang, co-founder and CEO of Rancher Labs, discussed the technological advances and new business opportunities created by the rapid adoption of containers. With the success of Amazon Web Services (AWS) and various open source technologies used to build private clouds, cloud computing has become an essential component of IT strategy. However, users continue to face challenges in implementing clouds, as older technologies evolve and newer ones like Docker c...
CloudEXPO New York 2018, colocated with DXWorldEXPO New York 2018 will be held November 11-13, 2018, in New York City and will bring together Cloud Computing, FinTech and Blockchain, Digital Transformation, Big Data, Internet of Things, DevOps, AI, Machine Learning and WebRTC to one location.
DevOpsSummit New York 2018, colocated with CloudEXPO | DXWorldEXPO New York 2018 will be held November 11-13, 2018, in New York City. Digital Transformation (DX) is a major focus with the introduction of DXWorldEXPO within the program. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term.