Welcome!

Microservices Expo Authors: Zakia Bouachraoui, Pat Romanski, Elizabeth White, Liz McMillan, Yeshim Deniz

Related Topics: @DevOpsSummit, Microservices Expo, Linux Containers, Containers Expo Blog, @CloudExpo

@DevOpsSummit: Blog Post

What Does an SLA Mean to You? | @DevOpsSummit #DevOps #SLA #ContinuousDelivery

You can factor performance and load testing into your continuous integration process by focusing on performance SLAs

When it comes to testing an application, many project managers and test leads do not fit performance and load testing early on in the development life cycle. Often, performance and load testing are conducted once the application has been completed and after all functional testing is done. In fact, it's frequently a last step - almost an afterthought - before the app is ready to go into production.

The problem with this approach is the classic problem with late-stage testing. When issues are found by the testers, developers have to go back and open up long-finalized code to fix them. There's risk in causing other parts of the application to break. Addressing problems after-the-fact becomes time consuming and expensive. Furthermore, any delay in releasing a new feature or a new app can directly impact revenue.

Even though you are running an agile development process, that doesn't necessarily mean that your performance testing is being conducted in a truly agile way. Saving performance testing for a "final sprint" before release still treats it like a waterfall development step, with all the cost and risk that comes with that. In this post, we will show you how to make load testing happen early and often by putting SLAs on the agile task board.

What Does an SLA Mean to You?
You can factor performance and load testing into your continuous integration process by focusing on performance SLAs, or service level agreements. Traditionally, SLAs are contracts put in place between two parties, such as a service firm and its client. This agreement sets various attributes about how the service will be delivered. For example, a large company that outsources its help desk operations to another company may require that all incoming emails be responded to within one hour. That agreement allows the client to build its business knowing that the desired service will be executed according to its specifications.

As a performance tester, you can think of an SLA in terms of your performance parameters. In your case, it's an implicit contract that you make with your user. It's almost like a standard that you hold your application to on behalf of your users. Your SLA may state that all pages need to load within 2 seconds, or that the first page of results of a search should be displayed within 4 seconds.

Though this isn't a hard-and-fast contract, that's how you want to think about it. Specify your SLA items during the application design process, and you can turn them into requirements that are placed on the task board as part of agile. That way, developers are taking performance into account as they develop the application, and not leaving it as an afterthought.

What Your SLA Should Include
There are all sorts of website metrics you can include in an SLA. The most important thing is to get something specified up front. This is how you are going to get developers coding for performance at the very beginning of the process.

When you are first starting, we recommend that you have enough specified in your performance SLAs to support automated smoke testing. Consider what happens in a smoke test. Once the application is built, it will be deployed and a series of tests will be run to make sure that everything works. This basic test might be as simple as setting up a user and having the user log in. Adding a performance SLA to that smoke test could be as simple as requiring that the login process be completed within a given time window.

Add that requirement to your task board, and suddenly you are automating performance testingright at the beginning of the development process. As more of the app gets developed, you can add more requirements alongside the modules and features being created.

When you create your SLAs, it's a good idea to identify a set of key pages to test regularly (home page, shopping cart, contact form, search results, chat window, etc.), and define requirements for a set of metrics such as:

  • DNS resolution time
  • Time-to-display
  • Time-to-last-byte
  • System uptime

Obviously your application will have its own critical performance metrics as well.

Finally, define a base level of network types (Wi-Fi, 3G) and platforms (operating systems and browsers) that your SLA will be measured from. Requiring zippy load times for someone running IE6 over a dial-up internet connection will only prompt hatred and ire from your developers, but providing realistic expectations that apply to the bulk of your users will keep everyone happy.

How Often to Conduct Load and Performance Testing
You now know how set expectations with SLAs in place, but how often should performance and load tests to be conducted? Remember, testing is happening throughout the agile development process, and at different scales. For many agile shops, your automated build process includes a number of tests that get executed every time a build happens. Then you may have additional testing that is done on a more thorough basis, but less frequently. Sometimes you have test scripts that are run every time code is checked in.

Match your automated performance testing to the process. It's unreasonable to expect a full load test be conducted every time the app is smoke tested, but something simple will match the scope, intent, and turn-around time of the smoke test, fitting in nicely with what's already happening. Use your best judgment and align your SLAs with the work being done, and you should be ok.

The SLA: One Secret to Quick and Efficient Load Testing
As your application grows and your process matures, you'll end up with a rich library of SLA requirements throughout your portfolio of test structures. You'll also have automated tests that can validate those SLAs within the right testing constructs: smoke testing, comprehensive automated tests, unit tests, and other areas. It won't take too many cycles for performance to become a regular expectation of the code that gets produced.

Photo Credit: Enrique Fernández

More Stories By Tim Hinds

Tim Hinds is the Product Marketing Manager for NeoLoad at Neotys. He has a background in Agile software development, Scrum, Kanban, Continuous Integration, Continuous Delivery, and Continuous Testing practices.

Previously, Tim was Product Marketing Manager at AccuRev, a company acquired by Micro Focus, where he worked with software configuration management, issue tracking, Agile project management, continuous integration, workflow automation, and distributed version control systems.

Microservices Articles
When building large, cloud-based applications that operate at a high scale, it’s important to maintain a high availability and resilience to failures. In order to do that, you must be tolerant of failures, even in light of failures in other areas of your application. “Fly two mistakes high” is an old adage in the radio control airplane hobby. It means, fly high enough so that if you make a mistake, you can continue flying with room to still make mistakes. In his session at 18th Cloud Expo, Lee A...
Lori MacVittie is a subject matter expert on emerging technology responsible for outbound evangelism across F5's entire product suite. MacVittie has extensive development and technical architecture experience in both high-tech and enterprise organizations, in addition to network and systems administration expertise. Prior to joining F5, MacVittie was an award-winning technology editor at Network Computing Magazine where she evaluated and tested application-focused technologies including app secu...
In his general session at 19th Cloud Expo, Manish Dixit, VP of Product and Engineering at Dice, discussed how Dice leverages data insights and tools to help both tech professionals and recruiters better understand how skills relate to each other and which skills are in high demand using interactive visualizations and salary indicator tools to maximize earning potential. Manish Dixit is VP of Product and Engineering at Dice. As the leader of the Product, Engineering and Data Sciences team at D...
Containers and Kubernetes allow for code portability across on-premise VMs, bare metal, or multiple cloud provider environments. Yet, despite this portability promise, developers may include configuration and application definitions that constrain or even eliminate application portability. In this session we'll describe best practices for "configuration as code" in a Kubernetes environment. We will demonstrate how a properly constructed containerized app can be deployed to both Amazon and Azure ...
Modern software design has fundamentally changed how we manage applications, causing many to turn to containers as the new virtual machine for resource management. As container adoption grows beyond stateless applications to stateful workloads, the need for persistent storage is foundational - something customers routinely cite as a top pain point. In his session at @DevOpsSummit at 21st Cloud Expo, Bill Borsari, Head of Systems Engineering at Datera, explored how organizations can reap the bene...
Using new techniques of information modeling, indexing, and processing, new cloud-based systems can support cloud-based workloads previously not possible for high-throughput insurance, banking, and case-based applications. In his session at 18th Cloud Expo, John Newton, CTO, Founder and Chairman of Alfresco, described how to scale cloud-based content management repositories to store, manage, and retrieve billions of documents and related information with fast and linear scalability. He addresse...
The now mainstream platform changes stemming from the first Internet boom brought many changes but didn’t really change the basic relationship between servers and the applications running on them. In fact, that was sort of the point. In his session at 18th Cloud Expo, Gordon Haff, senior cloud strategy marketing and evangelism manager at Red Hat, will discuss how today’s workloads require a new model and a new platform for development and execution. The platform must handle a wide range of rec...
SYS-CON Events announced today that DatacenterDynamics has been named “Media Sponsor” of SYS-CON's 18th International Cloud Expo, which will take place on June 7–9, 2016, at the Javits Center in New York City, NY. DatacenterDynamics is a brand of DCD Group, a global B2B media and publishing company that develops products to help senior professionals in the world's most ICT dependent organizations make risk-based infrastructure and capacity decisions.
Discussions of cloud computing have evolved in recent years from a focus on specific types of cloud, to a world of hybrid cloud, and to a world dominated by the APIs that make today's multi-cloud environments and hybrid clouds possible. In this Power Panel at 17th Cloud Expo, moderated by Conference Chair Roger Strukhoff, panelists addressed the importance of customers being able to use the specific technologies they need, through environments and ecosystems that expose their APIs to make true ...
In his keynote at 19th Cloud Expo, Sheng Liang, co-founder and CEO of Rancher Labs, discussed the technological advances and new business opportunities created by the rapid adoption of containers. With the success of Amazon Web Services (AWS) and various open source technologies used to build private clouds, cloud computing has become an essential component of IT strategy. However, users continue to face challenges in implementing clouds, as older technologies evolve and newer ones like Docker c...