Welcome!

Microservices Expo Authors: Zakia Bouachraoui, Pat Romanski, Elizabeth White, Liz McMillan, Yeshim Deniz

Related Topics: @DevOpsSummit, Linux Containers, Containers Expo Blog

@DevOpsSummit: Blog Post

Performance Testing Guide | @DevOpsSummit #CloudNative #Serverless #DevOps #Monitoring

Testing types, performance testing steps, best practices, and more

The Ultimate Guide to Performance Testing and Software Testing: Testing Types, Performance Testing Steps, Best Practices, and More
By Angela Stringfellow

Performance testing is a form of software testing that focuses on how a system running the system performs under a particular load. This is not about finding software bugs or defects. Performance testing measures according to benchmarks and standards. Performance testing should give developers the diagnostic information they need to eliminate bottlenecks.

Types of performance testing for software
To understand how software will perform on users' systems, there different types of performance tests that can be applied during software testing. This is non-functional testing, which is designed to determine the readiness of a system. (Functional testing focuses on individual functions of software.)

Testing Types

Image credit MindsMapped

Load testing
Load testing
measures system performance as the workload increases. That workload could mean concurrent users or transactions.The system is monitored to measure response time and system staying power as workload increases. That workload falls within the parameters of normal working conditions.

Stress testing
Unlike load testing, stress testing - also known as fatigue testing - is meant to measure system performance outside of the parameters of normal working conditions. The software is given more users or transactions that can be handled. The goal of stress testing is to measure the software stability. At what point does software fail, and how does the software recover from failure?

Spike testing
Spike testing is a type of stress testing that evaluates software performance when workloads are substantially increased quickly and repeatedly. The workload is beyond normal expectations for short amounts of time.

Endurance testing
Endurance testing - also known as soak testing - is an evaluation of how software performs with a normal workload over an extended amount of time. The goal of endurance testing is to check for system problems such as memory leaks. (A memory leak occurs when a system fails to release discarded memory. The memory leak can impair system performance or cause it to fail.)

Scalability testing
Scalability testing is used to determine if software is effectively handling increasing workloads. This can be determined by gradually adding to the user load or data volume while monitoring system performance. Also, the workload may stay at the same level while resources such as CPUs and memory are changed.

Volume testing
Volume testing determines how efficiently software performs with a large, projected amounts of data. It is also known as flood testing because the test floods the system with data.

Most Common Problems Observed in Performance Testing
During performance testing of software, developers are looking for performance symptoms and issues. Speed issues - slow responses and long load times for example - often are observed and addressed. But there are other performance problems that can be observed:

  • Bottlenecking - This occurs when data flow is interrupted or halted because there is not enough capacity to handle the workload.
  • Poor scalability - If software cannot handle the desired number of concurrent tasks, results could be delayed, errors could increase, or other unexpected behavior could happen that affects:
    • Disk usage
    • CPU usage
    • Memory leaks
    • Operating system limitations
    • Poor network configuration
  • Software configuration issues - Often settings are not set at a sufficient level to handle the workload.
  • Insufficient hardware resources - Performance testing may reveal physical memory constraints or low-performing CPUs.

Seven Performance Testing Steps

Performance Testing Process

Image credit Gateway TestLabs

Also known as the test bed, a testing environment is where software, hardware, and networks are set up to execute performance tests. To use a testing environment for performance testing, developers can use these seven steps:

1. Identify the testing environment.
Identify the hardware, software, network configurations and tools available allows the testing team design the test and identify performance testing challenges early. Performance testing environment options include:

  • Subset of production system with fewer servers of lower specification
  • Subset of production system with fewer servers of the same specification
  • Replica of productions system
  • Actual production system

2. Identify performance metrics.
In addition to identifying metrics such as response time, throughput and constraints, identify what are the success criteria for performance testing.

3. Plan and design performance tests.
Identify performance test scenarios that take into account user variability, test data, and target metrics. This will create one or two models.

4. Configure the test environment.
Prepare the elements of the test environment and instruments needed to monitor resources.

5. Implement your test design.
Develop the tests.

6. Execute tests.
In addition to running the performance tests, monitor and capture the data generated.

7. Analyze, report, retest.
Analyze the data and share the findings. Run the performance tests again using the same parameters and different parameters.

What Performance Testing Metrics are Measured
Metrics are needed to understand the quality and effectiveness
of performance testing. Improvements cannot be made unless there are measurements. There are two definitions that need to be explained:

  • Measurements - The data being collected such as the seconds it takes to respond to a request.
  • Metrics - A calculation that uses measurements to define the quality of results such as average response time (total response time/requests).

There are many ways to measure speed, scalability, and stability but each round of performance testing cannot be expected to use all of them. Among the metrics used in performance testing, the following often are used:

Response time
Total time to send a request and get a response.

Wait time
Also known as average latency, this tells developers how long it takes to receive the first byte after a request is sent.

Average load time
The average amount of time it takes to deliver every request is a major indicator of quality from a user's perspective.

Peak response time
This is the measurement of the longest amount of time it takes to fulfill a request. A peak response time that is significantly longer than average may indicate an anomaly that will create problems.

Error rate
This calculation is a percentage of requests resulting in errors compared to all requests. These errors usually occur when the load exceeds capacity.

Concurrent users
This the most common measure of load - how many active users at any point. Also known as load size.

Requests per second
How many requests are handled.

Transactions passed/failed
A measurement of the total numbers of successful or unsuccessful requests.

Throughput
Measured by kilobytes per second, throughput shows the amount of bandwidth used during the test.

CPU utilization
How much time the CPU needs to process requests.

Memory utilization
How much memory is needed to process the request.

Performance Testing Best Practices
Perhaps the most important tip for performance testing is testing early, test often. A single test will not tell developers all they need to know. Successful performance testing is a collection of repeated and smaller tests:

  • Test as early as possible in development. Do not wait and rush performance testing as the project winds down.
  • Performance testing isn't just for completed projects. There is value in testing individual units or modules.
  • Conduct multiple performance tests to ensure consistent findings and determine metrics averages.
  • Applications often involve multiple systems such as databases, servers, and services. Test the individual units separately as well as together.

Testing Lifecycle

Image credit Varun Kapaganty

In addition to repeated testing, performance testing will be more successful by following a series of performance testing best practices:

  • Involve developers, IT and testers in creating a performance testing environment.
  • Remember real people will be using the software that is undergoing performance testing. Determine how the results will affect users not just test environment servers.
  • Go beyond performance test parameters. Develop a model by planning a test environment that takes into account as much user activity as possible.
  • Baseline measurements provide a starting point for determining success or failure.
  • Performance tests are best conducted in test environments that are as close to the production systems as possible.
  • Isolate the performance test environment from the environment used for quality assurance testing.
  • No performance testing tool will do everything needed. And limited resources may restrict choice even further. Research performance testing tools for the right fit.
  • Keep the test environment as consistent as possible.
  • Calculating averages will deliver actionable metrics. There is value in tracking outliers also. Those extreme measurements could reveal possible failures.
  • Consider the audience when preparing reports that share performance testing findings. Also, include any system and software changes in reports.

Five common performance testing mistakes
There are also some mistakes that can lead to less-than-reliable results when performance testing:

  1. Not enough time for testing.
  2. Not involving developers.
  3. Not using QA system similar to production system.
  4. Not sufficiently tuning software.
  5. Not having a troubleshooting plan.

Performance Testing Fallacies
Performance testing fallacies can lead to mistakes or failure to follow performance testing best practices. According to Sofia Palamarchuk, these beliefs can cost significant money and resources when developing software:

Performance testing is the last step in development.
As mentioned in the section on performance testing best practices, anticipating and solving performance issues should be an early part of software development. Implementing solutions early will less costly than major fixes at the end of software development.

More hardware can fix performance issues.
Adding processors, servers or memory simply adds to the cost without solving any problems. More efficient software will run better and avoid potential problems that can occur even when hardware is increased or upgraded.

The testing environment is close enough.
Conducting performance testing in a test environment that is similar to the production environment is a performance testing best practice for a reason. The differences between the elements can significantly affect system performance. It may not be possible to conduct performance testing in the exact production environment, but try to match:

Software Development Lifecycle

  • Hardware components
  • Operating system and settings
  • Other applications used on the system
  • Databases

What works now, works across the board.
Be careful about extrapolating results. Don't take the small set of performance testing results and assume that they will be the same when elements change. Also, it works in the opposite direction. Do not infer minimum performance and requirements based upon load testing. All assumptions should be verified through performance testing.

One performance testing scenario is enough.
Not every performance problem can be detected in one performance testing scenario. But resources do limit the amount of testing that can happen. In the middle are a series of performance tests that target the riskiest situations and have the greatest impact on performance. Also, problems can arise outside of well-planned and well-designed performance testing. Monitoring the production environment also can detect performance issues.

Testing each part equals testing the whole system.
While it is important to isolate functions for performance testing, the individual component test results do not add up to a system-wide assessment. But it may not be feasible to test all the functionalities of a system. A complete-as-possible performance test must be designed using the resources available. But be aware of what has not been tested.

What works for them, works for us.
If a given set of users does experience complications or performance issues, do not consider that a performance test for all users. Use performance testing to make sure the platform and configurations work as expected.

Software developers are too experienced to need performance testing.
Lack of experience is not the only reason behind performance issues. Mistakes are made - even by developers who have created issue-free software in the past. Many more variables come into play - especially when multiple concurrent users are in the system.

A full load test tells everything.
It's tempting to just run a test at the total load to find all the performance issues. Except for that kind of test tends to reveal so many performance issues that it's hard to focus on individual solutions. Starting at a lower load and scaling up incrementally may seem like an unnecessarily slow process, but it produces easier results that are more efficient to troubleshoot.

Test scripts are actual users.
Make sure the test automations are using the software in ways that real users would. This is especially important when performance test parameters are changed.

Additional Resources on Performance Testing

The post The Ultimate Guide to Performance Testing and Software Testing: Testing Types, Performance Testing Steps, Best Practices, and More appeared first on Stackify.


DXWorldEXPO LLC, the producer of the world's most influential technology conferences and trade shows has announced the conference tracks for CloudEXPO | DXWorldEXPO 2018 New York.

DXWordEXPO New York 2018, colocated with CloudEXPO New York 2018 will be held November 11-13, 2018, in New York City.

Digital Transformation (DX) is a major focus with the introduction of DXWorldEXPO within the program. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term.

A total of 88% of Fortune 500 companies from a generation ago are now out of business. Only 12% still survive. Similar percentages are found throughout enterprises of all sizes.

Register for Full Conference "Gold Pass" ▸ Here (Expo Hall ▸ Here)

Sponsorship Opportunities Here

Speaking Opportunities Here

Sponsorship and Speaking Inquiries: [email protected].

2018 Conference Agenda, Keynotes and 10 Conference Tracks

DXWordEXPO New York 2018 and Cloud Expo New York 2018 agenda present 222 rockstar faculty members, 200 sessions and 22 keynotes and general sessions in 10 distinct conference tracks.

  • Cloud-Native | Serverless
  • DevOpsSummit
  • FinTechEXPO - New York Blockchain Event
  • CloudEXPO - Enterprise Cloud
  • DXWorldEXPO - Digital Transformation (DX)
  • Smart Cities | IoT | IIoT
  • AI | Machine Learning | Cognitive Computing
  • BigData | Analytics
  • The API Enterprise | Mobility | Security
  • Hot Topics | FinTech | WebRTC

Register for Full Conference "Gold Pass" ▸ Here (Expo Hall ▸ Here)

DXWorldEXPO | CloudEXPO 2018 New York cover all of these tools, with the most comprehensive program and with 222 rockstar speakers throughout our industry presenting 22 Keynotes and General Sessions, 200 Breakout Sessions along 10 Tracks, as well as our signature Power Panels. Our Expo Floor brings together the world's leading companies throughout the world of Cloud Computing, DevOps, FinTech, Digital Transformation, and all they entail.

As your enterprise creates a vision and strategy that enables you to create your own unique, long-term success, learning about all the technologies involved is essential. Companies today not only form multi-cloud and hybrid cloud architectures, but create them with built-in cognitive capabilities.

Cloud-Native thinking is now the norm in financial services, manufacturing, telco, healthcare, transportation, energy, media, entertainment, retail and other consumer industries, as well as the public sector.

CloudEXPO is the world's most influential technology event where Cloud Computing was coined over a decade ago and where technology buyers and vendors meet to experience and discuss the big picture of Digital Transformation and all of the strategies, tactics, and tools they need to realize their goals.

FinTech Is Now Part of the DXWorldEXPO | CloudEXPO Program!

Financial enterprises in New York City, London, Singapore, and other world financial capitals are embracing a new generation of smart, automated FinTech that eliminates many cumbersome, slow, and expensive intermediate processes from their businesses.

Accordingly, attendees at the upcoming 22nd CloudEXPO | DXWorldEXPO November 11-13, 2018 in New York City will find fresh new content in two new tracks called:

  • FinTechEXPO
  • New York Blockchain Event

which will incorporate FinTech and Blockchain, as well as machine learning, artificial intelligence and deep learning in these two distinct tracks.

Register for Full Conference "Gold Pass" ▸ Here (Expo Hall ▸ Here)

Sponsorship Opportunities Here

Speaking Opportunities Here

Sponsorship and Speaking Inquiries: [email protected].

FinTech brings efficiency as well as the ability to deliver new services and a much improved customer experience throughout the global financial services industry. FinTech is a natural fit with cloud computing, as new services are quickly developed, deployed, and scaled on public, private, and hybrid clouds.

More than US$20 billion in venture capital is being invested in FinTech this year. DXWorldEXPOCloudEXPO are pleased to bring you the latest FinTech developments as an integral part of our program.

DXWorldEXPO | CloudEXPO are accepting speaking submissions for this new track, so please visit Cloud Computing Expo for the latest information or contact us at [email protected]

Register for Full Conference "Gold Pass" ▸ Here (Expo Hall ▸ Here)

Sponsorship Opportunities Here

Speaking Opportunities Here

Sponsorship and Speaking Inquiries: [email protected].

Download Slide Deck ▸ Here

Only DXWorldEXPO | CloudEXPO bring together all this in a single location:

Attend DXWorldEXPO | CloudEXPO. Build your own custom experience. Learn about the world's latest technologies and chart your course to Digital Transformation.

22nd International DXWorldEXPO | CloudEXPO, taking place November 11-13, 2018, in New York City, will feature technical sessions from a rock star conference faculty and the leading industry players in the world.

Register for Full Conference "Gold Pass" ▸ Here (Expo Hall ▸ Here)

Sponsorship Opportunities Here

Speaking Opportunities Here

Sponsorship and Speaking Inquiries: [email protected].

Download Slide Deck: ▸ Here

Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud strategy. Meanwhile, 94% of enterprises are using some form of XaaS - software, platform, and infrastructure as a service.

With major technology companies and startups seriously embracing Cloud strategies, now is the perfect time to attend and learn what is going on, contribute to the discussions, and ensure that your enterprise is on the right path to Digital Transformation.

Every Global 2000 enterprise in the world is now integrating cloud computing in some form into its IT development and operations. Midsize and small businesses are also migrating to the cloud in increasing numbers.

Register for Full Conference "Gold Pass" ▸ Here (Expo Hall ▸ Here)

Sponsorship Opportunities Here

Speaking Opportunities Here

Sponsorship and Speaking Inquiries: [email protected].

Download Slide Deck: ▸ Here

Companies are each developing their unique mix of cloud technologies and services, forming multi-cloud and hybrid cloud architectures and deployments across all major industries. Cloud-driven thinking has become the norm in financial services, manufacturing, telco, healthcare, transportation, energy, media, entertainment, retail and other consumer industries, and the public sector.

Sponsorship Opportunities

DXWorldEXPO | CloudEXPO are the single show where technology buyers and vendors can meet to experience and discus cloud computing and all that it entails. Sponsors of DXWorldEXPO | CloudEXPO will benefit from unmatched branding, profile building and lead generation opportunities through:

  • Featured on-site presentation and ongoing on-demand webcast exposure to a captive audience of industry decision-makers.
  • Showcase exhibition during our new extended dedicated expo hours
  • Breakout Session Priority scheduling for Sponsors that have been guaranteed a 35-minute technical session
  • Online advertising on 4,5 million article pages in SYS-CON's i-Technology Publications
  • Capitalize on our Comprehensive Marketing efforts leading up to the show with print mailings, e-newsletters and extensive online media coverage.
  • Unprecedented PR Coverage: Unmatched editorial coverage on Cloud Computing Journal.
  • Tweetup to over 100,000 plus Twitter followers
  • Press releases sent on major wire services to over 500 industry analysts.

Secrets of Our Most Popular Sponsors and Exhibitors ▸ Here

For more information on sponsorship, exhibit, and keynote opportunities, contact [email protected].

Sponsorship Opportunities Here

Download Slide Deck:Here

Speaking Opportunities

The upcoming 22nd International DXWorldEXPO | CloudEXPO November 11-13, 2018 in New York City, NY announces that its Call For Papers for speaking opportunities is now open.

Secrets of Our Most Popular Faculty Members ▸ Here

Submit your speaking proposal Here or by email [email protected].

Download Slide Deck: ▸ Here

About DXWorldEXPO LLC

DXWorldEXPO LLC is a Lighthouse Point, Florida-based trade show company and the creator of DXWorldEXPODigital Transformation Conference & Expo. The company produces and presents CloudEXPO, DevOpsSummitFinTechEXPO Blockchain Event, the world's most influential conferences and trade shows.

Read the original blog entry...

More Stories By Stackify Blog

Stackify offers the only developers-friendly solution that fully integrates error and log management with application performance monitoring and management. Allowing you to easily isolate issues, identify what needs to be fixed quicker and focus your efforts – Support less, Code more. Stackify provides software developers, operations and support managers with an innovative cloud based solution that gives them DevOps insight and allows them to monitor, detect and resolve application issues before they affect the business to ensure a better end user experience. Start your free trial now stackify.com

Microservices Articles
When building large, cloud-based applications that operate at a high scale, it’s important to maintain a high availability and resilience to failures. In order to do that, you must be tolerant of failures, even in light of failures in other areas of your application. “Fly two mistakes high” is an old adage in the radio control airplane hobby. It means, fly high enough so that if you make a mistake, you can continue flying with room to still make mistakes. In his session at 18th Cloud Expo, Lee A...
Lori MacVittie is a subject matter expert on emerging technology responsible for outbound evangelism across F5's entire product suite. MacVittie has extensive development and technical architecture experience in both high-tech and enterprise organizations, in addition to network and systems administration expertise. Prior to joining F5, MacVittie was an award-winning technology editor at Network Computing Magazine where she evaluated and tested application-focused technologies including app secu...
In his general session at 19th Cloud Expo, Manish Dixit, VP of Product and Engineering at Dice, discussed how Dice leverages data insights and tools to help both tech professionals and recruiters better understand how skills relate to each other and which skills are in high demand using interactive visualizations and salary indicator tools to maximize earning potential. Manish Dixit is VP of Product and Engineering at Dice. As the leader of the Product, Engineering and Data Sciences team at D...
Containers and Kubernetes allow for code portability across on-premise VMs, bare metal, or multiple cloud provider environments. Yet, despite this portability promise, developers may include configuration and application definitions that constrain or even eliminate application portability. In this session we'll describe best practices for "configuration as code" in a Kubernetes environment. We will demonstrate how a properly constructed containerized app can be deployed to both Amazon and Azure ...
Modern software design has fundamentally changed how we manage applications, causing many to turn to containers as the new virtual machine for resource management. As container adoption grows beyond stateless applications to stateful workloads, the need for persistent storage is foundational - something customers routinely cite as a top pain point. In his session at @DevOpsSummit at 21st Cloud Expo, Bill Borsari, Head of Systems Engineering at Datera, explored how organizations can reap the bene...
Using new techniques of information modeling, indexing, and processing, new cloud-based systems can support cloud-based workloads previously not possible for high-throughput insurance, banking, and case-based applications. In his session at 18th Cloud Expo, John Newton, CTO, Founder and Chairman of Alfresco, described how to scale cloud-based content management repositories to store, manage, and retrieve billions of documents and related information with fast and linear scalability. He addresse...
The now mainstream platform changes stemming from the first Internet boom brought many changes but didn’t really change the basic relationship between servers and the applications running on them. In fact, that was sort of the point. In his session at 18th Cloud Expo, Gordon Haff, senior cloud strategy marketing and evangelism manager at Red Hat, will discuss how today’s workloads require a new model and a new platform for development and execution. The platform must handle a wide range of rec...
SYS-CON Events announced today that DatacenterDynamics has been named “Media Sponsor” of SYS-CON's 18th International Cloud Expo, which will take place on June 7–9, 2016, at the Javits Center in New York City, NY. DatacenterDynamics is a brand of DCD Group, a global B2B media and publishing company that develops products to help senior professionals in the world's most ICT dependent organizations make risk-based infrastructure and capacity decisions.
Discussions of cloud computing have evolved in recent years from a focus on specific types of cloud, to a world of hybrid cloud, and to a world dominated by the APIs that make today's multi-cloud environments and hybrid clouds possible. In this Power Panel at 17th Cloud Expo, moderated by Conference Chair Roger Strukhoff, panelists addressed the importance of customers being able to use the specific technologies they need, through environments and ecosystems that expose their APIs to make true ...
In his keynote at 19th Cloud Expo, Sheng Liang, co-founder and CEO of Rancher Labs, discussed the technological advances and new business opportunities created by the rapid adoption of containers. With the success of Amazon Web Services (AWS) and various open source technologies used to build private clouds, cloud computing has become an essential component of IT strategy. However, users continue to face challenges in implementing clouds, as older technologies evolve and newer ones like Docker c...