Welcome!

Microservices Expo Authors: Stackify Blog, Elizabeth White, Liz McMillan, Pat Romanski, Jason Bloomberg

Related Topics: Microservices Expo, @CloudExpo

Microservices Expo: Article

Non-Functional Testing for Cloud

Testing for Cloud QoS

Functional & Non Functional Tests
As we know typical application testing will consist of the following types of test cycles:

Functional

  • Regression
  • Installation
  • Functional
  • Sanity Testing

Non Functional

  • Performance Test for Responsiveness of the application
  • Performance Test for a Given Load Profile (Load Test)
  • Performance Test for Abnormal Stress conditions (Stress Test)
  • Performance Test for System Availability

Functional Tests ensure that the system behaves as documented and non - functional tests ensure that the QoS (Quality Of Service) can be met.

Detailed  explanation of the above test types are available in various testing manuals and guides, however below we are elaborating some specific Test Scenarios that are more relevant to Cloud and how to plan for them.

Also there will not be any changes to the functional test planning and execution between a ‘On Premise' and  ‘Cloud Application', however the non functional test planning will differ and hence needs to be addressed.

Again for a SaaS application, the cloud provider takes care all the testing, but they also need to plan for the below  situations.

However, if you deploy your application on a PaaS or IaaS platform, the following scenarios are much valid  for you as a software development organization.

Additional Cloud Non Functional Test Types

1.  Cloud Burst Responsiveness Test : This will be a variation of the stress tests in the regular applications. However, elastic cloud applications will tend to have the following characteristics.

  • End-user access patterns contain a certain degree of randomness.
  • There are daily recurring cycles in people‘s behavior
  • Some variability is driven by industry dynamics.
  • Uncertain growth patterns

Cloud platform addresses these variations on the  usage patterns and dynamically scales the allocated resources.

However it is of utmost important such that the Cloud Test Cases / Test Planning and Test Execution includes the above mentioned scenarios.

So a Cloud Application needs to subject to unpredictable changes and fluctuations in demand and the same needs to be planned. This is also known as ‘Cloud Bursting',  which is the ability to load occasional surges without  significant capital  investments.

Some of the additional parameters to gather during the ‘Cloud Burst Responsiveness  Tests' are :

  • Business Transaction Size
  • Number of business users
  • Targeted Cloud Zone, for example most of the Cloud providers like Amazon EC2 have different regions and the speed can vary accordingly.
    • US- N Virginia
    • US - N California
    • EU - IreLand
    • APAC - Singapore
  • Instance size and other parameters like number of CPUs

2.  VM Migration & Availability  Test: Major efficiencies  of  Cloud Platform are possible through management of Virtual machines and migration of work load between the virtual machines.  Successful Virtual machine migrations ensure that the Availability of the Cloud applications are fully guaranteed.

The basic requirement when migrating virtual machines is the ability to fully save the running state of the virtual machine. This is far more involved than simply copying a virtual machine (configuration and image files) to another machine. The system needs to create a point in time snapshot of its memory, device I/O states, network connections and the contents of the virtual CPU registers.

While this is very difficult to simulate a Virtual Machine Migrations,   As a a consumer we can ask the Cloud Provider  to  help with simulating conditions for VM Migration and the behavior of the application can be looked into.

3. Multi Tenant Penetration Tests: A penetration test is a method of evaluating the security by simulating an attack by a malicious user. The intent of a penetration test is to determine feasibility of an attack and the amount of business impact of a successful exploit.

As indicated in my other article, attacks like SQL Injection, Cross Site Scripting are   possible to a greater extent in a multi  tenant application. (http://cloudcomputing.sys-con.com/node/1624391)

Test cases should be written to attack the system from the point of view of a  rogue user who possess the ID as one of the tenant of the application and that user's ability to penetrate the system and view the information of other tenants.

4.  Virtual IP Changes & Availability Test: The complex architecture of Cloud Infrastructure in itself  brings lot of challenges like  internal IP addresses, DNS names and mapping them appropriately.  For example if we consider Amazon EC2 , the following parameters affect the network connections.

  • All Amazon EC2 instances are allocated a private address by DHCP
  • Each instance is provided an internal DNS name that resolves to the private IP address of the instance from within Amazon EC2; it will not resolve outside of Amazon EC2.
  • Each instance is provided an external DNS name that resolves to the public IP address of the instance outside the Amazon EC2 network and the private IP address from within Amazon EC2 network.

So the issues around the IP addresses in a Virtualized environment poses  different testing needs .

Summary
Cloud Computing presents tangible benefits to businesses. However, along with these potential benefits comes a new set of concerns: security, privacy, availability, performance and integrity. Suitable testing must be at the core of any Cloud solution to ensure the delivery of a safe, integrated solution that meets the needs of the business it is to serve.

More Stories By Srinivasan Sundara Rajan

Highly passionate about utilizing Digital Technologies to enable next generation enterprise. Believes in enterprise transformation through the Natives (Cloud Native & Mobile Native).

@MicroservicesExpo Stories
Docker is sweeping across startups and enterprises alike, changing the way we build and ship applications. It's the most prominent and widely known software container platform, and it's particularly useful for eliminating common challenges when collaborating on code (like the "it works on my machine" phenomenon that most devs know all too well). With Docker, you can run and manage apps side-by-side - in isolated containers - resulting in better compute density. It's something that many developer...
The “Digital Era” is forcing us to engage with new methods to build, operate and maintain applications. This transformation also implies an evolution to more and more intelligent applications to better engage with the customers, while creating significant market differentiators. In both cases, the cloud has become a key enabler to embrace this digital revolution. So, moving to the cloud is no longer the question; the new questions are HOW and WHEN. To make this equation even more complex, most ...
Your homes and cars can be automated and self-serviced. Why can't your storage? From simply asking questions to analyze and troubleshoot your infrastructure, to provisioning storage with snapshots, recovery and replication, your wildest sci-fi dream has come true. In his session at @DevOpsSummit at 20th Cloud Expo, Dan Florea, Director of Product Management at Tintri, provided a ChatOps demo where you can talk to your storage and manage it from anywhere, through Slack and similar services with...
Don’t go chasing waterfall … development, that is. According to a recent post by Madison Moore on Medium featuring insights from several software delivery industry leaders, waterfall is – while still popular – not the best way to win in the marketplace. With methodologies like Agile, DevOps and Continuous Delivery becoming ever more prominent over the past 15 years or so, waterfall is old news. Or, is it? Moore cites a recent study by Gartner: “According to Gartner’s IT Key Metrics Data report, ...
What's the role of an IT self-service portal when you get to continuous delivery and Infrastructure as Code? This general session showed how to create the continuous delivery culture and eight accelerators for leading the change. Don Demcsak is a DevOps and Cloud Native Modernization Principal for Dell EMC based out of New Jersey. He is a former, long time, Microsoft Most Valuable Professional, specializing in building and architecting Application Delivery Pipelines for hybrid legacy, and cloud ...
Many organizations are now looking to DevOps maturity models to gauge their DevOps adoption and compare their maturity to their peers. However, as enterprise organizations rush to adopt DevOps, moving past experimentation to embrace it at scale, they are in danger of falling into the trap that they have fallen into time and time again. Unfortunately, we've seen this movie before, and we know how it ends: badly.
"I focus on what we are calling CAST Highlight, which is our SaaS application portfolio analysis tool. It is an extremely lightweight tool that can integrate with pretty much any build process right now," explained Andrew Siegmund, Application Migration Specialist for CAST, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
"We view the cloud not as a specific technology but as a way of doing business and that way of doing business is transforming the way software, infrastructure and services are being delivered to business," explained Matthew Rosen, CEO and Director at Fusion, in this SYS-CON.tv interview at 18th Cloud Expo (http://www.CloudComputingExpo.com), held June 7-9 at the Javits Center in New York City, NY.
You often hear the two titles of "DevOps" and "Immutable Infrastructure" used independently. In his session at DevOps Summit, John Willis, Technical Evangelist for Docker, covered the union between the two topics and why this is important. He provided an overview of Immutable Infrastructure then showed how an Immutable Continuous Delivery pipeline can be applied as a best practice for "DevOps." He ended the session with some interesting case study examples.
In his session at Cloud Expo, Alan Winters, U.S. Head of Business Development at MobiDev, presented a success story of an entrepreneur who has both suffered through and benefited from offshore development across multiple businesses: The smart choice, or how to select the right offshore development partner Warning signs, or how to minimize chances of making the wrong choice Collaboration, or how to establish the most effective work processes Budget control, or how to maximize project result...
"DivvyCloud as a company set out to help customers automate solutions to the most common cloud problems," noted Jeremy Snyder, VP of Business Development at DivvyCloud, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
Explosive growth in connected devices. Enormous amounts of data for collection and analysis. Critical use of data for split-second decision making and actionable information. All three are factors in making the Internet of Things a reality. Yet, any one factor would have an IT organization pondering its infrastructure strategy. How should your organization enhance its IT framework to enable an Internet of Things implementation? In his session at @ThingsExpo, James Kirkland, Red Hat's Chief Archi...
In his general session at 19th Cloud Expo, Manish Dixit, VP of Product and Engineering at Dice, discussed how Dice leverages data insights and tools to help both tech professionals and recruiters better understand how skills relate to each other and which skills are in high demand using interactive visualizations and salary indicator tools to maximize earning potential. Manish Dixit is VP of Product and Engineering at Dice. As the leader of the Product, Engineering and Data Sciences team at D...
Without a clear strategy for cost control and an architecture designed with cloud services in mind, costs and operational performance can quickly get out of control. To avoid multiple architectural redesigns requires extensive thought and planning. Boundary (now part of BMC) launched a new public-facing multi-tenant high resolution monitoring service on Amazon AWS two years ago, facing challenges and learning best practices in the early days of the new service.
In his keynote at 19th Cloud Expo, Sheng Liang, co-founder and CEO of Rancher Labs, discussed the technological advances and new business opportunities created by the rapid adoption of containers. With the success of Amazon Web Services (AWS) and various open source technologies used to build private clouds, cloud computing has become an essential component of IT strategy. However, users continue to face challenges in implementing clouds, as older technologies evolve and newer ones like Docker c...
We all know that end users experience the internet primarily with mobile devices. From an app development perspective, we know that successfully responding to the needs of mobile customers depends on rapid DevOps – failing fast, in short, until the right solution evolves in your customers' relationship to your business. Whether you’re decomposing an SOA monolith, or developing a new application cloud natively, it’s not a question of using microservices - not doing so will be a path to eventual ...
We all know that end users experience the Internet primarily with mobile devices. From an app development perspective, we know that successfully responding to the needs of mobile customers depends on rapid DevOps – failing fast, in short, until the right solution evolves in your customers' relationship to your business. Whether you’re decomposing an SOA monolith, or developing a new application cloud natively, it’s not a question of using microservices – not doing so will be a path to eventual b...
We all know that end users experience the internet primarily with mobile devices. From an app development perspective, we know that successfully responding to the needs of mobile customers depends on rapid DevOps – failing fast, in short, until the right solution evolves in your customers' relationship to your business. Whether you’re decomposing an SOA monolith, or developing a new application cloud natively, it’s not a question of using microservices - not doing so will be a path to eventual ...
All organizations that did not originate this moment have a pre-existing culture as well as legacy technology and processes that can be more or less amenable to DevOps implementation. That organizational culture is influenced by the personalities and management styles of Executive Management, the wider culture in which the organization is situated, and the personalities of key team members at all levels of the organization. This culture and entrenched interests usually throw a wrench in the work...
JetBlue Airways uses virtual environments to reduce software development costs, centralize performance testing, and create a climate for continuous integration and real-time monitoring of mobile applications. The next BriefingsDirect Voice of the Customer performance engineering case study discussion examines how JetBlue Airways in New York uses virtual environments to reduce software development costs, centralize performance testing, and create a climate for continuous integration and real-tim...