Click here to close now.




















Welcome!

Microservices Expo Authors: Elizabeth White, SmartBear Blog, Pat Romanski, Ruxit Blog, VictorOps Blog

Related Topics: Microservices Expo

Microservices Expo: Article

SOA World Product Review — Testing...1...2...3...Testing

A review of SOAtest 5.0

Few topics evoke more groans and eye rolling from software engineers and Web developers than the dreaded "TESTING." Testing falls into the same category as documentation, refactoring code, dusting, and visiting the dentist. Put it off until the last minute, do as little as possible, do it quickly, and move on to something else. I must confess that I have the same visceral reaction to the thought of 'testing' as others do. Consequently, I approached the prospect of reviewing a testing tool with the loathing of visiting the dentist. I was very relieved to discover that Parasoft's SOAtest 5.0 took a lot of the pain, frustration, and busy work out of the testing experience.

Parasoft's SOAtest 5.0 is a comprehensive testing and analysis tool suite tailored to the unique testing and validation needs of Service Oriented Architectures. It supports functional testing, scenario-based testing, stress testing, client testing with a mock service, and a whole range of validation capabilities (XML Schema, WSDL, WS-Security, BPEL, etc.). SOAtest 5.0 then further supports the creation and automated execution of regression test suites. It supports a broad range of SOA specifications and standards, and is designed from the ground up to support the dynamic and evolving nature of service-oriented systems.

Getting Started with SOAtest 5.0
Installing SOAtest was a breeze. The process is as follows:

  • Run the setup executable from media or by downloading it from www.parasoft.com,
  • Follow the on-screen prompts.
  • Decide if you want to set up SOAtest as a Windows service.

Once the software is installed, you're ready to begin using SOAtest. The first time that you launch the software, you'll be prompted either to input your individual license information or point to an available license server.

After providing licensing information, you will be confronted with an option to create a New Project, Open an Existing Project, or access the SOAtest Tutorial. The tutorial is well-written and provides a nice tour of the tool's major features.

Creating Test Cases
Tests can either be created individually or as a part of a larger test suite. The tool seems to drive you toward creating test suites rather than individual tests, which is nice for reuse, organization, and best of all - regression testing a collection of tests. SOAtest supports the creation of test suites from a wide range of sources including:

  • Web Services Description Language (WSDL) files
  • Business Process Execution Language (BPEL) files
  • Universal Description, Discovery & Integration (UDDI) registry end-points
  • Web Services Inspection Language (WSIL) files
  • BEA Aqualogic Enterprise Repository end-points

The test suite creation wizard will ask you to designate a file, URL, or end-point to query and also ask you to designate what tests you want to create. Upon completing the wizard, you'll have a whole set of tests automatically generated and ready-to-run as is or customized prior to execution.

Functional Testing
I was very impressed with the functional verification testing capabilities. SOAtest supports the following functional verification features:

  • Check schema validity against a WSDL
  • XML-aware diff engine that flags only true XML structure changes
  • Surgical inclusion/exclusion of XML message elements in the verification process via XPATH (see Figure 1)
  • A graphical rules engine for managing assertions

Every one of these features is enabled through simple intuitive menu options, dropdown lists, check boxes, and XML tree structures. The interface is simple to navigate and although there's a wealth of options, they're organized to avoid the feeling of being overwhelmed.

Executing one or more functional tests multiple times based on an external data set (database or spreadsheet) was another feature that I sought out. SOAtest came through in this area as well supporting the ability to point at a data source (CSV, Excel, relational databases, etc.) and run through a battery of tests that pull values out of the data source and send service requests containing the extracted data values. This lets you define a single test case or suite of tests and then automatically test the full range of data values that the test case needs to support without creating additional tests for each value.

Testing with Mock Services
In a truly test-driven environment, I'd create service interfaces and corresponding suites of test cases before any implementation code is developed. Although this is good in theory, without a supporting toolset, it's not very realistic. SOAtest provides all the capabilities needed to support this kind of development model. When creating a new project/test suite, simply point the tool at a WSDL file and check the "Generate Server Stubs" radio button. The server stubs will run on SOAtest's embedded Tomcat server and let you create and run testing scenarios before you've written a single line of implementation code. This way, when you're ready to implement the service interfaces, you have a defined a suite of tests for verification. This helps to control the scope (when you meet all the tests, stop working) and provides one or more test suites that can be incorporated into automatic regression testing (see below) to ensure that functionality isn't compromised as the project progresses.

Of all the features that I've worked with, I found the service mock-up process the most cumbersome. The other aspects of SOAtest were intuitive and easy-to-work with, whereas I had to wrestle a while to get the mock service capability working on anything other than the tutorial walk-through. In the end, the functionality of this capability was excellent, albeit a bit difficult to initially configure.

Regression Testing
Running your tests once is nice. Running your tests regressively is better. Running your tests regressively and automatically at night is divine. SOAtest supports all three scenarios. Any of the test suites that you define in a SOAtest project can be converted to regression test suites. Furthermore, using XPATH you can selectively indicate which portions of the test cases may change from test to test and which values should never change. Once you have a set of regression tests that you're happy with, SOAtest provides a command-line mechanism to kick off your test suites automatically. Thus in an agile, continuous integration environment you can run regression testing at night, at lunch, or every hour on the hour to ensure that you find bugs early and often.

Reporting
SOAtest provides several reporting features (auto-generated reports for nightly regression tests, on-demand, detailed summary reports for test suites, and WS-I interoperability reports). I found the quality and readability of the reports developed by SOAtest to be quite good. Figure 2 provides a snapshot of part of the SOAtest detailed report. The WS-I interoperability reports were pretty low quality, difficult to navigate, and provided information overload. In fairness to SOAtest, those reports are copyrighted by WS-I and seem to be auto-generated by one or more WS-I tools. Consequently, I'm not sure that Parasoft has any control over the quality of these reports. Nonetheless, I would have liked a WS-I conformance report of the same quality and user-friendliness as the native SOAtest reports.

Advanced Testing Features
The palette of testing and analysis features in SOAtest is extensive. In the interest of not filling up this entire magazine with feature descriptions, I'll list several of the compelling features that I've not covered already:
- Scenario-based testing where subsequent test cases depend on data returned from previous test cases
- Stress/load testing your SOA and specifying Quality of Service parameters
- Testing non-XML services (JMS, MQ, TIBCO, EJB, REST, Binary, Text, etc.)
- Validate SOAP security using WS-Security (encryption, digital signatures, and authentication)
- Asynchronous service testing
- Custom scripting with Python, JavaScript, or Java (Figure 3)

Drawbacks
As I mentioned earlier, the mock service feature is a bit awkward to work with initially. Also, the WS-I conformance report was not up to the same quality standards as the native SOAtest reports. With SOA's strong integration and interoperability play, there are a lot of enterprises that have Java and other services that all need to be involved in the same business process and testing scenario.

Summary
In spite of my general dislike for software analysis and testing, I found Parasoft's SOAtest 5.0 to be a well-designed tool that took a lot of the pain and work out of testing and validating a SOA. The tool isn't perfect, but it is easily one of the best SOA testing tools that I've ever worked with.

Product Snapshot

Target Audience: SOA architects, developers, QA/testers, and analysts

Level: Beginner to Advanced

Pros: Powerful, intuitive UI, robust testing and analysis tools

Cons: The mock service feature is a bit awkward, only supports Java clients for now

Testing environment: Dell Inspiron 640m, 1.6GHz Intel Core-Duo, 2GB RAM, Windows XP Pro with SP2

Platforms: Windows 2000/XP, Linux, Solaris

More Stories By Kyle Gabhart

Kyle Gabhart is a subject matter expert specializing in strategic planning and tactical delivery of enterprise technology solutions, blending EA, BPM, SOA, Cloud Computing, and other emerging technologies. Kyle currently serves as a director for Web Age Solutions, a premier provider of technology education and mentoring. Since 2001 he has contributed extensively to the IT community as an author, speaker, consultant, and open source contributor.

Comments (1) View Comments

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


Most Recent Comments
SOA News 04/04/07 04:11:45 PM EDT

Few topics evoke more groans and eye rolling from software engineers and Web developers than the dreaded 'TESTING.' Testing falls into the same category as documentation, refactoring code, dusting, and visiting the dentist. Put it off until the last minute, do as little as possible, do it quickly, and move on to something else. I must confess that I have the same visceral reaction to the thought of 'testing' as others do. Consequently, I approached the prospect of reviewing a testing tool with the loathing of visiting the dentist. I was very relieved to discover that Parasoft's SOAtest 5.0 took a lot of the pain, frustration, and busy work out of the testing experience.

@MicroservicesExpo Stories
Culture is the most important ingredient of DevOps. The challenge for most organizations is defining and communicating a vision of beneficial DevOps culture for their organizations, and then facilitating the changes needed to achieve that. Often this comes down to an ability to provide true leadership. As a CIO, are your direct reports IT managers or are they IT leaders? The hard truth is that many IT managers have risen through the ranks based on their technical skills, not their leadership ab...
Whether you like it or not, DevOps is on track for a remarkable alliance with security. The SEC didn’t approve the merger. And your boss hasn’t heard anything about it. Yet, this unruly triumvirate will soon dominate and deliver DevSecOps faster, cheaper, better, and on an unprecedented scale. In his session at DevOps Summit, Frank Bunger, VP of Customer Success at ScriptRock, will discuss how this cathartic moment will propel the DevOps movement from such stuff as dreams are made on to a prac...
Several years ago, I was a developer in a travel reservation aggregator. Our mission was to pull flight and hotel data from a bunch of cryptic reservation platforms, and provide it to other companies via an API library - for a fee. That was before companies like Expedia standardized such things. We started with simple methods like getFlightLeg() or addPassengerName(), each performing a small, well-understood function. But our customers wanted bigger, more encompassing services that would "do ...
SYS-CON Events announced today that Pythian, a global IT services company specializing in helping companies leverage disruptive technologies to optimize revenue-generating systems, has been named “Bronze Sponsor” of SYS-CON's 17th Cloud Expo, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. Founded in 1997, Pythian is a global IT services company that helps companies compete by adopting disruptive technologies such as cloud, Big Data, advance...
What does “big enough” mean? It’s sometimes useful to argue by reductio ad absurdum. Hello, world doesn’t need to be broken down into smaller services. At the other extreme, building a monolithic enterprise resource planning (ERP) system is just asking for trouble: it’s too big, and it needs to be decomposed.
Early in my DevOps Journey, I was introduced to a book of great significance circulating within the Web Operations industry titled The Phoenix Project. (You can read our review of Gene’s book, if interested.) Written as a novel and loosely based on many of the same principles explored in The Goal, this book has been read and referenced by many who have adopted DevOps into their continuous improvement and software delivery processes around the world. As I began planning my travel schedule last...
Docker containerization is increasingly being used in production environments. How can these environments best be monitored? Monitoring Docker containers as if they are lightweight virtual machines (i.e., monitoring the host from within the container), with all the common metrics that can be captured from an operating system, is an insufficient approach. Docker containers can’t be treated as lightweight virtual machines; they must be treated as what they are: isolated processes running on hosts....
SYS-CON Events announced today that G2G3 will exhibit at SYS-CON's @DevOpsSummit Silicon Valley, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. Based on a collective appreciation for user experience, design, and technology, G2G3 is uniquely qualified and motivated to redefine how organizations and people engage in an increasingly digital world.
SYS-CON Events announced today that HPM Networks will exhibit at the 17th International Cloud Expo®, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. For 20 years, HPM Networks has been integrating technology solutions that solve complex business challenges. HPM Networks has designed solutions for both SMB and enterprise customers throughout the San Francisco Bay Area.
SYS-CON Events announced today the Containers & Microservices Bootcamp, being held November 3-4, 2015, in conjunction with 17th Cloud Expo, @ThingsExpo, and @DevOpsSummit at the Santa Clara Convention Center in Santa Clara, CA. This is your chance to get started with the latest technology in the industry. Combined with real-world scenarios and use cases, the Containers and Microservices Bootcamp, led by Janakiram MSV, a Microsoft Regional Director, will include presentations as well as hands-on...
The pricing of tools or licenses for log aggregation can have a significant effect on organizational culture and the collaboration between Dev and Ops teams. Modern tools for log aggregation (of which Logentries is one example) can be hugely enabling for DevOps approaches to building and operating business-critical software systems. However, the pricing of an aggregated logging solution can affect the adoption of modern logging techniques, as well as organizational capabilities and cross-team ...
In today's digital world, change is the one constant. Disruptive innovations like cloud, mobility, social media, and the Internet of Things have reshaped the market and set new standards in customer expectations. To remain competitive, businesses must tap the potential of emerging technologies and markets through the rapid release of new products and services. However, the rigid and siloed structures of traditional IT platforms and processes are slowing them down – resulting in lengthy delivery ...
Skeuomorphism usually means retaining existing design cues in something new that doesn’t actually need them. However, the concept of skeuomorphism can be thought of as relating more broadly to applying existing patterns to new technologies that, in fact, cry out for new approaches. In his session at DevOps Summit, Gordon Haff, Senior Cloud Strategy Marketing and Evangelism Manager at Red Hat, discussed why containers should be paired with new architectural practices such as microservices rathe...
Any Ops team trying to support a company in today’s cloud-connected world knows that a new way of thinking is required – one just as dramatic than the shift from Ops to DevOps. The diversity of modern operations requires teams to focus their impact on breadth vs. depth. In his session at DevOps Summit, Adam Serediuk, Director of Operations at xMatters, Inc., will discuss the strategic requirements of evolving from Ops to DevOps, and why modern Operations has begun leveraging the “NoOps” approa...
Puppet Labs has announced the next major update to its flagship product: Puppet Enterprise 2015.2. This release includes new features providing DevOps teams with clarity, simplicity and additional management capabilities, including an all-new user interface, an interactive graph for visualizing infrastructure code, a new unified agent and broader infrastructure support.
DevOps has traditionally played important roles in development and IT operations, but the practice is quickly becoming core to other business functions such as customer success, business intelligence, and marketing analytics. Modern marketers today are driven by data and rely on many different analytics tools. They need DevOps engineers in general and server log data specifically to do their jobs well. Here’s why: Server log files contain the only data that is completely full and accurate in th...
It’s been proven time and time again that in tech, diversity drives greater innovation, better team productivity and greater profits and market share. So what can we do in our DevOps teams to embrace diversity and help transform the culture of development and operations into a true “DevOps” team? In her session at DevOps Summit, Stefana Muller, Director, Product Management – Continuous Delivery at CA Technologies, answered that question citing examples, showing how to create opportunities for ...
SYS-CON Events announced today that DataClear Inc. will exhibit at the 17th International Cloud Expo®, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. The DataClear ‘BlackBox’ is the only solution that moves your PC, browsing and data out of the United States and away from prying (and spying) eyes. Its solution automatically builds you a clean, on-demand, virus free, new virtual cloud based PC outside of the United States, and wipes it clean...
In his session at 17th Cloud Expo, Ernest Mueller, Product Manager at Idera, will explain the best practices and lessons learned for tracking and optimizing costs while delivering a cloud-hosted service. He will describe a DevOps approach where the applications and systems work together to track usage, model costs in a granular fashion, and make smart decisions at runtime to minimize costs. The trickier parts covered include triggering off the right metrics; balancing resilience and redundancy ...
The Microservices architectural pattern promises increased DevOps agility and can help enable continuous delivery of software. This session is for developers who are transforming existing applications to cloud-native applications, or creating new microservices style applications. In his session at DevOps Summit, Jim Bugwadia, CEO of Nirmata, will introduce best practices, patterns, challenges, and solutions for the development and operations of microservices style applications. He will discuss ...