So, what do you test for in services? First, it's important to follow a few basic principles.

First and foremost, services should be tested for reuse (reusability). Services become a part of any number of other applications and must be tested so they properly provide behavior and information but not be application- or technology-specific. This is a difficult paradigm for many developers since one-off custom software that digs deeply into native features is what they've been doing for most of their careers. So the patterns must be applicable to more than a single problem domain, or application, or standard, meaning you must have use for your reusable service and it must be in good working order.

To test for reusability you must create a list of candidate uses for the service. For instance, a shipping service that plugs into accounting, inventory, and sales systems (see Figure 2). Then the service should be consumed by the client, either through a real application (in a testing domain) or a simulator, and the results noted.

In addition, the service should be tested for heterogeneity. Web Services should be built and tested so that there are no calls to native interfaces or platforms. This is due to the fact that a service, say one built on Linux, may be leveraged by applications on Windows, Macs, and even mainframes. Those that leverage your service should do so without regard for how it was created, and should be completely platform-independent. The approach to testing this is rather obvious; simply consume the service on several different platforms and note any calls to the native subsystems.

You should also test for abstraction. Abstraction allows access to services from multiple simultaneous consumers; hiding technology details from the service developer. The use of abstraction is required to get around the many protocols, data access layers, and even security mechanisms that may be in place, thus hiding these very different technologies behind a layer that can emulate a single layer of abstraction.

Abstraction is tested effectively by doing, meaning implementing instances and then testing the results. Regression and integration testing is the best approach, from the highest to the lowest layers of abstraction.

When we build or design services we need to test for aggregation. Many services will become parts of other services, and thus composite services leveraged by an application, and you must consider that in their design. For instance, a customer validation service may be part of a customer processing service, which is part of the inventory control systems. Aggregations are clusters of services bound together to create a solution, and should be tested holistically through integration testing procedures.

Service testing means different things to different organizations due to the fact that SOA is so new. Most people who are testing services attempt to figure it out as they go along, typically rolling their own tools for testing, including service consumption test harnesses and service producer test harnesses for the particular use cases they are testing. Others are learning to leverage off-the-shelf testing tools, such as Parasoft SOAtest or Mindreef SOAPscope.

Security-Level Testing
Security strategy, technology, and implementation should be systemic to a SOA and even bring along new concepts such as identity management. When testing your SOA for security issues, the best approach is first to understand the security requirements, and then design a test plan around those requirements, pointing at specific vulnerabilities. Most are finding that black box testing is the best way to test for security issues in the world of SOA, including penetration testing, vulnerability testing, etc., using existing techniques and tools.

A further concern of SOA security is the fact that many SOAs allow services to be consumed outside the enterprise and so create a new set of vulnerabilities including information security issues and denial of service attacks. Moreover, many SOAs also make the reverse trip, allowing for the consumption of services outside of the firewall into the SOA. That opens the door for other types of attacks and the security needs to be tested in this case as well. Vulnerabilities here include malicious services.

Again, most people who are testing their SOA for security issues have a tendency to roll their own approaches and build there own tools. However, a few new tools are appearing on the market such as Vordel SOAPbox to test the security of XML applications, such as XML Web Services. It's used during development and deployment phases to test an XML application's compliance to security standards. SOAPbox highlights security tokens, signatures, and encrypted content in XML documents.

Orchestration-Level Testing
For our purposes, we can define orchestration as a standards-based mechanism that defines how Web Services work together, including business logic, sequencing, exception handling, and process decomposition, including service and process reuse.

Orchestrations may span a few internal systems, systems between organizations, or both. Moreover, orchestrations are long-running, multi-step transactions, almost always controlled by one business party and are loosely coupled and asynchronous in nature.

We can consider orchestration as really another complete layer over and above more traditional application integration approaches, including information- and service-oriented integration. Orchestration encapsulates these integration points, binding them together to form higher-level processes and composite services. Indeed, orchestrations themselves should become services. So you test them as you would other services, including abstraction, reuse, granularity, etc. However, you should note that these services sit above existing services and the testing should regress from the top-level services down to the bottom-level services. Or from the orchestration layer down to the primitive services. Tools such as Mindreef SOAPscope: Solutions for Web Services and SOA may be effective here as well.

Governance-Level Testing
SOA Governance is an emerging discipline, which enables organizations to provide guidance and control of their SOA initiatives and programs.

Although there are many SOA governance vendors and thus many SOA governance definitions, the best is from Wikipedia.

Many organizations are attempting to transition from silo-oriented applications to agile, composite clients and services. This transition requires that the 'service' become the new unit of work. The IT organization must now manage these services across the entire lifecycle, from inception through analysis, design, construction, testing, deployment, and production execution. At each stage, certain rules or policies must be carried out to ensure that the services provide value to the consumers. SOA governance is the discipline of creating policies, communicating, and enforcing them.

So, in essence you have a lifecycle and policy management layers, including testing, that needs testing. No problem. You test governance systems by matching up the policies the governance system is looking to manage and control with the actual way they manage and control them. Thus, it's a simple matter of listing the policies and establishing test cases for each. Such as:

XYZ Service can only leverage services within the firewall test policy
Does the policy disallow that service from leveraging services outside the firewall?

Easy as that, and there are no tools for testing SOA governance systems other than those provided by the SOA governance solution vendors.

Integration-Level Testing
As with traditional integration testing, the purpose of this step is to figure out if all of the interfaces, including behavior and information sharing between the services, are working correctly. The type of integration testing you carry out should work through the layers of communications. Working up through the network to the protocols and inter-process communications, including testing the REST or SOAP interfaces to the services or whatever communication mechanism are employed by the services you're deploying.

Things to look for here include:

• Can communications be established with late binding, meaning dynamically as-needed?
• Is the integration stable under an increasing load?
• Is the transmitted information correct for the service or applications?
• Are the security mechanisms working properly?
• How does the SOA recover from application, database, and network failures?

At the end of the day you'll find that SOA testing incorporates all of the testing technology and approaches we've developed over the years for other distributed systems and adds some new dimensions such as services, orchestration, and governance testing. Those who understand testing will have to expand their skills and understanding to include those new dimensions. Many failed SOA projects are directly attributable to lack of testing...many assuming that the new technology would work flawlessly. Unfortunately, that's just not the real world yet.