Tier-Two Testing: Penetration Testing
Not all security vulnerabilities can be found though static analysis, so penetration testing comes into the picture to expose such problems. Penetration testing dynamically exercises and scans the Web Service deployed on a staging or production server.

Understanding the security threats lets the tester design tests that can expose them with the help of good tools. For example, external entity attacks and XML bombs can be thrown at the service to see if the service refuses to process XML processing instructions or DTDs by returning a SOAP fault. WSDL access vulnerabilities can be detected by attempting to get a WSDL without the expected security channel if it's protected. For example, if the WSDL is protected with client-side SSL on port 443, it shouldn't be accessible on port 80; it's possible to forget an open connector in the Web server, which leaves multiple open channels. When it comes to thwarting WSDL scanning threats, then it's important to inspect the WSDL for redundant artifacts such as schemas or unused message definitions.

Capture and replay attacks can be simulated by sending multiple requests with the same message identifier that determines its uniqueness. For example, if you're using Username Tokens, you should test the service by sending multiple messages with the same nonce values and verify that the service rejects such requests properly. The service should implement a sufficient, but limited cache size for the recently accepted nonce values. Many WS-Security implementations don't take this into consideration by default, which makes them vulnerable to capture and replay attacks.

To test a Web Service's vulnerability to DoS attacks caused by heavy loads, such DoS attacks should be simulated in fashion that's suitable to Web Services. You can't tell if a service can sustain a certain load scenario unless such a scenario has been tested. However, it's important to execute such load tests in a manner that's effective.

Some test engineers have the tendency to do load tests with the same static request to generate a load. Although this is a viable test scenario, it's not sufficient because such DoS attacks can be detected by network security appliances. Therefore, Web Service DoS attack simulations should be generated with dynamic request values that are semantically valid and can exercise wider code coverage in the Web Service's application logic to test the Web Services to its limits. Such attacks are difficult to generate by manual coding, but they're possible with load testing tools that are specialized for Web Services. In fact, the mere existence of such tools should alert Web Service engineers that such attacks can be done easily by a hacker if such tools fell into their hands. For example, to test a Web Service that accepts Username Tokens with timestamps and nonces, it's important to apply a load on the service where the timestamps and nonces are dynamically generated for each request. Otherwise, errors such as the ones caused by concurrency problems would go undetected. Another example would be load tests that send signed requests, where the hash and signature values should differ from one message to another.

Not only should Web Service load tests generate dynamic requests, but such tests should also simulate real-use case scenarios or usage patterns. For example, a use case scenario could be a Web Service client retrieving an authorization token (such as a SAML assertion) from a security authority, then using that token for subsequent Web Service invocations on different services. To test that scenario, load tests that keep using the same authorization token over and over again don't represent the real-world scenario since a real-use scenario would have multiple users requesting and using multiple tokens at the same time. Executing such a realistic load test can expose concurrency or scalability problems that result in vulnerabilities. In this example, it's possible for the Web Service to reject valid requests or accept unauthorized ones under a certain load even if such problems don't occur during regular functional testing.

To detect invalid responses during a load test, the load tests should be backed with sufficient response validations that ensure the detection of regressions from the correct behavior, because it's difficult to verify that all requests were met with the correct responses unless regression detection was done while the load is being generated. Without response validation, only network connections and HTTP errors would be exposed, which doesn't provide sufficient test coverage. For example, responses can be well-formed SOAP messages but with invalid data, or perhaps they contain an error message when they shouldn't. Without placing sufficient response validation during a load test, such incorrect responses can go undetected.

Tier-Three Testing: Runtime Analysis
Runtime analysis of the state of Web application code is needed to detect certain security problems that can't be detected with the previous two tiers of testing. For example, in C/C++ applications that are exposed as Web Services, memory corruption (especially memory corruption on the stack) indicates a potential for buffer overflows that could cause serious security problems, and memory leaks make the application more vulnerable to denial of service attacks. Dynamic analysis can find security vulnerabilities that can result from the integration of otherwise secure components because it takes data flow analysis into consideration, whereas static analysis provides large code coverage with a narrower scope on data flows.

Combining the Three Tiers
Since each security-testing tier provides a methodology exposing vulnerabilities from a unique aspect, combining two or more of the three tiers could provide a powerful approach to security testing. For example, static analysis can be used to determine the scope of the required penetration testing by recommending a more selective set of possible vulnerabilities to penetrate.

Runtime analysis combined with penetration testing gives the tester visibility into the application as it performs under a variety of conditions. For example, one can do runtime analysis during load testing to find memory leaks.

Step 4: Create & Maintain Reusable, Re-runnable Tests
The above testing practices can become too expensive to do unless proper automation is applied to the testing process. Many organizations don't have the resources to do these tests if they were to be done manually and repeated for each project milestone.

Modern software development processes are iterative. Software engineering activities should be done on a recurring, iterative basis rather than following a rigid, one-directional development model that tests only at the end. Testing only at the end of the development cycle is one of the main reasons for late deliveries and exceeded project costs and Web Services are no exception to this fact.

However, such an iterative development model can only be effective if the engineering activities are backed with proper automation. Therefore, it's necessary to establish a Web Services testing environment that's driven by automation that can help create the tests, maintain them, manage them, and execute them on a regular basis; typically every night as part of the existing "nightly" build and test process for the product. The alternative would be to run the various Web Services tests manually, each one at a time, by modifying a client's request, which is a tedious, non-efficient process. It's therefore better to keep and maintain all the Web Service tests that are created so they can be re-run quickly, easily, and so you can run them all automatically as regression tests whenever a Web Service is updated.

After running security tests along the three tiers we described, one can find problems that require fixes that ripple through Web Service at a time when they're too risky or expensive to fix, which is why such tests are better executed early and regularly.

When a problem is discovered then the test that exposed the problem should ideally be added to the existing test pool and re-run on a recurring basis with all the other tests so it prevents that error from occurring again.

Conclusion
Securing your Web Services is a vital aspect of ensuring a successful deployment. When deployed externally for consumption by partners or customers, only secure Web Services can provide a justifiable integration solution, because the benefits they expose should far outweigh the risks. The key to effective Web Services security is to know and be aware of the various types of security threats, understand the technical solutions for mitigating these threats then establish and follow a defined engineering process that takes security into consideration from the beginning and throughout the Web Service lifecycle. By following the four steps outlined in this article, you can ensure complete Web Service security.