Testing Versus Production
SOA performance can be broken down into two major phases based on the application life cycle. The first phase of performance testing occurs during the application development process. In this phase, load testing intended to simulate the production environment is used at various stages to ensure the application functionally behaves as intended at acceptable performance levels. The final QA phase of the development cycle, sometimes using synthetic transactions, typically includes testing to place maximum stress on the application code in the test environment before declaring it ready to be released for day-to-day production. This level of testing is a given in any development organization in that few organizations would consider releasing a mission-critical application without a final QA process that includes some level of performance testing. When the application exits the QA process and enters production, the focus on performance shifts from code path validation and stress testing to ongoing production performance management.

This second phase involving performance is the ongoing monitoring, reporting, and triage associated with the application once it is released to production. This ongoing activity is typically characterized as Application Performance Management. This is as important as the QA test activities and perhaps more so since poor performance in actual live production impacts the users of application services and can be highly visible both internally and externally. Unfortunately this is often the phase that is overlooked or given little attention during the initial SOA strategy planning process. Organizations may assume that the existing performance tools in place can handle the task. In addition they may overlook the need for a common language and agreed process to measure and report on performance in terms of Service Level Agreements that represent the performance commitment to the business unit.

The remainder of the discussion of SOA performance that follows focuses on considerations surrounding the complexity and dynamics of the production environment along with key considerations and challenges organizations must address to ensure success.

Managing performance is far more complex than producing historical reports on average response times and outages. Successful performance management must be based on real-time, end-to-end customer transaction monitoring and reporting across the entire SOA infrastructure. Historical data can, however, be a very valuable input to correlation and probable cause analysis. Successful performance monitoring should help IT operations better execute its new charter of being the business relationship manager between the end user and the business unit. It must provide a comprehensive view of the entire customer experience with the ability to detect slow transaction times as they develop and provide alerts and probable cause analysis. Armed with this information, IT operations can proactively address the issue before the end user is affected. Successful performance management provides a common language and reference data for use across the infrastructure for problem triage. Ideally all of this is accomplished with minimum overhead and real-world actual customer transaction experience information.

Service-Level Agreement Management
Many organizations establish a contract in the form of an SLA to define both target and minimum acceptable performance levels for an application. Frequently people think of SLAs when discussing external user-facing applications such as a business-to-consumer (BC) environment like retail sales, online banking, or other self-service portals. In fact SLAs should be part of the DNA of all applications. External customer or supplier SLAs may be more stringent but an enterprise's own employees have a need for predictable and reliable internal application performance as well. While external-facing application failures or slowdowns can be costly in terms of lost revenue or damaged reputation, poor internal-facing application performance can also be costly in terms of lost productivity and employee moral.

Much like the distinction between different SLA targets for external versus internal users, SLAs can be established at various points in the business process based on factors such as the nature of the transaction or the specific user. A brokerage firm, for example, would likely place the highest service-level priority on a specific business process such as executing a trade. A delay in any step of a trade transaction could make a significant difference in the amount of money saved or lost based on the overall transaction time. The SLA for that particular process therefore would set at a very demanding level. On the other hand the service level required for a less-critical function like printing a portfolio summary would likely be set at a less stringent level. Similarly that same brokerage firm may choose to set a higher service level for a very large customer executing frequent or high-volume, high-value trades versus an occasional user executing smaller or infrequent trades. While both users have high expectations for fast, reliable service, one may have a higher business value to the enterprise.

Another important aspect of APM in general, but SLA management specifically, is to have a common language and measurement for the SLA that reflects the actual customer experience that can be understood by both IT and the business process owner. While IT will have a need for much more detailed technical information for problem identification and avoidance, this level of detail is of little value or interest at the business unit level. The tools therefore that are used for SLA management should be capable of producing useful and meaningful information for both the technologist and the business process owner.