Service Oriented Architecture (SOA) projects have evolved. A couple of years ago it would have been sufficient to demonstrate connectivity between systems that were previously isolated. Nowadays these connections must provide guarantees of reliability, security, and performance. Delivering on such requirements presents a number of challenges.

Not all of the challenges faced during an SOA project are technical. However, it's the technical requirements that are best understood and most often explicitly stated as requirements. Other issues relate to the changes required in the way IT operations are organized in a company. While it's often the technical challenges that are focused on in the early projects, many organizations are discovering that this approach doesn't make for a successful SOA deployment.

The use of an Enterprise Service Bus (ESB) to implement an SOA addresses many of the technical challenges encountered by such projects, allowing organizations to concentrate on the other challenges that they'll face. A recent report by Gartner Dataquest showed that the Enterprise ESB market is the fastest growing application integration middleware market segment - with growth rates exceeding 100% year-over-year. So why are so many enterprises turning to ESBs to deliver their SOA projects?

This article discusses some of the technical issues solved by the ESB. It then goes on to look at the other SOA challenges relating to organizational changes that are seldom acknowledged at the start of an SOA project and often deserves the most attention.

The Problems Addressed by the ESB Connectivity
The systems in any large organization will use a variety of transport layers. For example, there will be at least one message queue system. FTP, e-mail, files dropped into directories, and of course HTTP may also be used. An ESB can connect to all of these transports in a consistent manner, isolating the developer from the specific issues associated with each one. Appropriate use of an ESB can make the selection of a transport a deployment time issue rather than a development one.

Data Transformation
Systems will invariably use different message formats. SOA must first standardize on the use of XML. The goal then is to to use a common vocabulary of XML elements across most messages. Legacy systems are unlikely to use XML, so each format will have to be transformed to and from XML. The ESB provides tools and runtime components to handle this data transformation.

An initial SOA project may struggle to define a set of common XML elements. This will result in the need to perform additional XML-to-XML transformation on the ESB. In some cases, this may seem like surplus overhead. However, it's a necessary step on the way to achieving the goal of common message formats.

Reliability
The reliability and traceability of messages flowing through the ESB is now a fundamental requirement of most SOA deployments. This means that the ESB should be based on a reliable message layer - typically one of the JMS implementations. Alternatively, the ESB can provide reliability on top of unreliable transports (typically HTTP) by using protocols such as WS-RM.

A reliable transport isolates the application from communication errors that occur at runtime. This reduces the error recovery code required in the application. It also eliminates any application code that attempts to redeliver failed messages since this is done automatically at the transport layer. Failure to adopt a reliable transport can result in the logic to catch faults and perform retries being implemented in the application layer. This is not a wise use of developer time.

Security
Many ESBs provide security by using protocols such as HTTPS. Howeverusing these protocols tie you to a single transport. Using WS-Security provides a transport-independent security mechanism. WS-Security can be added at the message-processing layer without needing modifications to the applications developed. This lets security requirements be considered relatively independently of the functional requirements for a project.

In SOA, there's increasing emphasis on the use of policies. Features that would have previously been developed as part of the application are being separated out and provided by the ESB. Security and reliability are two areas where a policy-driven approach is being used. The ESB is configured using a policy document to both provide and enforce the level of security required.

Asynchronous
Systems should be designed from the start to communicate using asynchronous messaging. A reliable asynchronous messaging system can isolate systems from faults relating to the temporary unavailability of a single service. Asynchronous messaging also provides a way to handle an increase in traffic more gracefully than synchronous messaging. The use of asynchronous messaging introduces the need to address issues such as message correlation. The ESB performs this task by using protocols such as WS-Addressing or by correlating features of languages such as WS-BPEL.

Orchestration
A fundamental aspect of an SOA is the need to provide services that perform specific tasks as defined by business need rather than the technology. Unfortunately, there's seldom a one-to-one relationship between these. After exposing a number of legacy systems on the ESB, it will be necessary to orchestrate these services into a new service that reflects the business task. WS-BPEL provides a powerful way to achieve this level of orchestration. BPEL can easily deal with the correlation necessary to combine multiple asynchronous message exchanges into a single service. It also provides a reliable environment for process execution, thus ensuring that the reliability offered at the transport layer isn't lost due to problems in the application layer.

The Issues Enterprises Need To Address
The initial reaction of any skilled development team would be to say that it can satisfy any of the requirements above without having to use an ESB. That may be true: however, it may be a case of reinventing the wheel and while a development team is focused on that, it may not be addressing the real issues. This section looks at some of the non-technical issues that can cause problems during the first SOA projects.

Training Existing Staff
Seldom is an SOA built on a green field site. The need to reuse existing assets inevitably means that most real-world SOA deployments include a number of legacy applications. While SOA promises simplicity, integrating a number of legacy systems into an SOA can prove to be a complex task. This requires expertise in both SOA and the legacy systems. In short, this means that most of an SOA will be built by existing IT staff. Therefore, they will need to be trained in the use of the Web Services technologies required by an SOA.

Driving the Adoption of Shared Message Formats
As part of adopting an SOA, many organizations will need someone who is responsible for maintaining XML Schemas. SOAs are deployed incrementally. Many initial projects struggle to produce data structures, which are reusable. Reuse of XML message formats is an important milestone in the adoption of SOA. Failure to adopt common data structures will result in the need for transformations every time a service is reused. Many standard XML Schemas exist, however it's not always possible to find an appropriate standard, so each organization will need to create XML Schemas for their own use.

Where two or more teams are involved in an integration project, there's an additional upfront cost involved in adopting a common set of XML Schemas. Often, the greatest difficulty is in coordinating changes to these schemas in a timely fashion. Once these XML Schemas are in use, the ability to modify them in a backwards-compatible way becomes important. Designing these XML Schemas requires a good understanding of XML Schema itself and also the ability to balance the requirements of several teams.

Reuse of entire message formats is unlikely to be possible during initial projects, so the goal should be to create a number of element definitions that can be reused in a variety of message formats. This will ease integration and cut down on the need for data transformations.

Issues such as versioning should be planned from the start. At a basic level, this can take the form of ensuring that each message on the ESB contains version information. The selection and implementation of an appropriate versioning strategy is another responsibility of the person who maintains the XML Schemas.

Moving the Boundaries of Division of Labor
Any integration project involving a number of development teams will result in some level of negotiation on who does the work. Traditional division of labor can work strongly against the goals of an SOA. A successful SOA requires the creation of services that will be reused. In a traditional integration project, there's always the temptation to do the minimum amount of work to make the service accessible. This pushes some effort onto the client; perhaps an original text based message format will be used. SOA moves this work from the client to the service. The service must be created in such a way that's as convenient as possible for clients to access. Organizational changes may be required to ensure that the extra effort involved is acknowledged and rewarded.

Coordinating Team Activity
The task of coordinating multiple development teams shouldn't be underestimated, especially when they're working together for the first time. The touch points between the teams such as transports, protocols, and message formats are often areas that cause difficulties. Changes to the various touch points have to be planned and coordinated to prevent a continual dripfeed of modifications into the system.

Testing
Testing in an SOA project has to be approached in a very deliberate manner. Integration projects involving a number of applications are very sensitive to changes in any one of those applications. For example, late delivery of an application used as part of an orchestration can delay the start of testing for the overall project. Testing strategies need to plan for this eventuality. Testing needs to start early and be done in parallel with development. Stub versions of each service should be developed to facilitate isolated testing. This will limit the probability that delays relating to a single service will delay the overall project.

Conclusion
Many of the technical issues solved by the ESB may appear at first glance to be straightforward. However, the cost of delivering quality of service features such as reliability, security, and performance shouldn't be underestimated. Don't fall into the trap of attempting to implement these from the ground up. As many others are now doing, use an ESB to deliver these features. The true value of an ESB is in its ability to free up your IT resources to concentrate on the higher-level issues associated with rolling out a SOA.