Barriers to Integration Success & the Original SOA
In the mid-90s, commercial integration packages were used to build large cross-system business processes for the first time. However, the limitations of first-generation technology became a serious barrier to success:

1.  Lack of scalability. Integrating every system into every other system results in complexity that grows as the square of the number of endpoints. This clearly won't scale. The introduction of hub-and-bus models reduced these connections, with all of the data flowing through a central hub and repository.

2.  Reliance on low-level APIs and adapters. When all of the data for a given system flowed through a central point, designers could think about reducing the amount of work required to maintain the endpoints. Even moderately complex systems have dozens of endpoints and the bulk of IT administration effort involves maintaining connectivity as the endpoints change. For example, SAP comes out with a new version of software, Oracle buys PeopleSoft, the mainframe payments system moves to a Unix server or vice versa.

Early integration systems were usually tied directly to low-level APIs at the endpoints using adapters. Unfortunately, standards were few. In 1995, even ODBC didn't always work consistently. This resulted in large numbers of adapters having to be co-located with the target systems - at least one per application type and often more - to deal with multiple versions of an ERP system. Every time the endpoint changed, that interface had to be reworked. Worse, all of the business logic behind it was often tightly tied to the adapter, and it also had to be rewritten or ported. This led to the idea of a higher-level abstraction than a programming API.

3.  Proliferation of abstract services. The concept of using abstract services, rather than low-level APIs, was an improvement, but it, too, has a serious weakness. The number of interfaces that are services isn't really any smaller than the number of physical connections to end systems. By creating a service, you get a better abstraction, but there are still too many of them. Worse, it's often the case that a well-designed service touches more than one underlying physical application. This creates the potential for overlapping services, data duplication, and data inconsistency. Again, the design doesn't scale.

In response, IT professionals embraced SOA. The central concepts of the original SOA were simplified connectivity, service abstraction, and data mastering. Unless all of these ideas are present in your SOA design, you won't derive maximum benefit from the resulting system. Conversely, a true SOA not only insulates you from underlying change, but lets you reuse the services.

The Key to SOA Implementation: Separation for Success
An SOA can be implemented in many different ways, using a variety of tools and technologies, including Web Services, middleware, frameworks, and integration platforms. Insight into an overriding SOA trend, however, may help to minimize your implementation costs and efforts. Integration tools were originally undifferentiated; there was one tool for every connection to every system. As the SOA concept evolved, the architecture differentiated into high-level categories with specific structures and functions (see Figure 1).

1.  Business process: This SOA layer makes use of services and embodies those business processes that span multiple services such as meta-business processes, the basis of BPM and workflow.

2.  Directory services: Common services such as entitlements and service directories plug into the service layer to insulate the overlying business process from changes.

3.  Existing applications: The underlying programs that actually do the work.

4.  Data synchronization: These integration components are used for building and maintaining parent/child data relationships.

These components have different functions and very different structures. They are often best constructed using different technologies.

Selecting the Right SOA Technologies
The complexity of an SOA project can be greatly reduced by choosing appropriate tools. You probably wouldn't build your purchase order system on top of LDAP. You also shouldn't use a BPM tool for data integration.

Good BPM tools are very expressive and abstract. They think in terms of end-to-end business collaborations that often involve multiple individual transactions done by humans. BPM tools are ideal for building high-end workflows containing complex business semantics and implementing best practices across multiple groups of users. They are also complex to learn, difficult to implement, and costly to maintain. If you really need what they have to offer, however, they're worth it.

In situations without complex manual workflows, an integration appliance is often more cost-effective, quicker to implement, and easier to maintain.

For example, a major global CPG maker is enabling SOA and accelerating SAP adoption using integration appliances. The company's first implementation of appliances was to replace a legacy middleware infrastructure in the synchronization of purchasing, foreign exchange, and retail data between ERP and other systems in approximately 20 countries. The results of this approach are quite impressive: integration appliances yielded an ROI payback in fewer months, reduced total cost of ownership by 74%, and delivered projects in 30 days or less. As the manufacturer migrated to an SAP-centric application footprint across its 170 subsidiaries in 50 different countries, it adopted a SOA-based enterprise integration framework - SAP instances are integrated with each other using SAP's own middleware platform, integration appliances are used to extend SAP to other applications worldwide, and a SOA framework has been implemented to bridge the SAP middleware with integration appliances (see Figure 2).

Another leading online marketing services provider is using integration appliances to implement a SOA solution for business-to-business communication. The company has adopted salesforce.com as its CRM solution and uses Microsoft Dynamics for back-office ERP functions. It uses integration appliances to expose order processing services to its vendors; these services are standards-based abstractions of discrete ERP and CRM operations like order creation and order visibility. Vendors communicate with the integration appliances to access these services in a real-time manner; the appliance in turn handles the native connectivity to the back-end applications as well as the data mapping, workflow, and error-processing logic (see Figure 3).

Using Integration Appliances in SOA
Even the best planned SOA project has its complexities and requires a broad skill set to implement and maintain. The thinking that originally led to SOA was focused on reducing that complexity. Complexity reduction pays dividends each time a change is made anywhere in a SOA, as the number of moving parts affected by the change is directly related to cost, time, and difficulty of implementation.