Microservices Expo Authors: Liz McMillan, Pat Romanski, Elizabeth White, Stackify Blog, Andreas Grabner

Related Topics: Microservices Expo

Microservices Expo: Article

Toward SOA with ESB

A real-world view from design to implementation

Enterprises have become increasingly sensitive to the need to become sense-and-response- enabled. That is, enterprises perceive a strategic need to be able to both respond faster and more effectively to, and initiate changes in their environment.

On-demand capabilities require extensive flexibility and adaptability, which in turn require a highly integrated, zero-latency enterprise with respect to both IT systems and business and IT integration within and across organizational boundaries. Collectively, these requirements stipulate a new approach to both business and IT design and service-oriented architecture methodologies and technologies to address these issues.

From a design and architectural perspective, SOA, Web services, and associated technologies, such as BPEL, have been accepted as integral components to solve the integration of distributed systems. Not surprisingly, we see the emergence of a new generation of middleware: not integration glue, but core enterprise applications that help realize the architecture, runtime, and tooling requirements of a flexible, adaptive-on-demand enterprise. Specifically, we see the (new generation) enterprise services bus (ESB) as a salient, network-centric integration core to realize SOA.

This article looks at the design and implementation requirements of an ESB in the context of the SOA enablement of a retail client. Due to the scope of this article, a host of important (Web) services management issues are not presented. Pertinent information is available from the authors.

The Business Case: Faster, Cheaper, Better Across All Channels
Our client approached us with the problem of improving various business and operational aspects of their concern. The client is in the notoriously competitive retail business, managing some 600 stores across the U.S., a 24,000-item inventory, 20,000 employees, and $2 billion annual revenue. Moreover, our client already set significant growth targets, i.e. approximately 100%, across all major business drivers, such as store presence, per- square-foot store revenue, and profitability. These goals were to be met over a relatively short period of time (see Figure 1).

A client-specific analysis based on IBM's retail industry business model, framed the business analysis for the desired on-demand transformation. A key finding associated with our client's strategic plans concerned the improved flexibility and adaptability around the design, implementation, and management of business processes, not only within its organizational boundaries but also across the entire supply chain (see Figure 2).

After assessing both business and IT operations in the context of the overall strategic goals, it became apparent that a fundamental change was required. Specifically, the incumbent IT infrastructure was too distanced from the business operations and IT itself needed to be much more integrated to be able to respond in a timely and effective manner. Also, incumbent, complex "bundled" business processes needed to be unbundled to accommodate the desired flexibility and an adaptive, cost-effective integrated supply chain. In combination, these requirements supported an ESB-enabled SOA. Figure 3 summarizes the key findings, mapping the client's strategy into services-driven business priorities.

After reviewing several alternative solution approaches, the client and IBM team jointly decided to utilize an SOA and to embark on an ESB-enabled, Web services based strategy. The SOA-based design, architecture, and implementation roadmap were developed around an ESB (see Figure 4).

It is critical to appreciate the interrelatedness of business and IT analysis when approaching and designing aSOA-based solution. SOA purports to close the business-IT gap; hence, both business and IT requirements need to be analyzed in a recursive manner. This two-pronged approach aligns the solution to the salient problems and directly impacts on the identification, (re-)definition, and implementation of business services. This, in turn, affects legacy, ISV and custom application development, especially in light of an ESB-based solution.

It also needs to be recognized that the SOA design, architecture, and roadmap were put in place with an enterprise point of view. Yet, the client's strategic priorities emphasized the timely transformation of particular business, and associated IT, functions, thereby taking advantage of SOA's intrinsic incrementalism.

SOA and ESB Background
Fundamentally, an SOA is a framework for describing, publishing, and discovering services, where services are viewed in the context of business process descriptions. More specifically, SOA constitutes a logical construct that is realized as a new logical services layer and subsequently enabled by the ESB (see Figure 5).

In the context of an SOA, processes orchestrate one or more services to fulfill the appropriate business functions and the services, in turn, are realized through components.

Web services are instrumental in enabling SOAs in an interoperable manner. XML, coupled with a set of Web services - e.g., WS-* specifications - allow the implementation of broadly sharable, loosely coupled, and reusable services within an SOA framework. The orchestration, choreography, and management of services fulfilling business processes, especially in the context of an enterprise solution, require the appropriate nervous system, the ESB has been deemed necessary to perform that function.

The IBM ESB is J2EE-based and an essential element of an SOA realization. It supports a highly distributed services model comprised of service providers and service requesters. Moreover, it enables standards-based integration between loosely coupled applications and services within and across SOAs - an important consideration as many, if not most, enterprises adopt SOA in an incremental fashion and with an eye toward significant reuse of existing systems.

Overall, an ESB purports to enable application integration across different platforms, programming models, and messaging standards, and underlies the more extensive business integration models of business process and partner collaboration. The essential infrastructure services that an ESB needs to provide include transport, routing, quality of service, mediation, event management, de/composition, system management, security, data protocols, and policy management (see Figure 6).

An important aspect of an ESB concerns its support of Web services and associated technologies, such as BPEL. The Web services support in the IBM ESB uses relevant industry interoperability standards and supports the WS-I profile(s). Some of the published but not yet standardized WS-* specifications are supported as well. The actual access to the ESB from the Java platform uses the relevant standards, such as JMS, JAX-RPC, Web Services for J2EE (JSR 109), and JAXR.

From Design to Implementation
SOA design and implementation is an intrinsically simple endeavor - if it was occurring in a legacy-free vacuum. Most enterprises, however, operate in a context marked by incumbent business and (IT) systems. Moreover, most enterprises have neither the time nor the appetite to develop custom applications, or components, unless necessary and instead prefer to take advantage of packaged ISV solutions. In combination, these realities can amount to a significant set of constraints toward a pure play SOA, making it even more important upfront to put in place a solid SOA strategy.

A concept regarding the use of an ESB is that all services should expose a common data model. Incumbent systems, e.g., legacy or ISV applications, generally have different representations of data and hence a business process needs to choreograph the data communication across disparate systems, requiring costly transformations.

Our ESB view is to define and drive a common, standards-based data model. This enables the sustainable, long-term management of the enterprise's information model, sharable across disparate systems, and alleviates extensive, max.n2, transformations. In the case of our retail client, the ARTS (Association for Retail Technology Standards) data model and IXRetail (International XML Retail) XML Schemas are used as the common data model.

Overall, a successful ESB-based SOA implementation requires a considerable degree of analysis, taking into consideration both business and IT systems. After all, an SOA-enabling ESB is an a priori enterprise core application facilitating the integration of highly distributed, loosely coupled services, which is vastly different from yesteryear's ad hoc integration glue approach.

Requirements Analysis
In the case of our client project, as is true in most client scenarios, the incumbent environment, especially an IT environment, functioned as a baseline. A business (process) and IT environment assessment is a prerequisite for an ESB-based SOA transformation (see Figure 7).

These assessments and analyses are a nontrivial, fact-finding mission and a critical prerequisite to an ESB-based on-demand transformation. Since we are moving toward a highly distributed system "constrained" by existing systems and the well-founded desire of maximum [application] reuse, we also found it quite practical to manage the many "moving pieces" via a comprehensive, functional proof-of-concept (PoC) implementation. Hence, key service interface definitions and implementations were one of the first outcomes of this assessment and tooling quickly became an integral part of the early project phases, allowing maximum reuse for subsequent phases.

Operating Environment Reference Model
Another important aspect of the assessment and design phases concerns the choice of an operating environment reference model. Over the course of the client project, it was decided to utilize IBM's on-demand Operating Environment (odOE). The IBM odOE is based on the concepts of an SOA and utilizes an ESB as its integration core (see Figure 8). The odOE's inherent support for SOA drives the views of every application or resource as a service implementing a specific, identifiable set of [business] functions. In addition to the business functions it may also implement management interfaces to participate in the broader configuration, operation, and monitoring of the environment, providing a complete, end-to-end life-cycle model.

Determination of Reusable Services
An inherent aspect of the SOA strategy governing implementation concerns the identification of suitable business services. Based on the analysis and reference model outlined above, three distinct sets of business services were identified and formalized against the IBM odOE (see Figure 9): service work order (SWO), point of sales (POS), and shared business services.

Although it goes beyond the scope of this article to review each of the service categories in detail, we want to review the Catalog Services, part of the Shared Business Services portfolio, to illustrate the deterministic attributes of what actually constitutes a suitable business.

The purpose of the Catalog Service is to make available the complete product catalog in a timely and accurate fashion to both human and machine, i.e., A2A interfaces. The catalog is used by many of our client's applications, e.g., retail POS, service work order, etc., and constitutes a strategic competence of the integrated supply chain. Moreover, the catalog is governed by a shared set of functionalities, such as adding, removing, pricing, etc. of catalog items. Aside from the multiple access, catalog service must be able to be composed with other services in order to enable complete business processes.

In this SOA model, the Catalog Service connects through the IBM ESB via many external systems to form a "virtual catalog." The backend systems supplying the components of the Catalog Service have many, often idiosyncratic, protocols, access mechanisms, and data formats, which needed to be abstracted. For a logical representation of the ESB-based services implementation see Figure 10.

Client-Specific IBM ESB Implementation
A variety of technologies can be used to support the implementation and exposure of Web services onto the ESB, such as IBM MQ (publish/subscribe), adapters (JCA, IBM WBI Adapters, etc.), data transformers (XSLT), and database access (JDBC). This ESB construct enables the legacy system to be exposed in a standards-based manner and provides the infrastructure needed to choreograph services into business processes. A selector framework allows for the service client, or consumer, to be spared from worrying about the location of the service implementation.

The ensuing integration layer represents a logical ESB (see Figure 11), which spans both the client's retail stores and enterprise IT systems. A key feature of SOA is that the physical location of each service is hidden from the applications and can be changed at any time, without application changes. Our implementation facilitates this critical SOA feature.

The services that it provides are delivered to calling applications as proxy Java classes that are included in the application's runtime environment. The application makes simple Java calls to these proxy objects, exchanging input and output parameters in the form of simple data-transfer objects. The services themselves are implemented through a series of adapters that connect to the requisite legacy, or external system or data source, that the calling applications need to access.

Our client's environment is comprised of multiple legacy applications that run in the retail stores today, e.g., inventory, commercial sales, etc. While the client has the long-term goal of replacing or porting these systems, in the near future these applications need to remain in place and be wrapped as services to simplify their eventual conversion. The rationale for retaining these systems is, in part, based on our client's time-to-market goals, which could not be met if these systems had to be revamped.

IBM's WBI Server Message Broker is deployed on the client's enterprise systems and facilitates the transformation of all data that passes between the stores and the enterprise legacy systems, such as pricing. In order to provide the necessary system resilience, critical data is passed between the stores and enterprise IT systems, using IBM WebSphere MQ to provide assured delivery.

Figure 12 illustrates in more detail the n-tier enterprise ESB solution, where service requests are processed as follows:

  • Requests are initiated by client applications.
  • A selector framework routes the request to the appropriate service handler.
  • The service handler:
    - Processes the request, invoking legacy applications and third-party systems where appropriate,
    - Performs any data re-mapping necessary to translate between the service request and the legacy and third-party application formats, and
    - Maps the responses from the legacy and third-party applications to the appropriate format for the service response.
  • Synchronous responses are returned to the calling application.
  • In the event of problems, a simple consistent set of error messages may be returned to the calling application.
All client access into the integration layer is through WSDL-described Web service requests and all services implemented on the integration layer are exposed as Web services. Moreover, legacy and ISV systems are wrapped with JCA and front-ended with stateless session beans, which themselves are exposed as Web services. Proxies are generated to provide the client applications with a simple access point to these services.

Business process choreography is implemented to orchestrate the execution of services based on exposed business rules. Activities within the process choreography are implemented as stateless session beans. Common infrastructure subsystems provide common services to business processes within the integration layer, which include logging and error handling, systems management, security, and configuration management.

Specific considerations must be given to ISV applications. Most ISV applications to date lack the degree of componentization required to expose, and possibly consume, services to truly instantiate an SOA. Although the IBM ESB has the ability to integrate across heterogeneous IT systems, thereby mitigating some of these constraints, the ability to easily implement end-to-end business process capabilities is hampered.

In our retail case, many of the identified shared services already existed in some (functional) form in an ISV application. A fundamental rule of an ESB is to expose these services via Web services on the bus using a common data model. Therefore, ISV packages are exposed via a new Web service based on the common data model. The implementation of the service accesses the ISV package in its native manner and transforms the output to the common data model.

Increasingly, we are finding that most third-party ISVs are very interested and highly motivated in componentizing their applications, often utilizing the IBM's odOE reference model. This architectural transition invariably improves the ISV's ability to integrate with their customers' legacy systems while simultaneously reducing their maintenance requirements. Yet today's packaged applications don't foster the required degree of componentization and a certain degree of art, i.e., project-based, is required to compensate for this shortcoming.

This article provided a detailed overview of the design, architecture, and implementation requirements to implement an ESB-based SOA for a retail client. The overall design and architecture has resulted in a demonstrable implementation approach that exploits SOA to transform existing IT systems to satisfy imminent business needs. The ensuing solution creates much of the flexibility and adaptability required to support and enable business strategy by effectively bridging the business-IT gap. Moreover, the IBM ESB-enabled SOA implementation described above has led to a significant and sustainable reduction in systems and process complexity by representing services via standard interface descriptions.

We want to thank the project team for their outstanding contribution to generate and realize innovation in what constitutes largely uncharted territory; also, Jason Weisser, VP, IBM SWG EI; George Galambos, IBM Fellow; Manish Modh, Senior Solutions Engineer, IBM SWG EI; and Marc Fiamante, Executive IT Architect, IBM SWG EI, for their invaluable comments and support.


  • Channabasavaiah, Kishore, et al (2003). "Migrating to a service-oriented architecture, Part 1," IBM developerWorks, www-106.ibm.com/developerworks/webservices/library/ws-migratesoa and www-106.ibm.com/developerworks/webservices/library/ws-migratesoa2
  • Robinson, Rick. (2004). "Understand Enterprise Service Bus Scenarios and Solutions in Service-Oriented Architecture, Parts 1, 2, and 3," IBM developerWorks. www-106.ibm.com/developerworks/webservices/library/ws-esbscen, www-106.ibm.com/developerworks/webservices/library/ws-esbscen2.html, and www-106.ibm.com/developerworks/webservices/library/ws-esbscen3
  • Endrei, Mark, et al. (2004). "Patterns: Service-Oriented Architecture and Web Services." IBM Redbook, SG24-6303-00
  • More Stories By Bernhard Borges

    Bernhard Borges is a technical executive and solutions architect in the IBM Software Group's Enterprise Integration group. He specializes in distributed, open, interoperable, and portable systems, especially in the context of SOA, ESB, and web services. Bernhard ia an IBM Distinguished Engineer and a member of IBM's Academy of Technology. He is also a member of the IBM Software Architecture Board and several workgroups, IBM's SOA and Web Services Technical council, and an extended member of IBM's BCS SOA-Web Services Center of Excellence.

    More Stories By Ed Kahan

    Ed Kahan is an executive IT architect with IBM Software Group, Enterprise Integration, and managing solution architect of its North American Architecture group. He is an IBM Distinguished Engineer, Certified Consultant, and member of the IBM Academy of Technology. His current responsibilities include design and development of Web services, service-oriented architectures, and enterprise integration products and solutions for IBM's clients. He has extensive knowledge of design and implementation of complext systems in several industries. Ed holds a degree in mechanical engineering.

    Comments (0)

    Share your thoughts on this story.

    Add your comment
    You must be signed in to add a comment. Sign-in | Register

    In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.

    Microservices Articles
    Is advanced scheduling in Kubernetes achievable?Yes, however, how do you properly accommodate every real-life scenario that a Kubernetes user might encounter? How do you leverage advanced scheduling techniques to shape and describe each scenario in easy-to-use rules and configurations? In his session at @DevOpsSummit at 21st Cloud Expo, Oleg Chunikhin, CTO at Kublr, answered these questions and demonstrated techniques for implementing advanced scheduling. For example, using spot instances and co...
    Skeuomorphism usually means retaining existing design cues in something new that doesn’t actually need them. However, the concept of skeuomorphism can be thought of as relating more broadly to applying existing patterns to new technologies that, in fact, cry out for new approaches. In his session at DevOps Summit, Gordon Haff, Senior Cloud Strategy Marketing and Evangelism Manager at Red Hat, discussed why containers should be paired with new architectural practices such as microservices rathe...
    In his session at 20th Cloud Expo, Mike Johnston, an infrastructure engineer at Supergiant.io, will discuss how to use Kubernetes to setup a SaaS infrastructure for your business. Mike Johnston is an infrastructure engineer at Supergiant.io with over 12 years of experience designing, deploying, and maintaining server and workstation infrastructure at all scales. He has experience with brick and mortar data centers as well as cloud providers like Digital Ocean, Amazon Web Services, and Rackspace....
    SYS-CON Events announced today the Kubernetes and Google Container Engine Workshop, being held November 3, 2016, in conjunction with @DevOpsSummit at 19th Cloud Expo at the Santa Clara Convention Center in Santa Clara, CA. This workshop led by Sebastian Scheele introduces participants to Kubernetes and Google Container Engine (GKE). Through a combination of instructor-led presentations, demonstrations, and hands-on labs, students learn the key concepts and practices for deploying and maintainin...
    Docker is sweeping across startups and enterprises alike, changing the way we build and ship applications. It's the most prominent and widely known software container platform, and it's particularly useful for eliminating common challenges when collaborating on code (like the "it works on my machine" phenomenon that most devs know all too well). With Docker, you can run and manage apps side-by-side - in isolated containers - resulting in better compute density. It's something that many developer...
    Modern software design has fundamentally changed how we manage applications, causing many to turn to containers as the new virtual machine for resource management. As container adoption grows beyond stateless applications to stateful workloads, the need for persistent storage is foundational - something customers routinely cite as a top pain point. In his session at @DevOpsSummit at 21st Cloud Expo, Bill Borsari, Head of Systems Engineering at Datera, explored how organizations can reap the bene...
    As software becomes more and more complex, we, as software developers, have been splitting up our code into smaller and smaller components. This is also true for the environment in which we run our code: going from bare metal, to VMs to the modern-day Cloud Native world of containers, schedulers and micro services. While we have figured out how to run containerized applications in the cloud using schedulers, we've yet to come up with a good solution to bridge the gap between getting your contain...
    In his general session at 19th Cloud Expo, Manish Dixit, VP of Product and Engineering at Dice, discussed how Dice leverages data insights and tools to help both tech professionals and recruiters better understand how skills relate to each other and which skills are in high demand using interactive visualizations and salary indicator tools to maximize earning potential. Manish Dixit is VP of Product and Engineering at Dice. As the leader of the Product, Engineering and Data Sciences team at D...
    DevOps is speeding towards the IT world like a freight train and the hype around it is deafening. There is no reason to be afraid of this change as it is the natural reaction to the agile movement that revolutionized development just a few years ago. By definition, DevOps is the natural alignment of IT performance to business profitability. The relevance of this has yet to be quantified but it has been suggested that the route to the CEO’s chair will come from the IT leaders that successfully ma...
    Skeuomorphism usually means retaining existing design cues in something new that doesn’t actually need them. However, the concept of skeuomorphism can be thought of as relating more broadly to applying existing patterns to new technologies that, in fact, cry out for new approaches. In his session at DevOps Summit, Gordon Haff, Senior Cloud Strategy Marketing and Evangelism Manager at Red Hat, will discuss why containers should be paired with new architectural practices such as microservices ra...