Microservices Expo Authors: Elizabeth White, Liz McMillan, Zakia Bouachraoui, Pat Romanski, Yeshim Deniz

Related Topics: Microservices Expo

Microservices Expo: Article

EAI Industry Health Check

EAI Industry Health Check

Demand for business integration continues to intensify across a broad range of industries. Yet disappointing returns from enterprise application integration (EAI) projects and increased competition are conspiring to squeeze pure-play integration vendors from all directions. In this article, we'll explore some of the current challenges faced by enterprise integration vendors.

The Integration Imperative
For most industries - manufacturing, retail, financial services, healthcare, government, aerospace and defense, natural resources - visibility and access to information flowing through integrated business processes has never been more important. Integrated business processes include activities involving multiple systems and/or people, often crossing organizational boundaries such as departments, divisions or enterprises. Examples include order-to-cash, trade-to-settlement, claims processing, service provisioning, logistics management, and customer self-service.

Enterprises continually strive for agility through improved business process management, enhanced trading partner connectivity, supply chain optimization, and collaboration in design and manufacturing. Many enterprises want to adopt common processes for business functions yet support local variations in certain activities. Removing latency (non-value-added time) in business processes is a major driver for straight-through processing in financial services. Increasing information portability and security is the primary goal of the HIPAA mandate in health care. Increased collaboration between independent entities is key for U.S. homeland security and for collaborative design initiatives in aerospace and manufacturing. The common thread across these efforts is what CIO magazine calls "the integration imperative."

From an IT standpoint, these initiatives all require increased intra-enterprise integration between front- and back-office applications, databases, legacy systems, Web portals, and other Web applications. Inter-enterprise integration is also often involved among trading partners using XML-based protocols, EDI, or Web services. To facilitate rapid integration, differences between operating platforms, communication protocols, data representations, and application interface styles must be minimized. Enterprise integration must be distributed, scalable, reliable, manageable, and secure. Often it must be "transactional" - recoverable in case of error. (Cross-application transaction processing is conceptually similar to two-phase commit. However, rather than directly rolling back failed transactions, "compensating" activities are invoked to effectively undo the original, failed transaction.)

To remain competitive, most medium-to-large enterprises now recognize that application integration must become a core competency. Sophisticated enterprise architectures incorporate integration services as a required component. Core integration services include messaging and message routing, adapters for connecting to applications and other data sources, data transformation, business process automation, trading partner management, metadata management, and system management and monitoring. Pure-play EAI vendors such as SeeBeyond, webMethods, TIBCO, Vitria, and many others have developed technology specifically to address the integration imperative.

Business Integration Challenges
Despite, or perhaps because of, the hype surrounding EAI two or three years ago - similar to the hype surrounding Web services today - return on investment (ROI) for some EAI projects has been disappointing. Contributing factors include lack of business sponsorship, unrealistic or poorly managed expectations, weak program governance, product instability, and inappropriate use of the technology (which happens more often than you might think). Two key factors include underestimating the complexity of the task and lack of an architectural approach for using sophisticated EAI technology.

Underestimating Complexity
Caught up in their own hype, EAI vendors oversold the ease with which EAI technology enables business integration. EAI marketing presentations would lead customers to believe they can achieve complex integration in a few short weeks. Customers should realize that these estimates typically only include the time to actually configure the EAI software; they conveniently ignore the time needed to determine what should be configured and why. Analyzing specific integration requirements, designing a reusable solution, and hardening it for a production environment are fundamental activities for any industrial-strength solution using EAI technology. These will substantially increase a project development schedule beyond original EAI vendor numbers.

Customers undertaking EAI projects often underestimate the organizational commitment required to achieve business integration, which is not the same as application interfacing. Business integration is about improving business processes to ensure that information flows seamlessly and automatically throughout the process. Application interfacing is about physically enabling applications to send or receive data, with little regard for what happens beyond the application's boundaries (i.e., the integration context). While often considered a tedious chore by developers and project managers, application interfacing is critical to but not sufficient for effective business integration.

Actively involving both business process owners and IT integration experts in planning and design activities is a critical success factor for business integration. With active participation from the business community, we've been able to identify $3 in process-related improvements from EAI technology for every $1 in cost reduction. Thus, engaging the business and IT communities early, not only bolsters the business case needed for executive sponsorship, it also helps build the excitement and organizational alignment needed for effective business integration.

You would never hand someone a box of tools and expect them to build a skyscraper. Skilled architects must design the building, taking into account functional requirements and making trade-offs based on design criteria such as building codes, materials, and project budget. Only then can experienced tradesmen from many disciplines construct the building. They use best practices, experience, and state-of-the-art tools to ensure that it is secure, scalable, and reliable. The same is true for EAI projects. In other words, business integration doesn't magically happen just because you've installed the EAI toolbox.

Ad Hoc vs Architecture
Another common mistake is to take an ad hoc rather than architectural approach to using EAI technology. Most people now accept that point-to-point integration is expensive, yet it is still easy to build such solutions using EAI tools. Remember, none of these tools will prevent you from creating a poor implementation! Like the skilled tradesmen mentioned earlier, successful business integration projects require experienced practitioners using a proven methodology and best practice techniques.

For example, CSC developed a methodology that takes a disciplined and holistic approach to EAI implementation, with a strong focus on analysis and optimization of integrated business processes and on the architectural issues surrounding use of EAI technology. We encourage business and application technical experts to jointly develop business integration requirements such as:

  • What are the activities, events, and information flows for each integrated business process? What are the pre-conditions or post-conditions (if any)? Is this process similar to any others in the enterprise?
  • What is the nature of the information passing through each activity? How do other processes or applications use this information?
  • How are process exceptions handled? Are compensating activities required?
  • Are there any service levels or timing constraints that must be maintained?

    It is also important during this phase to address areas not fully covered by EAI vendor products or methodologies such as configuration management, change control, organizational impact, security, system integration testing, and production system deployment. Addressing such requirements early in the architecture phase helps identify integration patterns and uncover opportunities to create reusable integration artifacts (business process models, integration entities (data), adapters, transformations, etc.). This is crucial for delivering on a fundamental value proposition of EAI - reuse - that is often not realized in practice.

    EAI projects that minimize or ignore the architectural approach do so at their peril. One client launched multiple independent integration projects using a shared EAI infrastructure, with little coordination between the projects. In the short term, this strategy worked. However, around the time that the fifth or sixth project was deployed, the client's EAI IT infrastructure fell over! Upon investigation, they discovered that system resources were depleted, due in large part to each project having created redundant schema that consumed a large amount of memory (e.g., four almost-identical definitions for a purchase order). With proper architecture and project coordination, these could have been designed for shared use, dramatically reducing both the system load and the effort expended by each project.

    Another EAI project left configuration management until very late in the project. Much to their chagrin, the development team discovered, just prior to system integration test, that it was very difficult to migrate code between development, test, and production environments in an automated manner using their EAI tool. In an 11th hour scramble, they were forced to hand-migrate all the code in order to meet the project deadlines. Taking a holistic approach that includes configuration management as part of architecture could have avoided this embarrassing situation.

    Increasing Competition
    EAI vendors are continually challenged to recast their value propositions. More and more integration services are being standardized by the J2EE and .NET platforms, and industry consortia such as W3C. Emerging integration standards include JMS for messaging, JCA for application connectors, XSLT for data transformation, and multiple Web services standards for application service definition, invocation, and coordination. While these developing standards still lack the functionality currently available from leading pure-play EAI vendors, it's clear that many of the services offered by the pure-play integration vendors are rapidly becoming commodity items. Consequently, EAI vendors are being forced to move up the value chain. They now emphasize such capabilities as business process management (BPM), business activity monitoring (BAM), Web services integration, and increased support for vertical industry processes or protocols.

    Competitors are coming out of the woodwork. In terms of functionality, business process management vendors - including Fuegotech, Savvion, and Intalio - continue to push the envelope of process-centric business integration. BPM provides visibility and control over the work of people, systems, and partners. This goes beyond simply integrating applications and automating business processes; sophisticated BPM systems also support business process design, process simulation, process execution, process monitoring, and process analysis. In terms of price, a host of newer vendors such as Fiorano and Sonic Software now offer JMS-based integration suites at a fraction of the cost of established players. Additional competition comes from vendors in related disciplines such as extract-transform-load (ETL) and electronic data interchange (EDI), who are redefining their offerings as integration solutions.

    Most threatening are the industry "800 lb. gorillas," who are clearly taking aim at the business integration market. These offerings will appeal to those customers who prefer to stick with an industry leader even if a smaller, less proven company may have superior technology. IBM announced significant advancements to its WebSphere Business Integration portfolio that provides businesses with a broad set of functions to integrate, manage, and monitor business processes across an enterprise and with trading partners. BEA announced WebLogic Integration 8.1, which provides customers with a unified framework for business integration, simplified production and management, and a new extensible architecture for the rapid assembly and integration of applications, business processes, and trading partner communities. Microsoft's upcoming "Jupiter" technologies will include revolutionary business process management and monitoring capabilities; additional support for XML Web services standards, including Business Process Execution Language for Web Services (BPEL 4WS); and richer developer and information worker support through enhanced integration with Visual Studio .NET and Office.

    The leading application vendors are also joining the party. Having finally recognized that they are not the center of the universe, most have responded by adding integration brokers and/or pre-built business processes to their application suites. Some have partnered with EAI vendors to provide integration technology; others opted to build it themselves. SAP recently announced NetWeaver, the foundation of the SAP xApps and mySAP Business Suite solution, which offers a comprehensive integration and applications platform. PeopleSoft's Integration Broker offers integration using Web services, flat files, custom code, or a JMS-based connector to IBM MQSeries. Yantra's enterprise software for real-time coordination of the extended supply chain includes a business process-oriented integration platform and prebuilt processes for coordinating and controlling distributed orders and inventory across multiple business units, customers, suppliers and trading partners. Siebel has taken a slightly different tack by creating the Universal Application Network (UAN), which claims to transform application integration "from a complex and expensive technical challenge into the strategic ability to implement customer-facing business processes across and beyond the enterprise." (All the leading pure-play EAI vendors, as well as IBM and Microsoft, have announced support for UAN.)

    Economically, it was a tough year for the pure-play EAI software vendors. All have seen dramatic revenue reductions as EAI projects are delayed, scaled back, or cancelled outright. Several vendors are responding to the perception that EAI projects are too expensive by reducing license costs. At least one vendor now reports more revenue from software services than software licenses (an untenable situation for a software product vendor). Most pure-play EAI vendors have reduced their workforces and all are rapidly burning through their cash reserves; some have seen their stock market valuations essentially evaporate.

    Despite competing claims, the integration marketplace is still very much a horse race. Although there are half a dozen "usual suspects," none can yet really be considered the dominant player. Stiff competition means everyone must continue to spend significantly on marketing, sales, and engineering in order to win market share, while continuing to reduce costs. Now that the established industry players are joining the fray, the competition will only intensify.

    Going Forward
    Though EAI technology has been available for several years now, significant challenges still exist to successfully deploy it. To be fair, not all the problems are directly related to EAI technology itself; they have as much to do with effectively managing complex programs, managing change in large organizations, and nuts-and-bolts system integration expertise. As with any new technology, there have been some teething problems but the integration approach is essentially sound. While the road to business integration has been rocky, it would be a mistake to conclude that the future for EAI is bleak.

    Far from being a flash in the pan, it seems clear that EAI technology is following the historical path described by the Gartner Group's Technology Hype Curve (see Figure 1).


    Initially, EAI technology climbed the Peak of Inflated Expectations, overhyped by analysts and vendors alike as the next new thing. As we've discussed, challenges in early business integration deployments resulted in disappointing ROI for some projects. Others predicted that new technologies like Web Services would make EAI obsolete. (Personally, I believe Web services and EAI are complementary rather than competing technologies, as I discussed here last May [WSJ, Vol. 2, issue 5).) This encouraged doomsayers to push EAI into the Trough of Disillusionment. Yet increasing demand for intra- and interenterprise integration validates the merits of the EAI approach, as evidenced by efforts to standardize integration-related services and the race to embrace business integration by leading application and platform vendors. Today, many customers across a broad range of industries are steadily climbing the Slope of Enlightenment using EAI. They have more realistic expectations and the benefits of lessons learned on how (and how not) to implement this powerful technology. Thus, the future for EAI technology (if not for certain vendors) is still pretty bright. It should only be a matter of time until we reach the Plateau of Productivity, in which real-world benefits of business integration are broadly demonstrated and accepted.

  • More Stories By Carol Murphy

    Carol Murphy is a partner with CSC Consulting and the solution architect for its Enterprise Application Integration (EAI) practice. Carol educates clients on using EAI technology for business integration, including vendor selection, project planning & implementation best practices. Carol coordinates development of CSC Consulting eAI service offerings and is the primary author of the CSC Business Integration Practice Guide.

    Comments (1)

    Share your thoughts on this story.

    Add your comment
    You must be signed in to add a comment. Sign-in | Register

    In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.

    Microservices Articles
    In his general session at 19th Cloud Expo, Manish Dixit, VP of Product and Engineering at Dice, discussed how Dice leverages data insights and tools to help both tech professionals and recruiters better understand how skills relate to each other and which skills are in high demand using interactive visualizations and salary indicator tools to maximize earning potential. Manish Dixit is VP of Product and Engineering at Dice. As the leader of the Product, Engineering and Data Sciences team at D...
    The now mainstream platform changes stemming from the first Internet boom brought many changes but didn’t really change the basic relationship between servers and the applications running on them. In fact, that was sort of the point. In his session at 18th Cloud Expo, Gordon Haff, senior cloud strategy marketing and evangelism manager at Red Hat, will discuss how today’s workloads require a new model and a new platform for development and execution. The platform must handle a wide range of rec...
    When building large, cloud-based applications that operate at a high scale, it’s important to maintain a high availability and resilience to failures. In order to do that, you must be tolerant of failures, even in light of failures in other areas of your application. “Fly two mistakes high” is an old adage in the radio control airplane hobby. It means, fly high enough so that if you make a mistake, you can continue flying with room to still make mistakes. In his session at 18th Cloud Expo, Lee A...
    Lori MacVittie is a subject matter expert on emerging technology responsible for outbound evangelism across F5's entire product suite. MacVittie has extensive development and technical architecture experience in both high-tech and enterprise organizations, in addition to network and systems administration expertise. Prior to joining F5, MacVittie was an award-winning technology editor at Network Computing Magazine where she evaluated and tested application-focused technologies including app secu...
    Containers and Kubernetes allow for code portability across on-premise VMs, bare metal, or multiple cloud provider environments. Yet, despite this portability promise, developers may include configuration and application definitions that constrain or even eliminate application portability. In this session we'll describe best practices for "configuration as code" in a Kubernetes environment. We will demonstrate how a properly constructed containerized app can be deployed to both Amazon and Azure ...
    Modern software design has fundamentally changed how we manage applications, causing many to turn to containers as the new virtual machine for resource management. As container adoption grows beyond stateless applications to stateful workloads, the need for persistent storage is foundational - something customers routinely cite as a top pain point. In his session at @DevOpsSummit at 21st Cloud Expo, Bill Borsari, Head of Systems Engineering at Datera, explored how organizations can reap the bene...
    Using new techniques of information modeling, indexing, and processing, new cloud-based systems can support cloud-based workloads previously not possible for high-throughput insurance, banking, and case-based applications. In his session at 18th Cloud Expo, John Newton, CTO, Founder and Chairman of Alfresco, described how to scale cloud-based content management repositories to store, manage, and retrieve billions of documents and related information with fast and linear scalability. He addresse...
    SYS-CON Events announced today that DatacenterDynamics has been named “Media Sponsor” of SYS-CON's 18th International Cloud Expo, which will take place on June 7–9, 2016, at the Javits Center in New York City, NY. DatacenterDynamics is a brand of DCD Group, a global B2B media and publishing company that develops products to help senior professionals in the world's most ICT dependent organizations make risk-based infrastructure and capacity decisions.
    Discussions of cloud computing have evolved in recent years from a focus on specific types of cloud, to a world of hybrid cloud, and to a world dominated by the APIs that make today's multi-cloud environments and hybrid clouds possible. In this Power Panel at 17th Cloud Expo, moderated by Conference Chair Roger Strukhoff, panelists addressed the importance of customers being able to use the specific technologies they need, through environments and ecosystems that expose their APIs to make true ...
    In his keynote at 19th Cloud Expo, Sheng Liang, co-founder and CEO of Rancher Labs, discussed the technological advances and new business opportunities created by the rapid adoption of containers. With the success of Amazon Web Services (AWS) and various open source technologies used to build private clouds, cloud computing has become an essential component of IT strategy. However, users continue to face challenges in implementing clouds, as older technologies evolve and newer ones like Docker c...