Microservices Expo Authors: Liz McMillan, Pat Romanski, Carmen Gonzalez, Elizabeth White, Jason Bloomberg

Related Topics: Microservices Expo, Java IoT

Microservices Expo: Blog Post

Avoid Getting Lost on the (Enterprise Service) Bus

The risks of ESB-first SOA projects and how to mitigate them

So, you've been following ZapThink long enough to know that beginning a Service-Oriented Architecture (SOA) project by purchasing an Enterprise Service Bus (ESB) is starting at the wrong end of the initiative. Purchasing any technology, especially an ESB, at the beginning of an architecture project is a recipe for failure, you've been telling anyone who'll listen. But for whatever reason, your organization didn't pay attention to you, and now they've dropped a bundle on an ESB. Maybe your boss golfs with the vendor sales rep, or maybe the powers-that-be are listening to the wrong analyst firm, who knows. But in any case, you're stuck with that decision, and you're now expected to implement SOA with it.

Fortunately, while purchasing an ESB too early in a SOA project does substantially increase your risk of failure, all is not lost. After all, you're not alone; this mistake is one of the most prevalent SOA snafus in IT shops around the world today, and not all of those projects end up as failures. Many of today's ESBs are now mature products, and can be an important part of a fully functional SOA implementation. Understanding the risks that buying an ESB too early in a SOA initiative presents, and dealing with those risks proactively, can turn a bad situation around and get your SOA initiative back on the right track.

Understanding the Risks
Fundamentally, the problem with buying the ESB first is that you might fall into the trap of doing things the way the ESB would like you to do them, in light of the fact that many ESBs are in many ways traditional middleware under the covers. After all, if middleware solved all your problems, then you wouldn't be considering SOA in the first place -- and adding Service capabilities to your middleware doesn't change this fundamental fact.

In fact, the pitfalls that the ESB-first approach introduce fall into three broad categories:

  • Taking an overly integration-centric perspective of the project -- Most ESBs are generally really good at connecting things -- in other words, most ESBs are quite capable integration middleware solutions. The problem is, SOA isn't about connecting things, it's about building loosely-coupled Services the business can leverage to meet changing process needs. We want to get away from the "connecting things" approach to distributed computing, and instead move to a "composing Services" paradigm, where integration becomes a byproduct of composition.
  • The "middleware for your middleware" problem -- if it were practical (or even possible) to take a single piece of middleware and put it all by itself in the middle of your IT infrastructure, that would be one thing, but for most large (and many midsize) organizations, the vision of relying upon one piece of middleware to solve all integration problems is an unrealistic fantasy. In reality, organizations tend to have several different pieces of middleware, of different vintages and for different purposes. Introducing one or more ESBs into the mix means that now you have to integrate your ESBs with existing middleware as well as with each other, leading to the requirement of middleware for your middleware. Where will it ever end?
  • The "good money after bad" fallacy -- The "good money after bad" fallacy is actually much broader than IT. People would rather throw money at an approach that's already cost a bundle than to switch approaches to a less expensive, but more effective alternative. If you've been buying middleware from a vendor for years, and now they tell you that you need an ESB, you're likely to take that advice, even if an alternative is lower cost and lower risk, simply because you've already spent so much with that vendor.

ESB-First SOA Best Practices
Now that you've steered your bus past the pitfalls, let's see if we can point it in the right direction moving forward. The most important thing to remember is that your architecture should drive the technology, not the other way around. Remember that ESBs, like any mature solution, come with a boatload of features -- many of which may not be appropriate for your situation. It is often figuring out which features not to use rather than the capabilities you should actually use that is the key to being successful with a product like an ESB.

In particular, it is essential to take a process-driven approach to your infrastructure, instead of an integration-centric approach. Remember that building Service compositions that implement processes typically compose capabilities across multiple execution environments. Furthermore, those compositions are both dynamic and unpredictable -- the business process specialist in charge of the compositions may change them around long after you've deployed the Services. Governance becomes the key to managing that unpredictability, rather than pre-defined integrations.

As a result, you shouldn't rely upon any one execution environment for your Service implementations, or any one process management environment either, for that matter. ESBs can offer an effective, managed execution environment for some of your Service interfaces, but you rarely if ever want to rely upon any one runtime environment for all of your Services. In other words, you should balance the advantages of running your Services "on the bus" with the fact that SOA allows you to leverage heterogeneity both on and off the bus.

One essential point here is that SOA leverages interoperability more so than portability. In a virtual machine-based "write once, run anywhere" environment, whether Java or Microsoft Common Language Runtime (CLR)-based, distributed computing relies upon the portability of code. SOA, however, leverages the interoperability of the Service interfaces so that you don't need to move the underlying Service implementations around. As a result, running all your Services on the ESB can actually impede your SOA implementation, rather than support it.

So, if you shouldn't think of your ESB as either integration middleware or as a universal Service execution environment, then what role should your ESB play? The answer is a Service intermediary. Transformations and content-based routing are the essential capabilities a Service intermediary should deliver, in conjunction with robust security and management. Building the Business Service abstraction depends upon transformations and content-based routing, and fortunately, most ESBs offer these capabilities. So, only use the traditional messaging middleware capabilities of your ESB if you really need them, and only leverage the Service runtime your ESB provides when convenient, but configure your ESB as an intermediary to get full value out of it as part of your SOA infrastructure.

Not only does using an ESB as an intermediary enable you to architect the Business Service abstraction, it also resolves the "middleware for your middleware" problem, because intermediaries can intermediate between disparate integration technologies just as well as they can intermediate between Service providers and consumers. If you feel you need to use your ESB's message queuing technology, for example, just because it's there, however, then you won't get this benefit.

The ZapThink Take
Yes, you need security, governance, quality, and management, in addition to the transformation and content-based routing capabilities of Service intermediaries, in order to build an effective SOA infrastructure. But remember, ESBs aren't the be-all and end-all of SOA infrastructure -- many ESBs on the market include most of the above capabilities, but rarely if ever offer everything an organization requires. In fact, XML appliances are likely a better approach to security and policy enforcement, a registry/repository combined with a full-lifecycle SOA quality solution might serve as your design time and run time governance tools, while a robust SOA management solution might be a critical part of your infrastructure as well. In fact, many organizations leverage such products in conjunction with existing middleware to build out their SOA infrastructure without having to buy an ESB at all.

The bottom line is to always remember that the business drives the architecture, and the architecture drives the technology. Don't let the technology drive the architecture! SOA is particularly adept at abstracting existing technology, which can include recently purchased products in addition to your legacy environment. But knowing which of your existing capabilities to leverage -- and which to forego -- can make or break your SOA initiative.

Learn more at one of our upcoming Licensed ZapThink Architect Bootcamps or SOA & Cloud Governance courses.

More Stories By Jason Bloomberg

Jason Bloomberg is a leading IT industry analyst, Forbes contributor, keynote speaker, and globally recognized expert on multiple disruptive trends in enterprise technology and digital transformation. He is ranked #5 on Onalytica’s list of top Digital Transformation influencers for 2018 and #15 on Jax’s list of top DevOps influencers for 2017, the only person to appear on both lists.

As founder and president of Agile Digital Transformation analyst firm Intellyx, he advises, writes, and speaks on a diverse set of topics, including digital transformation, artificial intelligence, cloud computing, devops, big data/analytics, cybersecurity, blockchain/bitcoin/cryptocurrency, no-code/low-code platforms and tools, organizational transformation, internet of things, enterprise architecture, SD-WAN/SDX, mainframes, hybrid IT, and legacy transformation, among other topics.

Mr. Bloomberg’s articles in Forbes are often viewed by more than 100,000 readers. During his career, he has published over 1,200 articles (over 200 for Forbes alone), spoken at over 400 conferences and webinars, and he has been quoted in the press and blogosphere over 2,000 times.

Mr. Bloomberg is the author or coauthor of four books: The Agile Architecture Revolution (Wiley, 2013), Service Orient or Be Doomed! How Service Orientation Will Change Your Business (Wiley, 2006), XML and Web Services Unleashed (SAMS Publishing, 2002), and Web Page Scripting Techniques (Hayden Books, 1996). His next book, Agile Digital Transformation, is due within the next year.

At SOA-focused industry analyst firm ZapThink from 2001 to 2013, Mr. Bloomberg created and delivered the Licensed ZapThink Architect (LZA) Service-Oriented Architecture (SOA) course and associated credential, certifying over 1,700 professionals worldwide. He is one of the original Managing Partners of ZapThink LLC, which was acquired by Dovel Technologies in 2011.

Prior to ZapThink, Mr. Bloomberg built a diverse background in eBusiness technology management and industry analysis, including serving as a senior analyst in IDC’s eBusiness Advisory group, as well as holding eBusiness management positions at USWeb/CKS (later marchFIRST) and WaveBend Solutions (now Hitachi Consulting), and several software and web development positions.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.

Microservices Articles
Modern software design has fundamentally changed how we manage applications, causing many to turn to containers as the new virtual machine for resource management. As container adoption grows beyond stateless applications to stateful workloads, the need for persistent storage is foundational - something customers routinely cite as a top pain point. In his session at @DevOpsSummit at 21st Cloud Expo, Bill Borsari, Head of Systems Engineering at Datera, explored how organizations can reap the bene...
"NetApp's vision is how we help organizations manage data - delivering the right data in the right place, in the right time, to the people who need it, and doing it agnostic to what the platform is," explained Josh Atwell, Developer Advocate for NetApp, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
The Jevons Paradox suggests that when technological advances increase efficiency of a resource, it results in an overall increase in consumption. Writing on the increased use of coal as a result of technological improvements, 19th-century economist William Stanley Jevons found that these improvements led to the development of new ways to utilize coal. In his session at 19th Cloud Expo, Mark Thiele, Chief Strategy Officer for Apcera, compared the Jevons Paradox to modern-day enterprise IT, examin...
In his session at 20th Cloud Expo, Mike Johnston, an infrastructure engineer at Supergiant.io, discussed how to use Kubernetes to set up a SaaS infrastructure for your business. Mike Johnston is an infrastructure engineer at Supergiant.io with over 12 years of experience designing, deploying, and maintaining server and workstation infrastructure at all scales. He has experience with brick and mortar data centers as well as cloud providers like Digital Ocean, Amazon Web Services, and Rackspace. H...
Skeuomorphism usually means retaining existing design cues in something new that doesn’t actually need them. However, the concept of skeuomorphism can be thought of as relating more broadly to applying existing patterns to new technologies that, in fact, cry out for new approaches. In his session at DevOps Summit, Gordon Haff, Senior Cloud Strategy Marketing and Evangelism Manager at Red Hat, will discuss why containers should be paired with new architectural practices such as microservices ra...
In his session at 20th Cloud Expo, Scott Davis, CTO of Embotics, discussed how automation can provide the dynamic management required to cost-effectively deliver microservices and container solutions at scale. He also discussed how flexible automation is the key to effectively bridging and seamlessly coordinating both IT and developer needs for component orchestration across disparate clouds – an increasingly important requirement at today’s multi-cloud enterprise.
The Software Defined Data Center (SDDC), which enables organizations to seamlessly run in a hybrid cloud model (public + private cloud), is here to stay. IDC estimates that the software-defined networking market will be valued at $3.7 billion by 2016. Security is a key component and benefit of the SDDC, and offers an opportunity to build security 'from the ground up' and weave it into the environment from day one. In his session at 16th Cloud Expo, Reuven Harrison, CTO and Co-Founder of Tufin, ...
DevOps is often described as a combination of technology and culture. Without both, DevOps isn't complete. However, applying the culture to outdated technology is a recipe for disaster; as response times grow and connections between teams are delayed by technology, the culture will die. A Nutanix Enterprise Cloud has many benefits that provide the needed base for a true DevOps paradigm. In their Day 3 Keynote at 20th Cloud Expo, Chris Brown, a Solutions Marketing Manager at Nutanix, and Mark Lav...
Many organizations are now looking to DevOps maturity models to gauge their DevOps adoption and compare their maturity to their peers. However, as enterprise organizations rush to adopt DevOps, moving past experimentation to embrace it at scale, they are in danger of falling into the trap that they have fallen into time and time again. Unfortunately, we've seen this movie before, and we know how it ends: badly.
TCP (Transmission Control Protocol) is a common and reliable transmission protocol on the Internet. TCP was introduced in the 70s by Stanford University for US Defense to establish connectivity between distributed systems to maintain a backup of defense information. At the time, TCP was introduced to communicate amongst a selected set of devices for a smaller dataset over shorter distances. As the Internet evolved, however, the number of applications and users, and the types of data accessed and...