Microservices Expo Authors: Liz McMillan, Pat Romanski, Elizabeth White, Mehdi Daoudi, Yeshim Deniz

Related Topics: Microservices Expo, Java IoT

Microservices Expo: Blog Post

Avoid Getting Lost on the (Enterprise Service) Bus

The risks of ESB-first SOA projects and how to mitigate them

So, you've been following ZapThink long enough to know that beginning a Service-Oriented Architecture (SOA) project by purchasing an Enterprise Service Bus (ESB) is starting at the wrong end of the initiative. Purchasing any technology, especially an ESB, at the beginning of an architecture project is a recipe for failure, you've been telling anyone who'll listen. But for whatever reason, your organization didn't pay attention to you, and now they've dropped a bundle on an ESB. Maybe your boss golfs with the vendor sales rep, or maybe the powers-that-be are listening to the wrong analyst firm, who knows. But in any case, you're stuck with that decision, and you're now expected to implement SOA with it.

Fortunately, while purchasing an ESB too early in a SOA project does substantially increase your risk of failure, all is not lost. After all, you're not alone; this mistake is one of the most prevalent SOA snafus in IT shops around the world today, and not all of those projects end up as failures. Many of today's ESBs are now mature products, and can be an important part of a fully functional SOA implementation. Understanding the risks that buying an ESB too early in a SOA initiative presents, and dealing with those risks proactively, can turn a bad situation around and get your SOA initiative back on the right track.

Understanding the Risks
Fundamentally, the problem with buying the ESB first is that you might fall into the trap of doing things the way the ESB would like you to do them, in light of the fact that many ESBs are in many ways traditional middleware under the covers. After all, if middleware solved all your problems, then you wouldn't be considering SOA in the first place -- and adding Service capabilities to your middleware doesn't change this fundamental fact.

In fact, the pitfalls that the ESB-first approach introduce fall into three broad categories:

  • Taking an overly integration-centric perspective of the project -- Most ESBs are generally really good at connecting things -- in other words, most ESBs are quite capable integration middleware solutions. The problem is, SOA isn't about connecting things, it's about building loosely-coupled Services the business can leverage to meet changing process needs. We want to get away from the "connecting things" approach to distributed computing, and instead move to a "composing Services" paradigm, where integration becomes a byproduct of composition.
  • The "middleware for your middleware" problem -- if it were practical (or even possible) to take a single piece of middleware and put it all by itself in the middle of your IT infrastructure, that would be one thing, but for most large (and many midsize) organizations, the vision of relying upon one piece of middleware to solve all integration problems is an unrealistic fantasy. In reality, organizations tend to have several different pieces of middleware, of different vintages and for different purposes. Introducing one or more ESBs into the mix means that now you have to integrate your ESBs with existing middleware as well as with each other, leading to the requirement of middleware for your middleware. Where will it ever end?
  • The "good money after bad" fallacy -- The "good money after bad" fallacy is actually much broader than IT. People would rather throw money at an approach that's already cost a bundle than to switch approaches to a less expensive, but more effective alternative. If you've been buying middleware from a vendor for years, and now they tell you that you need an ESB, you're likely to take that advice, even if an alternative is lower cost and lower risk, simply because you've already spent so much with that vendor.

ESB-First SOA Best Practices
Now that you've steered your bus past the pitfalls, let's see if we can point it in the right direction moving forward. The most important thing to remember is that your architecture should drive the technology, not the other way around. Remember that ESBs, like any mature solution, come with a boatload of features -- many of which may not be appropriate for your situation. It is often figuring out which features not to use rather than the capabilities you should actually use that is the key to being successful with a product like an ESB.

In particular, it is essential to take a process-driven approach to your infrastructure, instead of an integration-centric approach. Remember that building Service compositions that implement processes typically compose capabilities across multiple execution environments. Furthermore, those compositions are both dynamic and unpredictable -- the business process specialist in charge of the compositions may change them around long after you've deployed the Services. Governance becomes the key to managing that unpredictability, rather than pre-defined integrations.

As a result, you shouldn't rely upon any one execution environment for your Service implementations, or any one process management environment either, for that matter. ESBs can offer an effective, managed execution environment for some of your Service interfaces, but you rarely if ever want to rely upon any one runtime environment for all of your Services. In other words, you should balance the advantages of running your Services "on the bus" with the fact that SOA allows you to leverage heterogeneity both on and off the bus.

One essential point here is that SOA leverages interoperability more so than portability. In a virtual machine-based "write once, run anywhere" environment, whether Java or Microsoft Common Language Runtime (CLR)-based, distributed computing relies upon the portability of code. SOA, however, leverages the interoperability of the Service interfaces so that you don't need to move the underlying Service implementations around. As a result, running all your Services on the ESB can actually impede your SOA implementation, rather than support it.

So, if you shouldn't think of your ESB as either integration middleware or as a universal Service execution environment, then what role should your ESB play? The answer is a Service intermediary. Transformations and content-based routing are the essential capabilities a Service intermediary should deliver, in conjunction with robust security and management. Building the Business Service abstraction depends upon transformations and content-based routing, and fortunately, most ESBs offer these capabilities. So, only use the traditional messaging middleware capabilities of your ESB if you really need them, and only leverage the Service runtime your ESB provides when convenient, but configure your ESB as an intermediary to get full value out of it as part of your SOA infrastructure.

Not only does using an ESB as an intermediary enable you to architect the Business Service abstraction, it also resolves the "middleware for your middleware" problem, because intermediaries can intermediate between disparate integration technologies just as well as they can intermediate between Service providers and consumers. If you feel you need to use your ESB's message queuing technology, for example, just because it's there, however, then you won't get this benefit.

The ZapThink Take
Yes, you need security, governance, quality, and management, in addition to the transformation and content-based routing capabilities of Service intermediaries, in order to build an effective SOA infrastructure. But remember, ESBs aren't the be-all and end-all of SOA infrastructure -- many ESBs on the market include most of the above capabilities, but rarely if ever offer everything an organization requires. In fact, XML appliances are likely a better approach to security and policy enforcement, a registry/repository combined with a full-lifecycle SOA quality solution might serve as your design time and run time governance tools, while a robust SOA management solution might be a critical part of your infrastructure as well. In fact, many organizations leverage such products in conjunction with existing middleware to build out their SOA infrastructure without having to buy an ESB at all.

The bottom line is to always remember that the business drives the architecture, and the architecture drives the technology. Don't let the technology drive the architecture! SOA is particularly adept at abstracting existing technology, which can include recently purchased products in addition to your legacy environment. But knowing which of your existing capabilities to leverage -- and which to forego -- can make or break your SOA initiative.

Learn more at one of our upcoming Licensed ZapThink Architect Bootcamps or SOA & Cloud Governance courses.

More Stories By Jason Bloomberg

Jason Bloomberg is a leading IT industry analyst, Forbes contributor, keynote speaker, and globally recognized expert on multiple disruptive trends in enterprise technology and digital transformation. He is ranked #5 on Onalytica’s list of top Digital Transformation influencers for 2018 and #15 on Jax’s list of top DevOps influencers for 2017, the only person to appear on both lists.

As founder and president of Agile Digital Transformation analyst firm Intellyx, he advises, writes, and speaks on a diverse set of topics, including digital transformation, artificial intelligence, cloud computing, devops, big data/analytics, cybersecurity, blockchain/bitcoin/cryptocurrency, no-code/low-code platforms and tools, organizational transformation, internet of things, enterprise architecture, SD-WAN/SDX, mainframes, hybrid IT, and legacy transformation, among other topics.

Mr. Bloomberg’s articles in Forbes are often viewed by more than 100,000 readers. During his career, he has published over 1,200 articles (over 200 for Forbes alone), spoken at over 400 conferences and webinars, and he has been quoted in the press and blogosphere over 2,000 times.

Mr. Bloomberg is the author or coauthor of four books: The Agile Architecture Revolution (Wiley, 2013), Service Orient or Be Doomed! How Service Orientation Will Change Your Business (Wiley, 2006), XML and Web Services Unleashed (SAMS Publishing, 2002), and Web Page Scripting Techniques (Hayden Books, 1996). His next book, Agile Digital Transformation, is due within the next year.

At SOA-focused industry analyst firm ZapThink from 2001 to 2013, Mr. Bloomberg created and delivered the Licensed ZapThink Architect (LZA) Service-Oriented Architecture (SOA) course and associated credential, certifying over 1,700 professionals worldwide. He is one of the original Managing Partners of ZapThink LLC, which was acquired by Dovel Technologies in 2011.

Prior to ZapThink, Mr. Bloomberg built a diverse background in eBusiness technology management and industry analysis, including serving as a senior analyst in IDC’s eBusiness Advisory group, as well as holding eBusiness management positions at USWeb/CKS (later marchFIRST) and WaveBend Solutions (now Hitachi Consulting), and several software and web development positions.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.

Microservices Articles
The explosion of new web/cloud/IoT-based applications and the data they generate are transforming our world right before our eyes. In this rush to adopt these new technologies, organizations are often ignoring fundamental questions concerning who owns the data and failing to ask for permission to conduct invasive surveillance of their customers. Organizations that are not transparent about how their systems gather data telemetry without offering shared data ownership risk product rejection, regu...
Containers and Kubernetes allow for code portability across on-premise VMs, bare metal, or multiple cloud provider environments. Yet, despite this portability promise, developers may include configuration and application definitions that constrain or even eliminate application portability. In this session we'll describe best practices for "configuration as code" in a Kubernetes environment. We will demonstrate how a properly constructed containerized app can be deployed to both Amazon and Azure ...
DevOps is often described as a combination of technology and culture. Without both, DevOps isn't complete. However, applying the culture to outdated technology is a recipe for disaster; as response times grow and connections between teams are delayed by technology, the culture will die. A Nutanix Enterprise Cloud has many benefits that provide the needed base for a true DevOps paradigm. In their Day 3 Keynote at 20th Cloud Expo, Chris Brown, a Solutions Marketing Manager at Nutanix, and Mark Lav...
The now mainstream platform changes stemming from the first Internet boom brought many changes but didn’t really change the basic relationship between servers and the applications running on them. In fact, that was sort of the point. In his session at 18th Cloud Expo, Gordon Haff, senior cloud strategy marketing and evangelism manager at Red Hat, will discuss how today’s workloads require a new model and a new platform for development and execution. The platform must handle a wide range of rec...
The Internet of Things is clearly many things: data collection and analytics, wearables, Smart Grids and Smart Cities, the Industrial Internet, and more. Cool platforms like Arduino, Raspberry Pi, Intel's Galileo and Edison, and a diverse world of sensors are making the IoT a great toy box for developers in all these areas. In this Power Panel at @ThingsExpo, moderated by Conference Chair Roger Strukhoff, panelists discussed what things are the most important, which will have the most profound e...
If your cloud deployment is on AWS with predictable workloads, Reserved Instances (RIs) can provide your business substantial savings compared to pay-as-you-go, on-demand services alone. Continuous monitoring of cloud usage and active management of Elastic Compute Cloud (EC2), Relational Database Service (RDS) and ElastiCache through RIs will optimize performance. Learn how you can purchase and apply the right Reserved Instances for optimum utilization and increased ROI.
TCP (Transmission Control Protocol) is a common and reliable transmission protocol on the Internet. TCP was introduced in the 70s by Stanford University for US Defense to establish connectivity between distributed systems to maintain a backup of defense information. At the time, TCP was introduced to communicate amongst a selected set of devices for a smaller dataset over shorter distances. As the Internet evolved, however, the number of applications and users, and the types of data accessed and...
Consumer-driven contracts are an essential part of a mature microservice testing portfolio enabling independent service deployments. In this presentation we'll provide an overview of the tools, patterns and pain points we've seen when implementing contract testing in large development organizations.
In his session at 19th Cloud Expo, Claude Remillard, Principal Program Manager in Developer Division at Microsoft, contrasted how his team used config as code and immutable patterns for continuous delivery of microservices and apps to the cloud. He showed how the immutable patterns helps developers do away with most of the complexity of config as code-enabling scenarios such as rollback, zero downtime upgrades with far greater simplicity. He also demoed building immutable pipelines in the cloud ...
You have great SaaS business app ideas. You want to turn your idea quickly into a functional and engaging proof of concept. You need to be able to modify it to meet customers' needs, and you need to deliver a complete and secure SaaS application. How could you achieve all the above and yet avoid unforeseen IT requirements that add unnecessary cost and complexity? You also want your app to be responsive in any device at any time. In his session at 19th Cloud Expo, Mark Allen, General Manager of...