Welcome!

Microservices Expo Authors: Liz McMillan, Todd Matters, Pat Romanski, Elizabeth White, Stefana Muller

Related Topics: Microservices Expo, Industrial IoT

Microservices Expo: Article

Turning Service-Oriented Events into Business Insight

Event-Stream Processing - tools for an event-driven service oriented architecture

The quest for agility has spurred the recent rise of Service Oriented Architecture (SOA) and the face of modern IT integration architecture is changing. Technology stovepipes of the past are now being connected by Enterprise Service Bus (ESB) technology, which provides the backbone for the networking, communication, mediation, and service container management needed to support an SOA. Every integration software vendor provides some form of ESB in its products and the ESB has risen to the status of a de facto standard for SOA integrating. But what's the next step in the evolution of the IT integration fabric?

The next step is a new class of software called event-stream processing, or ESP. ESP has been hailed as the "next big thing" by both software vendors and analysts because it helps an SOA integration fabric become intelligent and responsive. ESP empowers a business to think differently about its operations and IT infrastructure because it can understand the state of a business in the now, rather than just in the past.

But aren't ESBs already emitting events and aren't all SOA infrastructure elements capable of emitting and carrying events? The answer is "yes," ESBs are event-capable today. However, they don't prescribe what to do with the events that are emitted by their services. That's the critical value that ESP brings.

ESP enables an event-driven SOA to decipher causal (if A is followed by B followed by C), temporal (within four seconds), and spatial (within 10 feet) relationships among events - and can do so in real-time. This kind of "enterprise wiretap" lets a business continuously analyse key performance indicators in real-time, identify threat and opportunity in real-time, and act instantly. These capabilities require a new style of computing - stream computing - that can deliver the missing link between an event-driven SOA and real-time business insight.

This article describes the characteristics of stream computing, how an ESP works, and how ESP and the ESB combine to provide an agile, intelligent integration fabric, and an intelligent, real-time enterprise.

Event-Driven SOA Delivers Real-Time Insight
The goal of an event-driven SOA is not merely the capture of all - or some - of the events generated by the architecture. Its goal is to derive business insight from those events - and derive that insight quickly enough to act. An example of a system that delivers real-time intelligence is the "Delta Nervous System."

Figure 1

Delta Airlines, under pressure from competitors like Southwest Airlines, described a concept in 2003 designed to alter its operational infrastructure. The Delta Nervous System monitors operational events, analyses them in real-time, and makes operational decisions in real-time.

The Delta Nervous System monitors weather forecasts and flight positions, identifies patterns among those events - such as changes in the weather - that determine flights that need rerouting. If flights have to be rerouted, that action triggers further events that cause passengers to be rebooked in real-time. Dashboards alert Delta flight operations personnel of the status of flights, with messages sent to kiosks in terminals and gate agents that keep the whole operational organization synchronized.

This real-time operational infrastructure is an example of an event-driven SOA: with flight positions transported by ESB; passenger itineraries checked by database query; dashboards updated with the state of flights, crews, and passengers; and decisions enacted by emitting events. This is a system that needs event processing to provide real-time insight.

The Missing Piece of an Event-Driven SOA: Event-Stream Processing
Real-time intelligence lets a business think differently about its operations and IT infrastructure because it can understand the state of the business in the now rather than just the past. Stream computing is a new style of computing that enables instant event pattern recognition, and it's vastly different from traditional styles of computing.

Figure 2

Traditional computing is static computing, and static computing uses static data. Static data is like a photograph, which captures information about a moment in time. For example, static business data is a table of customer data, transactions at a retail store, or records of shipments that have occurred in a company's supply chain. Static computing can be used to answer questions like: "How many shoes did we sell in our New York stores last week?" Over $200 billion of software is sold worldwide each year, and almost all of it is designed for static data. Relational databases, for example, are designed to manage static data.

Real-time computing is stream computing, and stream computing uses events. Events arrive in streams that resemble a movie in which streams of images and sounds flow by your senses. Patterns in the stream of images and sounds can make you laugh, cry, or scream. Much like a movie, streaming events in an enterprise let the business feel the pulse of operations as events travel through its arteries - the ESB. With stream computing, a business can identify patterns and make instant decisions while they still matter: "When four transactions against the same credit card number from four different shops happen in one minute, deny the next request, flag the account, and send a message to the fraud detection dashboard." And like a movie's impact on its audience, these patterns might well make a CFO laugh or cry.

Figure 3

Static computing and stream computing have fundamentally different physical characteristics - stream computing enables instant, real-time, intelligent decisions based on the patterns of business operations. These patterns are determined by the temporal, causal, or spatial relationships among events in the stream.

The concept of stream computing isn't new, but the efficacy of the tools that implement it is. In 1985, database researchers began to explore the element of time by extending the relational database with time-series extensions. But this approach was fatally flawed, since databases are designed to answer questions about the past, and event processing must make decisions in the now. So, beginning in 1998, a new approach - ESP - attacked the event-processing problem afresh, introduced the first wave of commercial ESP products, and ushered in a new physics of computing.

A BAM Dashboard Isn't Enough

Figure 4

An event-driven SOA is a technical architecture that lets software systems interoperate dynamically. But an effective SOA doesn't satisfy the requirements for instant business insight alone since it lacks the ability to identify temporal and causal relationships among events in real-time. Some software vendors promote Business Activity Monitoring, or BAM, as a new means of business insight:

BAM is "a business solution that is supported by an advanced technical infrastructure that enables rapid insight into new business strategies, the reduction of operating cost by real-time identification of issues, and improved process performance."

Unfortunately vendors have used BAM as a quasi-technical banner with which to sell flashy dashboards. But serious real-time intelligence demands more than a flashy dashboard. ESP provides a computing foundation from which it can answer real-time questions about streaming data, such as:

  • "How many plasma screens do we have in stock, how many have been ordered, what is our rate of consumption, and how are we doing against our forecasted rates - right now?"
  • "Is bad weather forecast within 10 miles of this flight's path? If so, re-route the plane over New York, determine which passengers will be impacted by the delay, rebook the passengers, update kiosks and gate agents at all locations - and update our real-time P&L."
  • "If transactions have occurred in the last five seconds against this credit card from different businesses in different physical locations deny the most recent request and alert fraud detection of the account in question."
Visualization technology isn't enough to answer these questions. An enterprise can ask and answer these questions with the temporal, causal, and spatial capabilities of ESP, and can visualize the results with the help of a dashboard. As a result, dashboards to visualize streams are an insufficient element to deliver event stream processing.

More Stories By Mark Palmer

Mark has over 14 years of experience in technology, most recently as
CTO of YouthStream Media Networks where he led all technology
initiatives, from internal operations to the creation of the Sodalis
platform for integrating and supporting several hundred of
YouthStream's partners, including leading colleges and universities
in the United States.

Comments (4) View Comments

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


Most Recent Comments
SYS-CON Belgium News Desk 08/15/06 05:25:40 PM EDT

The quest for agility has spurred the recent rise of Service Oriented Architecture (SOA) and the face of modern IT integration architecture is changing. Technology stovepipes of the past are now being connected by Enterprise Service Bus (ESB) technology, which provides the backbone for the networking, communication, mediation, and service container management needed to support an SOA. Every integration software vendor provides some form of ESB in its products and the ESB has risen to the status of a de facto standard for SOA integrating. But what's the next step in the evolution of the IT integration fabric?

SOA Web Services Journal News 08/15/06 03:52:42 PM EDT

The quest for agility has spurred the recent rise of Service Oriented Architecture (SOA) and the face of modern IT integration architecture is changing. Technology stovepipes of the past are now being connected by Enterprise Service Bus (ESB) technology, which provides the backbone for the networking, communication, mediation, and service container management needed to support an SOA. Every integration software vendor provides some form of ESB in its products and the ESB has risen to the status of a de facto standard for SOA integrating. But what's the next step in the evolution of the IT integration fabric?

SYS-CON Australia News Desk 08/01/06 03:36:04 PM EDT

The quest for agility has spurred the recent rise of Service Oriented Architecture (SOA) and the face of modern IT integration architecture is changing. Technology stovepipes of the past are now being connected by Enterprise Service Bus (ESB) technology, which provides the backbone for the networking, communication, mediation, and service container management needed to support an SOA. Every integration software vendor provides some form of ESB in its products and the ESB has risen to the status of a de facto standard for SOA integrating. But what's the next step in the evolution of the IT integration fabric?

SOA Web Services Journal News 07/31/06 08:38:52 AM EDT

The quest for agility has spurred the recent rise of Service Oriented Architecture (SOA) and the face of modern IT integration architecture is changing. Technology stovepipes of the past are now being connected by Enterprise Service Bus (ESB) technology, which provides the backbone for the networking, communication, mediation, and service container management needed to support an SOA. Every integration software vendor provides some form of ESB in its products and the ESB has risen to the status of a de facto standard for SOA integrating. But what's the next step in the evolution of the IT integration fabric?

@MicroservicesExpo Stories
New competitors, disruptive technologies, and growing expectations are pushing every business to both adopt and deliver new digital services. This ‘Digital Transformation’ demands rapid delivery and continuous iteration of new competitive services via multiple channels, which in turn demands new service delivery techniques – including DevOps. In this power panel at @DevOpsSummit 20th Cloud Expo, moderated by DevOps Conference Co-Chair Andi Mann, panelists examined how DevOps helps to meet the de...
For most organizations, the move to hybrid cloud is now a question of when, not if. Fully 82% of enterprises plan to have a hybrid cloud strategy this year, according to Infoholic Research. The worldwide hybrid cloud computing market is expected to grow about 34% annually over the next five years, reaching $241.13 billion by 2022. Companies are embracing hybrid cloud because of the many advantages it offers compared to relying on a single provider for all of their cloud needs. Hybrid offers bala...
"When we talk about cloud without compromise what we're talking about is that when people think about 'I need the flexibility of the cloud' - it's the ability to create applications and run them in a cloud environment that's far more flexible,” explained Matthew Finnie, CTO of Interoute, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
@DevOpsSummit at Cloud Expo taking place Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center, Santa Clara, CA, is co-located with the 21st International Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is ...
What's the role of an IT self-service portal when you get to continuous delivery and Infrastructure as Code? This general session showed how to create the continuous delivery culture and eight accelerators for leading the change. Don Demcsak is a DevOps and Cloud Native Modernization Principal for Dell EMC based out of New Jersey. He is a former, long time, Microsoft Most Valuable Professional, specializing in building and architecting Application Delivery Pipelines for hybrid legacy, and cloud ...
Containers, microservices and DevOps are all the rage lately. You can read about how great they are and how they’ll change your life and the industry everywhere. So naturally when we started a new company and were deciding how to architect our app, we went with microservices, containers and DevOps. About now you’re expecting a story of how everything went so smoothly, we’re now pushing out code ten times a day, but the reality is quite different.
There's a lot to gain from cloud computing, but success requires a thoughtful and enterprise focused approach. Cloud computing decouples data and information from the infrastructure on which it lies. A process that is a LOT more involved than dragging some folders from your desktop to a shared drive. Cloud computing as a mission transformation activity, not a technological one. As an organization moves from local information hosting to the cloud, one of the most important challenges is addressi...
For organizations that have amassed large sums of software complexity, taking a microservices approach is the first step toward DevOps and continuous improvement / development. Integrating system-level analysis with microservices makes it easier to change and add functionality to applications at any time without the increase of risk. Before you start big transformation projects or a cloud migration, make sure these changes won’t take down your entire organization.
21st International Cloud Expo, taking place October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud strategy. Me...
"We are a monitoring company. We work with Salesforce, BBC, and quite a few other big logos. We basically provide monitoring for them, structure for their cloud services and we fit into the DevOps world" explained David Gildeh, Co-founder and CEO of Outlyer, in this SYS-CON.tv interview at DevOps Summit at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
Microservices are increasingly used in the development world as developers work to create larger, more complex applications that are better developed and managed as a combination of smaller services that work cohesively together for larger, application-wide functionality. Tools such as Service Fabric are rising to meet the need to think about and build apps using a piece-by-piece methodology that is, frankly, less mind-boggling than considering the whole of the application at once. Today, we'll ...
Cloud Expo, Inc. has announced today that Andi Mann and Aruna Ravichandran have been named Co-Chairs of @DevOpsSummit at Cloud Expo Silicon Valley which will take place Oct. 31-Nov. 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. "DevOps is at the intersection of technology and business-optimizing tools, organizations and processes to bring measurable improvements in productivity and profitability," said Aruna Ravichandran, vice president, DevOps product and solutions marketing...
In his session at Cloud Expo, Alan Winters, an entertainment executive/TV producer turned serial entrepreneur, presented a success story of an entrepreneur who has both suffered through and benefited from offshore development across multiple businesses: The smart choice, or how to select the right offshore development partner Warning signs, or how to minimize chances of making the wrong choice Collaboration, or how to establish the most effective work processes Budget control, or how to ma...
SYS-CON Events announced today that CA Technologies has been named "Platinum Sponsor" of SYS-CON's 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. CA Technologies helps customers succeed in a future where every business - from apparel to energy - is being rewritten by software. From planning to development to management to security, CA creates software that fuels transformation for companies in the applic...
In the decade following his article, cloud computing further cemented Carr’s perspective. Compute, storage, and network resources have become simple utilities, available at the proverbial turn of the faucet. The value they provide is immense, but the cloud playing field is amazingly level. Carr’s quote above presaged the cloud to a T. Today, however, we’re in the digital era. Mark Andreesen’s ‘software is eating the world’ prognostication is coming to pass, as enterprises realize they must be...
A common misconception about the cloud is that one size fits all. Companies expecting to run all of their operations using one cloud solution or service must realize that doing so is akin to forcing the totality of their business functionality into a straightjacket. Unlocking the full potential of the cloud means embracing the multi-cloud future where businesses use their own cloud, and/or clouds from different vendors, to support separate functions or product groups. There is no single cloud so...
Both SaaS vendors and SaaS buyers are going “all-in” to hyperscale IaaS platforms such as AWS, which is disrupting the SaaS value proposition. Why should the enterprise SaaS consumer pay for the SaaS service if their data is resident in adjacent AWS S3 buckets? If both SaaS sellers and buyers are using the same cloud tools, automation and pay-per-transaction model offered by IaaS platforms, then why not host the “shrink-wrapped” software in the customers’ cloud? Further, serverless computing, cl...
Hybrid IT is today’s reality, and while its implementation may seem daunting at times, more and more organizations are migrating to the cloud. In fact, according to SolarWinds 2017 IT Trends Index: Portrait of a Hybrid IT Organization 95 percent of organizations have migrated crucial applications to the cloud in the past year. As such, it’s in every IT professional’s best interest to know what to expect.
The taxi industry never saw Uber coming. Startups are a threat to incumbents like never before, and a major enabler for startups is that they are instantly “cloud ready.” If innovation moves at the pace of IT, then your company is in trouble. Why? Because your data center will not keep up with frenetic pace AWS, Microsoft and Google are rolling out new capabilities. In his session at 20th Cloud Expo, Don Browning, VP of Cloud Architecture at Turner, posited that disruption is inevitable for comp...
Companies have always been concerned that traditional enterprise software is slow and complex to install, often disrupting critical and time-sensitive operations during roll-out. With the growing need to integrate new digital technologies into the enterprise to transform business processes, this concern has become even more pressing. A 2016 Panorama Consulting Solutions study revealed that enterprise resource planning (ERP) projects took an average of 21 months to install, with 57 percent of th...