Welcome!

Microservices Expo Authors: Pat Romanski, Automic Blog, Liz McMillan, Jason Bloomberg, Elizabeth White

Related Topics: Industrial IoT

Industrial IoT: Article

Integrating Enterprise Information on Demand with XQuery - Part 2

Integrating Enterprise Information on Demand with XQuery - Part 2

In Part I of this article (XML-J, Vol. 4, issue 6), we introduced the enterprise information integration (EII) problem and explained how the XML query language XQuery and related technologies - specifically XML, XML Schema, and Web services - are central to enabling this age-old problem to be successfully addressed at last.

We provided a technical overview of the XQuery language and presented a simple "single view of Customer" example to illustrate XQuery's role in the EII domain. The example was based on an electronics retailer that wanted to share customer information across three portals - portals for customer self-service, credit approval, and product service. The information to be integrated resided in a variety of back-end information sources, including two relational database management systems, an SAP system, and a Web service.

In this article, our XQuery/EII saga continues. In this installment, we look at how EII relates to two other technologies designed for integration tasks, namely enterprise application integration (EAI) and extract-transform-load (ETL) tools. We also take a brief look at BEA Liquid Data for WebLogic, an XQuery-based EII offering, and discuss how XQuery and Liquid Data were put to use recently in a telecommunications-related customer project.

What About EAI?
Given the industry buzz around EAI today, a natural question about EII is "so why bother?" That is, why isn't a modern EAI solution alone - for example, a workflow engine with XML-based data transformation capabilities - sufficient to solve the EII problem? The answer is, in principle, that EAI is in fact sufficient to solve the EII problem. A developer could always choose to hand-build a set of workflows, writing one workflow per application-level "query" to deliver the desired information back to the calling applications. In the example from Part I of this article, three hand-tailored workflows could instead be written to provide information retrieval capabilities comparable to our XQuery-based solution. But is that the best approach, in terms of development time and maintenance cost?

The basic question here is when to use a declarative query language (XQuery in the case of modern EII) versus constructing code in a procedural language (a workflow language in the case of EAI). The lessons from the relational database revolution are clear: When applicable, a declarative approach offers significant advantages. Instead of hand-constructing a "query plan" (EAI workflow) to extract the needed data from each of the data sources in some manually predefined order, the EII approach allows a single, smaller, and simpler declarative query to be written.

The resulting benefits should be obvious. First, the user does not need to build each query plan by hand, which could involve a considerable effort. Instead, the user specifies (when defining the core view) what data sources are relevant and what logical conditions relate and characterize the data to be retrieved. Second, queries can be optimized automatically by the EII middleware, resulting in an optimal query execution plan (order of accessing the sources, queries or methods to extract the data, etc.) for each different query. For example, using EAI, one central workflow could be written to retrieve all of the customer information in Part I's example, and then other workflows could be written to first call this workflow and then further filter the results. However, in the EII approach, the query processor will (for each query) prune out irrelevant data sources as well as push SQL selection conditions (such as only retrieving "Open" support cases in Listing 2 of Part I) down to any RDBMS data sources. Third, as the data sources change over time in terms of their schemas, statistics, or performance, the EII user will not be forced to rewrite all of his or her queries. Simply maintaining each base view query and re-optimizing the other queries will adapt their query execution plans to the new situation. In contrast, in the case of EAI, many workflows would have to be rewritten to handle most such changes.

There really isn't an either/or choice to be made between EAI and EII at all. Both technologies have critical roles to play in an overall enterprise integration solution. These technologies are complementary: EII provides ease of data integration, while EAI provides ease of process integration. EII is appropriate for composing integrated views and queries over enterprise data. EAI is the appropriate technology for creating composite applications that orchestrate the functional capabilities of a set of related but independent applications, Web services, etc. Moreover, EII can be used to handily augment EAI in scenarios where workflows need to access integrated data views. For example, if our electronics retailer wanted its order process to offer free shipping to customers who have ordered more than $1,000 of goods during the year and who have accumulated more than 5,000 reward points, the integrated view of customer from Part I could be used to easily access the relevant information from within the order entry workflow.

What About ETL?
Another technology related to EII is ETL. In fact, ETL tools are designed precisely for the purpose of integrating data from multiple sources. These tools are therefore another category of software that naturally leads to a "why bother with EII?" question - why isn't ETL technology the answer? As you'll see, the answer is again that both technologies have their place in modern IT architectures.

ETL tools are designed for use in moving data from a variety of sources into a data warehouse for offline analysis and reporting purposes. As the name suggests, ETL tools provide facilities for extracting data from a source; transforming that data into a more suitable form for inclusion in the data warehouse, possibly cleansing it in the process; and then loading the transformed data into the warehouse's database. Typical ETL tools are therefore focused on supporting the design and administration of data migration, cleansing, and transformation processes. These are often batch processes that occur on a daily or weekly basis.

Data warehouses and the ETL tools that feed them are invaluable for enabling businesses to aggregate and analyze historical information. For example, our electronics retailer might very well want to keep track of customer data, sales data, and product issue data over a period of years in order to analyze customer behavior by geographic region over time, improve their credit card risk model, and so on. A data warehouse is the appropriate place to retain such data and run large analytical queries against it, and ETL technology is the right technology today for creating, cleaning, and maintaining the data in the warehouse. However, ETL is not the right technology for building applications that need access to current operational data - it doesn't support the declarative creation of views or real-time access to operational data through queries.

For applications that need to integrate current information, Part I of this article showed how XQuery can be used to declaratively specify reusable views that aggregate data from multiple operational stores and how XQuery can be used to write XML queries over such integrated views. We also explained how standard database query processing techniques, including view expansion, predicate pushdown, and distributed query optimization, can be applied to XQuery, making XQuery-based EII an excellent technological fit for such applications.

Clearly, both ETL and EII technologies have important roles to play in today's enterprise. ETL serves to feed data warehouses, while EII is an enabler for applications that need timely access to current, integrated information from a variety of operational enterprise data sources. As with EAI, there are also cases where the two technologies come together. As one example, an ETL tool could be used to help create and maintain a cross-reference table to relate different notions of "customer id" for use in creating XQuery-based EII views across different back-end systems. As another example, an ETL-fed data warehouse could be used to build a portal for analyzing the historical behavior of a company's top customers, with an EII tool used to allow click-through inspection of the customers' purchases in the past 24 hours.

Putting XQuery-Based EII to Work
For the reasons discussed in this article, XQuery-based EII middleware is an emerging product segment that promises to deliver the tools and technology needed in this important space. One commercially available XQuery-based middleware product is BEA Liquid Data for WebLogic. Liquid Data is capable of accessing data from relational database management systems, Web services, packaged applications (through J2EE CA adapters and application views), XML files, XML messages, and, through a custom function mechanism, most any other data source as well. For illustration purposes, the architecture of Liquid Data is depicted in Figure 1. Liquid Data provides default XML views of all of its data sources and provides an XQuery-based graphical view and query editor for use in integrating and enhancing information drawn from one or more data sources. It includes a distributed query processing engine as well as providing advanced features such as support for query result caching and both data-source-level and stored query-level access control.

As a final example of the applicability of XQuery to enterprise information integration problems, we'll describe an actual customer integration exercise where Liquid Data was put to use. In that project, a large telecommunications vendor wanted to create a single view of order information for one of its business divisions. The goal of the project was to make integrated order information available to the division's customers (other businesses) through a Web portal, enabling their customers to log in and check on the status of their orders, as well as making information available to the division's own customer service representatives.

The division had data distributed across multiple systems, including a relational database containing order summary information and two different order management systems. Order details were kept in one or the other of the two order management systems, depending on the type of order. Functionality-wise, a limited view of order details was provided through the customer order status portal that the division built using Liquid Data, whereas customer service representatives were permitted to see all of the order data through their portal. In both cases, it was possible to search for order information by various combinations of purchase order number, date range, and order.

The use of XQuery-based EII technology enabled the customer to complete their portal project in much less time than they had expected it to take with traditional technologies, and their total cost of ownership was also lower due to the reusability of Liquid Data assets and the low cost of maintenance enabled by EII.

Summary
In this article, we have explained how XQuery is beginning to transform the integration world, making it possible to finally tackle the enterprise information integration problem where past attempts have failed. In Part I we provided an overview of XQuery and illustrated how it could be used to integrate the disparate information sources of a hypothetical electronics retailer. In Part II we discussed the relationship of EII to EAI and ETL technologies and then briefly presented BEA's XQuery-based EII product and described one of the customer projects in which it was used.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@MicroservicesExpo Stories
SYS-CON Events announced today that Peak 10, Inc., a national IT infrastructure and cloud services provider, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. Peak 10 provides reliable, tailored data center and network services, cloud and managed services. Its solutions are designed to scale and adapt to customers’ changing business needs, enabling them to lower costs, improve performance and focus inter...
From the conception of Docker containers to the unfolding microservices revolution we see today, here is a brief history of what I like to call 'containerology'. In 2013, we were solidly in the monolithic application era. I had noticed that a growing amount of effort was going into deploying and configuring applications. As applications had grown in complexity and interdependency over the years, the effort to install and configure them was becoming significant. But the road did not end with a ...
You deployed your app with the Bluemix PaaS and it's gaining some serious traction, so it's time to make some tweaks. Did you design your application in a way that it can scale in the cloud? Were you even thinking about the cloud when you built the app? If not, chances are your app is going to break. Check out this webcast to learn various techniques for designing applications that will scale successfully in Bluemix, for the confidence you need to take your apps to the next level and beyond.
Digital means customer preferences and behavior are driving enterprise technology decisions to be sure, but let’s not forget our employees. After all, when we say customer, we mean customer writ large, including partners, supply chain participants, and yes, those salaried denizens whose daily labor forms the cornerstone of the enterprise. While your customers bask in the warm rays of your digital efforts, are your employees toiling away in the dark recesses of your enterprise, pecking data into...
Wow, if you ever wanted to learn about Rugged DevOps (some call it DevSecOps), sit down for a spell with Shannon Lietz, Ian Allison and Scott Kennedy from Intuit. We discussed a number of important topics including internal war games, culture hacking, gamification of Rugged DevOps and starting as a small team. There are 100 gold nuggets in this conversation for novices and experts alike.
SYS-CON Events announced today that DatacenterDynamics has been named “Media Sponsor” of SYS-CON's 18th International Cloud Expo, which will take place on June 7–9, 2016, at the Javits Center in New York City, NY. DatacenterDynamics is a brand of DCD Group, a global B2B media and publishing company that develops products to help senior professionals in the world's most ICT dependent organizations make risk-based infrastructure and capacity decisions.
With DevOps becoming more well-known and established practice in nearly every industry that delivers software, it is important to continually reassess its efficacy. This week’s top 10 includes a discussion on how the quick uptake of DevOps adoption in the enterprise has posed some serious challenges. Additionally, organizations who have taken the DevOps plunge must find ways to find, hire and keep their DevOps talent in order to keep the machine running smoothly.
Call it DevOps or not, if you are concerned about releasing more code faster and at a higher quality, the resulting software delivery chain and process will look and smell like DevOps. But for existing development teams, no matter what the velocity objective is, getting from here to there is not something that can be done without a plan. Moving your release cadence from months to weeks is not just about learning Agile practices and getting some automation tools. It involves people, tooling and ...
Between the mockups and specs produced by analysts, and resulting applications built by developers, there exists a gulf where projects fail, costs spiral, and applications disappoint. Methodologies like Agile attempt to address this with intensified communication, with partial success but many limitations. In his session at 18th Cloud Expo, Charles Kendrick, CTO & Chief Architect at Isomorphic Software, will present a revolutionary model enabled by new technologies. Learn how business and devel...
The notion of customer journeys, of course, are central to the digital marketer’s playbook. Clearly, enterprises should focus their digital efforts on such journeys, as they represent customer interactions over time. But making customer journeys the centerpiece of the enterprise architecture, however, leaves more questions than answers. The challenge arises when EAs consider the context of the customer journey in the overall architecture as well as the architectural elements that make up each...
In 2006, Martin Fowler posted his now famous essay on Continuous Integration. Looking back, what seemed revolutionary, radical or just plain crazy is now common, pedestrian and "just what you do." I love it. Back then, building and releasing software was a real pain. Integration was something you did at the end, after code complete, and we didn't know how long it would take. Some people may recall how we, as an industry, spent a massive amount of time integrating code from one team with another...
As the software delivery industry continues to evolve and mature, the challenge of managing the growing list of the tools and processes becomes more daunting every day. Today, Application Lifecycle Management (ALM) platforms are proving most valuable by providing the governance, management and coordination for every stage of development, deployment and release. Recently, I spoke with Madison Moore at SD Times about the changing market and where ALM is headed.
Struggling to keep up with increasing application demand? Learn how Platform as a Service (PaaS) can streamline application development processes and make resource management easy.
If there is anything we have learned by now, is that every business paves their own unique path for releasing software- every pipeline, implementation and practices are a bit different, and DevOps comes in all shapes and sizes. Software delivery practices are often comprised of set of several complementing (or even competing) methodologies – such as leveraging Agile, DevOps and even a mix of ITIL, to create the combination that’s most suitable for your organization and that maximize your busines...
The goal of any tech business worth its salt is to provide the best product or service to its clients in the most efficient and cost-effective way possible. This is just as true in the development of software products as it is in other product design services. Microservices, an app architecture style that leans mostly on independent, self-contained programs, are quickly becoming the new norm, so to speak. With this change comes a declining reliance on older SOAs like COBRA, a push toward more s...
SYS-CON Events announced today that Stratoscale, the software company developing the next generation data center operating system, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. Stratoscale is revolutionizing the data center with a zero-to-cloud-in-minutes solution. With Stratoscale’s hardware-agnostic, Software Defined Data Center (SDDC) solution to store everything, run anything and scale everywhere...
SYS-CON Events announced today that Men & Mice, the leading global provider of DNS, DHCP and IP address management overlay solutions, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. The Men & Mice Suite overlay solution is already known for its powerful application in heterogeneous operating environments, enabling enterprises to scale without fuss. Building on a solid range of diverse platform support,...
Much of the discussion around cloud DevOps focuses on the speed with which companies need to get new code into production. This focus is important – because in an increasingly digital marketplace, new code enables new value propositions. New code is also often essential for maintaining competitive parity with market innovators. But new code doesn’t just have to deliver the functionality the business requires. It also has to behave well because the behavior of code in the cloud affects performan...
This is not a small hotel event. It is also not a big vendor party where politicians and entertainers are more important than real content. This is Cloud Expo, the world's longest-running conference and exhibition focused on Cloud Computing and all that it entails. If you want serious presentations and valuable insight about Cloud Computing for three straight days, then register now for Cloud Expo.
I had the opportunity to catch up with Chris Corriere - DevOps Engineer at AutoTrader - to talk about his experiences in the realm of Rugged DevOps. We discussed automation, culture and collaboration, and which thought leaders he is following. Chris Corriere: Hey, I'm Chris Corriere. I'm a DevOps Engineer AutoTrader. Derek Weeks: Today we're going to talk about Rugged DevOps. It's a subject that's gaining a lot of traction in the community but not a lot of people are really familiar with wh...