SOA, Web Services and Mass Data Movement via ETL

On the one hand, there's extreme pressure on businesses to deliver new customer offerings and innovative business capabilities, match increased competition, and deal with new partners and providers of niche services to offer cheaper service. On the other hand, most of us have legacy systems that prevent us from delivering to the business at the speed of business opportunities. Legacy applications house only parts of the business entity and the data is stored in proprietary structures.

Reader Feedback: Page 1 of 1

Surekha Durvasula commented on 3 Jan 2008

First of, I want to thank you all for your comments. I also agree with you John (Jones) that leveraging a real-time pub-sub interface is a great alternate option for keeping the Enterprise Repository synch'd up. However, here are some of the reasons I introduced an ETL interface into the mix for keeping the Enterprise Repository in synch.

A) For Systems of Records that are legacy systems and/or packaged vendor products it may not be feasible to extend these to create a publication process.

B) ETL has cross-business domain and enterprise business concept transformation logic and this logic is kept separated from the system of Record. Also, all of the relavant extraction and transformation logic required to create a single business concept from various Systetms of Record is included in a single ETL process instead of in the muliple pub/sub messaging processes.

C) Scheduled ETL is also effective in insuring that only the finalized stable state of long running business transactions across the various Systems of Record is applied to the Enterprise Repository.

D) ETL process includes information filter logic and it filters and transforms select System of Record transactions by correlating this information across business domains based on enterprise worthiness, relavance and priority.

E) Finally, the enterprise could choose scheduled ETL process as the primarily synchronization mechanism for the Enterprise Repository and this could be run during the SoR transaction downtimes so as to not affect the operational and transactional systetms and also to increase responsiveness of the business service.

Surekha.

John Jones commented on 3 Dec 2007

Why not set up a real-time pub/sub interface between the legacy systems and the Enterprise Data Store? Then there is no data latency waiting for the ETL jobs to run when the data required by the requesting service hasn't been populated yet.

Fernando Labastida commented on 30 Nov 2007

I'm glad to see an article about the use of ETL withn a SOA environment. At our company we're seeing the dichotomy between the hype of SOA, the ability for everything to be encapsulated as a service via loose coupling with an application's web services API, and the fact that a huge chunk of corporate data out there is still in legacy formats, such as VSAM, ISAM, COBOL, and other hard-to-get-to forats with no APIs.

Great article, and very informative!

YOUR FEEDBACK

	Data Services: Bridging the SOA and Metadata Management Gap By David Besemer Bill Miller wrote: Good article. Data Services is a great place to get value from SOA, and a great place to begin moving toward SOA. There are great open source tools for building SOA data services, including XAware.org. Bill Miller, XAware.org
	SOA Testing Framework By Anbarasu Krishnaswamy; Ravi Nagubadi; Rajeev Mahajan Robert Morschel wrote: Good article. Automated service testing is a vital component of SOA which is often forgotten. Robert soaprobe.blogspot.com
	How to Sell SOA by David Linthicum By David Linthicum Robert Morschel wrote: Selling SOA to the business to the degree that they remain engaged for the duration of what is going to be a lengthy and frustrating endeavour, and are willing to adapt to this new way of doing software, will be as tough as actually building anything. Of course I don't have concrete figures to subs...
	SOA 2 Point Oh No! By David Linthicum Robert Morschel wrote: It is very silly. I remember feeling the same thing when ESBs first came out - and used to describe ESB as an Enterprise Silver Bandwagon, the silver bullet that everyone likes to hop on! :) Robert soaprobe.blogspot.com
	SOA Vendors Focus Too Much on Integration . . . By David Linthicum wrote: Trackback Added: The Problem with SOA is Architecture; As I read David Linthicum's post on SOA vendors focus too much on integration... and not enough on architecture, this really clarified one of the messes we've gotten into with SOA. The problem is architecture. Dave's definition of architecture (...