Welcome!

SOA & WOA Authors: Elizabeth White, Paige Leidig, Liz McMillan, Lucian Loan, Nikita Ivanov

Related Topics: SOA & WOA, Virtualization

SOA & WOA: Article

Zettabytes of Data and Beyond

Is data discovery the answer to the data complexity problem?

Data Data Everywhere!
According to IDC's June 2011 report Extracting Value from Chaos, the amount of information currently stored is 1.8 zettabytes (1.8 trillion gigabytes).  It has grown by a factor of nine in the past five years.

How can anyone deal with the complexity associated with these volumes?

For IT teams who need to understand this data as a first step before they can figure out how to use it to create high-impact business applications, this complexity is overwhelming.

As a result, the business waits, opportunities are missed, and IT is once again the blame.

Too Complex to Model
Today's application development approaches are based on methods that were appropriate for earlier times when less data complexity was the norm.

Take data modeling for example. Data modeling is the critical step where someone manually models a logical view of the data that an application requires.  While lots of books have been written about the pros and cons of various modeling techniques, all assume the same prerequisite.  That is the person doing the modeling understands the source data and data relationships.

That's great if you stay in a single domain or subject area, such as SAP or finance. But what happens when the application you are building needs data from SAP, Oracle, salesforce.com, and two different data marts?

  • Does anyone in IT understand all data models, metadata, syntax, semantics and more across these sources?
  • Let alone understand how these might relate with one and other?
  • Or if they even do relate at all?

This challenge is far bigger than the need to understand five letter German acronyms in SAP R3 or flexfields in Oracle E-Business.   In this example, each of these sources might have hundreds of table and column names that include some variation of the word "customer."  Understanding these will take a lot of time.  And the odds are several iterations before any models derived from these sources are correct.

What If?
What if we could somehow help data modelers better understand their data?

  • Would they build their data models faster?
  • Would their models be more accurate?
  • Would the developers who used these models to integrate data using tools such as data virtualization, be able to build their virtualized views and data services faster?
  • Would the applications that use these views and services be delivered to the business faster?
  • And would the business be better off if the benefits of these new applications could be achieved sooner?

Yes. Yes. Yes. Yes. And Yes!

Discovery Tools Turn "What If" into Reality
There is a new class of data discovery products that can turn this what if's into reality. Some are standalone tools derived from data profiling offerings originally developed for use in data quality initiatives.  Others are integrated with downstream tools in integrated suites. An example is Composite Discovery which is fully integrated with the Composite Data Virtualization Platform.

These discovery products use advanced mathematical algorithms and heuristics to reveal data patterns that are difficult for even the most experienced data modelers to uncover.  Automatically crawling the source data and applying these methods, discovery tools reveal data and relationships across multiple source systems scattered throughout an organization.  These products then present the data to the modelers using visualization studios that make it easy for the modelers to examine data, locate key entities and comprehend seemingly hidden connections.

Modelers can use that knowledge to quickly build the data models or schemas required.  And then they can turn these over to data integration teams who can bind these models to the sources using views or data services. These rapidly built views are easy to validate and test with business users.  If iteration is required, it can be done quickly.  And once the views and data services are firmed, building out the application's user interface layer is a snap.

Data Discovery Delivers Faster Time to Solution
By accelerating the data relationship finding and modeling process, data modelers can use discovery tools to eliminate much of the time and effort typically employed in uncovering the entities and relationships necessary to build data models. Accelerating these initial critical development lifecycle steps reduces overall time to solution for new applications.   IT looks better.  And the business gains the application benefits sooner.

Data Discovery Delivers Better Quality Applications
Discovery tools help align data's business and technical contexts, facilitating greater collaboration between business and IT professionals.  Discovery tools such as Composite Discovery display meta-data in an easy-to-read format that allows modelers and end users to easily validate requirements with greater confidence.  More accurate validation at the front-end of the project reduces corrective actions in downstream steps.  This reduces frustration and ensures a higher quality application.

Data Discovery Frees Top Talent
Discovery tools provide ease-of-use and automation that reduces the need for data modeling expertise. Top data modelers can now redirect these efforts towards other data governance activities.

Go for it!
In the time it took to read this article, your organization likely added a gigabyte of new data.  Old methods and manual techniques cannot keep pace.  Try data discovery.  You'll be glad you did.

More Stories By Robert Eve

Robert Eve is the EVP of Marketing at Composite Software, the data virtualization gold standard and co-author of Data Virtualization: Going Beyond Traditional Data Integration to Achieve Business Agility. Bob's experience includes executive level roles at leading enterprise software companies such as Mercury Interactive, PeopleSoft, and Oracle. Bob holds a Masters of Science from the Massachusetts Institute of Technology and a Bachelor of Science from the University of California at Berkeley.