|By Robert Eve||
|July 17, 2011 06:30 AM EDT||
Data Data Everywhere!
According to IDC's June 2011 report Extracting Value from Chaos, the amount of information currently stored is 1.8 zettabytes (1.8 trillion gigabytes). It has grown by a factor of nine in the past five years.
How can anyone deal with the complexity associated with these volumes?
For IT teams who need to understand this data as a first step before they can figure out how to use it to create high-impact business applications, this complexity is overwhelming.
As a result, the business waits, opportunities are missed, and IT is once again the blame.
Too Complex to Model
Today's application development approaches are based on methods that were appropriate for earlier times when less data complexity was the norm.
Take data modeling for example. Data modeling is the critical step where someone manually models a logical view of the data that an application requires. While lots of books have been written about the pros and cons of various modeling techniques, all assume the same prerequisite. That is the person doing the modeling understands the source data and data relationships.
That's great if you stay in a single domain or subject area, such as SAP or finance. But what happens when the application you are building needs data from SAP, Oracle, salesforce.com, and two different data marts?
- Does anyone in IT understand all data models, metadata, syntax, semantics and more across these sources?
- Let alone understand how these might relate with one and other?
- Or if they even do relate at all?
This challenge is far bigger than the need to understand five letter German acronyms in SAP R3 or flexfields in Oracle E-Business. In this example, each of these sources might have hundreds of table and column names that include some variation of the word "customer." Understanding these will take a lot of time. And the odds are several iterations before any models derived from these sources are correct.
What if we could somehow help data modelers better understand their data?
- Would they build their data models faster?
- Would their models be more accurate?
- Would the developers who used these models to integrate data using tools such as data virtualization, be able to build their virtualized views and data services faster?
- Would the applications that use these views and services be delivered to the business faster?
- And would the business be better off if the benefits of these new applications could be achieved sooner?
Yes. Yes. Yes. Yes. And Yes!
Discovery Tools Turn "What If" into Reality
There is a new class of data discovery products that can turn this what if's into reality. Some are standalone tools derived from data profiling offerings originally developed for use in data quality initiatives. Others are integrated with downstream tools in integrated suites. An example is Composite Discovery which is fully integrated with the Composite Data Virtualization Platform.
These discovery products use advanced mathematical algorithms and heuristics to reveal data patterns that are difficult for even the most experienced data modelers to uncover. Automatically crawling the source data and applying these methods, discovery tools reveal data and relationships across multiple source systems scattered throughout an organization. These products then present the data to the modelers using visualization studios that make it easy for the modelers to examine data, locate key entities and comprehend seemingly hidden connections.
Modelers can use that knowledge to quickly build the data models or schemas required. And then they can turn these over to data integration teams who can bind these models to the sources using views or data services. These rapidly built views are easy to validate and test with business users. If iteration is required, it can be done quickly. And once the views and data services are firmed, building out the application's user interface layer is a snap.
Data Discovery Delivers Faster Time to Solution
By accelerating the data relationship finding and modeling process, data modelers can use discovery tools to eliminate much of the time and effort typically employed in uncovering the entities and relationships necessary to build data models. Accelerating these initial critical development lifecycle steps reduces overall time to solution for new applications. IT looks better. And the business gains the application benefits sooner.
Data Discovery Delivers Better Quality Applications
Discovery tools help align data's business and technical contexts, facilitating greater collaboration between business and IT professionals. Discovery tools such as Composite Discovery display meta-data in an easy-to-read format that allows modelers and end users to easily validate requirements with greater confidence. More accurate validation at the front-end of the project reduces corrective actions in downstream steps. This reduces frustration and ensures a higher quality application.
Data Discovery Frees Top Talent
Discovery tools provide ease-of-use and automation that reduces the need for data modeling expertise. Top data modelers can now redirect these efforts towards other data governance activities.
Go for it!
In the time it took to read this article, your organization likely added a gigabyte of new data. Old methods and manual techniques cannot keep pace. Try data discovery. You'll be glad you did.
The best way to leverage your Cloud Expo presence as a sponsor and exhibitor is to plan your news announcements around our events. The press covering Cloud Expo and @ThingsExpo will have access to these releases and will amplify your news announcements. More than two dozen Cloud companies either set deals at our shows or have announced their mergers and acquisitions at Cloud Expo. Product announcements during our show provide your company with the most reach through our targeted audiences.
Oct. 25, 2016 07:45 AM EDT Reads: 4,867
This is a no-hype, pragmatic post about why I think you should consider architecting your next project the way SOA and/or microservices suggest. No matter if it’s a greenfield approach or if you’re in dire need of refactoring. Please note: considering still keeps open the option of not taking that approach. After reading this, you will have a better idea about whether building multiple small components instead of a single, large component makes sense for your project. This post assumes that you...
Oct. 25, 2016 06:00 AM EDT Reads: 7,245
As software becomes more and more complex, we, as software developers, have been splitting up our code into smaller and smaller components. This is also true for the environment in which we run our code: going from bare metal, to VMs to the modern-day Cloud Native world of containers, schedulers and microservices. While we have figured out how to run containerized applications in the cloud using schedulers, we've yet to come up with a good solution to bridge the gap between getting your conta...
Oct. 25, 2016 05:30 AM EDT Reads: 1,518
DevOps is being widely accepted (if not fully adopted) as essential in enterprise IT. But as Enterprise DevOps gains maturity, expands scope, and increases velocity, the need for data-driven decisions across teams becomes more acute. DevOps teams in any modern business must wrangle the ‘digital exhaust’ from the delivery toolchain, "pervasive" and "cognitive" computing, APIs and services, mobile devices and applications, the Internet of Things, and now even blockchain. In this power panel at @...
Oct. 25, 2016 05:15 AM EDT Reads: 1,997
DevOps theory promotes a culture of continuous improvement built on collaboration, empowerment, systems thinking, and feedback loops. But how do you collaborate effectively across the traditional silos? How can you make decisions without system-wide visibility? How can you see the whole system when it is spread across teams and locations? How do you close feedback loops across teams and activities delivering complex multi-tier, cloud, container, serverless, and/or API-based services?
Oct. 25, 2016 04:45 AM EDT Reads: 1,088
SYS-CON Events announced today that SoftNet Solutions will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. SoftNet Solutions specializes in Enterprise Solutions for Hadoop and Big Data. It offers customers the most open, robust, and value-conscious portfolio of solutions, services, and tools for the shortest route to success with Big Data. The unique differentiator is the ability to architect and ...
Oct. 25, 2016 04:00 AM EDT Reads: 965
Without lifecycle traceability and visibility across the tool chain, stakeholders from Planning-to-Ops have limited insight and answers to who, what, when, why and how across the DevOps lifecycle. This impacts the ability to deliver high quality software at the needed velocity to drive positive business outcomes. In his session at @DevOpsSummit 19th Cloud Expo, Eric Robertson, General Manager at CollabNet, will show how customers are able to achieve a level of transparency that enables everyon...
Oct. 25, 2016 03:30 AM EDT Reads: 1,308
Today every business relies on software to drive the innovation necessary for a competitive edge in the Application Economy. This is why collaboration between development and operations, or DevOps, has become IT’s number one priority. Whether you are in Dev or Ops, understanding how to implement a DevOps strategy can deliver faster development cycles, improved software quality, reduced deployment times and overall better experiences for your customers.
Oct. 25, 2016 03:30 AM EDT Reads: 919
As we enter the final week before the 19th International Cloud Expo | @ThingsExpo in Santa Clara, CA, it's time for me to reflect on six big topics that will be important during the show. Hybrid Cloud This general-purpose term seems to provide a comfort zone for many enterprise IT managers. It sounds reassuring to be able to work with one of the major public-cloud providers like AWS or Microsoft Azure while still maintaining an on-site presence.
Oct. 25, 2016 02:45 AM EDT Reads: 819
At its core DevOps is all about collaboration. The lines of communication must be opened and it takes some effort to ensure that they stay that way. It’s easy to pay lip service to trends and talk about implementing new methodologies, but without action, real benefits cannot be realized. Success requires planning, advocates empowered to effect change, and, of course, the right tooling. To bring about a cultural shift it’s important to share challenges. In simple terms, ensuring that everyone k...
Oct. 25, 2016 02:15 AM EDT Reads: 12,652
What do dependency resolution, situational awareness, and superheroes have in common? Meet Chris Corriere, a DevOps/Software Engineer at Autotrader, speaking on creative ways to maximize usage of all of the above. Mark Miller, Community Advocate and senior storyteller at Sonatype, caught up with Chris to learn more about what his team is up to.
Oct. 25, 2016 02:00 AM EDT Reads: 1,863
SYS-CON Events announced today that Super Micro Computer, Inc., a global leader in Embedded and IoT solutions, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 7-9, 2017, at the Javits Center in New York City, NY. Supermicro (NASDAQ: SMCI), the leading innovator in high-performance, high-efficiency server technology, is a premier provider of advanced server Building Block Solutions® for Data Center, Cloud Computing, Enterprise IT, Hadoop/Big Data, HPC and ...
Oct. 25, 2016 01:15 AM EDT Reads: 3,618
@DevOpsSummit has been named the ‘Top DevOps Influencer' by iTrend. iTrend processes millions of conversations, tweets, interactions, news articles, press releases, blog posts - and extract meaning form them and analyzes mobile and desktop software platforms used to communicate, various metadata (such as geo location), and automation tools. In overall placement, @DevOpsSummit ranked as the number one ‘DevOps Influencer' followed by @CloudExpo at third, and @MicroservicesE at 24th.
Oct. 25, 2016 01:15 AM EDT Reads: 4,040
JetBlue Airways uses virtual environments to reduce software development costs, centralize performance testing, and create a climate for continuous integration and real-time monitoring of mobile applications. The next BriefingsDirect Voice of the Customer performance engineering case study discussion examines how JetBlue Airways in New York uses virtual environments to reduce software development costs, centralize performance testing, and create a climate for continuous integration and real-tim...
Oct. 25, 2016 12:45 AM EDT Reads: 1,304
Without lifecycle traceability and visibility across the tool chain, stakeholders from Planning-to-Ops have limited insight and answers to who, what, when, why and how across the DevOps lifecycle. This impacts the ability to deliver high quality software at the needed velocity to drive positive business outcomes. In his general session at @DevOpsSummit at 19th Cloud Expo, Eric Robertson, General Manager at CollabNet, will discuss how customers are able to achieve a level of transparency that e...
Oct. 25, 2016 12:30 AM EDT Reads: 952
So you think you are a DevOps warrior, huh? Put your money (not really, it’s free) where your metrics are and prove it by taking The Ultimate DevOps Geek Quiz Challenge, sponsored by DevOps Summit. Battle through the set of tough questions created by industry thought leaders to earn your bragging rights and win some cool prizes.
Oct. 25, 2016 12:00 AM EDT Reads: 4,084
SYS-CON Events announced today that LeaseWeb USA, a cloud Infrastructure-as-a-Service (IaaS) provider, will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. LeaseWeb is one of the world's largest hosting brands. The company helps customers define, develop and deploy IT infrastructure tailored to their exact business needs, by combining various kinds cloud solutions.
Oct. 24, 2016 11:15 PM EDT Reads: 3,859
“Being able to take needless work out of the system is more important than being able to put more work into the system.” This is one of my favorite quotes from Gene Kim’s book, The Phoenix Project, and it plays directly into why we're announcing the DevOps Express initiative today. Tracing the Steps. For years now, I have witnessed needless work being performed across the DevOps industry. No, not within our clients DevOps and continuous delivery practices. I have seen it in the buyer’s journe...
Oct. 24, 2016 11:00 PM EDT Reads: 1,333
In many organizations governance is still practiced by phase or stage gate peer review, and Agile projects are forced to accommodate, which leads to WaterScrumFall or worse. But governance criteria and policies are often very weak anyway, out of date or non-existent. Consequently governance is frequently a matter of opinion and experience, highly dependent upon the experience of individual reviewers. As we all know, a basic principle of Agile methods is delegation of responsibility, and ideally ...
Oct. 24, 2016 09:00 PM EDT Reads: 3,511
SYS-CON Events announced today that Enzu will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Enzu’s mission is to be the leading provider of enterprise cloud solutions worldwide. Enzu enables online businesses to use its IT infrastructure to their competitive advantage. By offering a suite of proven hosting and management services, Enzu wants companies to focus on the core of their online busine...
Oct. 24, 2016 08:30 PM EDT Reads: 1,354