Welcome!

Microservices Expo Authors: Carmen Gonzalez, Yeshim Deniz, Jyoti Bansal, XebiaLabs Blog, Elizabeth White

Related Topics: @BigDataExpo, Java IoT, Microservices Expo, Containers Expo Blog, @CloudExpo, SDN Journal

@BigDataExpo: Article

Big Data Morphs

Big Data is becoming the art and science of actually using information to make decisions

There is a backlash growing against the term "Big Data." Some of the negativity is just because of hype; people are attaching the name to their company for any reason, absurd reasons, and/or no reason at all.

I recently saw a pitch by a new venture that "creates Big Data for your business." Literally, they will create some for you if you don't already have any. It's hardly the most egregious example. We've seen mega-vendors twist their acquisitions into "Big Data" products, even though some of these technologies are 32-bit only, or have data sheets trumpeting claims like "handle up to 10M records." Big Data, indeed.

Another driver of the backlash is the cognitive dissonance that results from being told Big Data is all about huge data volumes or infrastructure that deal with it. Big Data has clearly emerged in the context of volume, but most companies are not Facebook, Zynga or the Large Hadron Collider. They simply do not have trucks filled with data unexpectedly arriving at the loading docks. Technical and data managers who grok the nature of the problem reject Big Data as a term in this context. They may even call it BS.

Yet another reason is disappointment and/or failure, as a result of overinvesting in "the volume problem." I recently spoke with a large bank that jumped on a popular open source product bandwagon, years before everyone else. They spent a fortune to do what they could have done much more easily and cheaply with existing tools. The lead manager admitted their team was more "driven by volume" and the hyped technologies used by exciting companies. They didn't consider that solving a different problem might produce more value for the bank, much more cost-effectively.

Where is Big Data headed? The good news is that rather than being rejected, the phenomenon of Big Data is morphing. It is becoming the art and science of actually using information to make decisions - more than ever before. And it is being driven by the success of companies that actually do that.

The real Big Data is driven by several factors... for one, the rapid creation of data, including voluminous data, from technical systems that interact with lots of people, where all those interactions are a multiple of the number of transactions that may ultimately occur. Another is the significant fall in the costs of storage and compute power, plus the emergence of 64-bit platforms that can use far more memory. But the real, underlying driver is...value.

Companies that have combined data volume with ubiquitous computing power and have actually brought the information together and analyzed it have found more... more customers, more profits and more growth. This is why Big Data is truly a phenomenon. Better approaches can be realized most everywhere. It cuts across industries, sectors, even countries. It is not just about things we have already defined well: volume, velocity, variety, etc. It is about the need to compete, and win, using information. Sometimes it is about figuring out how to get the data to make a better decision, and not just moving forward with gut instinct, in lieu of that.

It will stay big because when you look at what it really takes to make better decisions, there is a lot to it. Structured and unstructured. Internal and external. SQL and NoSQL. Volume, velocity and variety. Big Data is the discovery of how much data is not being used, and then realizing that changing that fact is the real challenge...and the real opportunity.

More Stories By Sid Probstein

Sid Probstein is the Chief Technology Officer of Attivio, responsible for product & technology strategy, implementation and delivery. He has over 20 years of experience in managing R&D organizations and delivering award-winning, high-value enterprise software and solutions. Previously, he was CTO at GetConnected, Inc. (GCi), a market-leading transaction processing platform enabling the sale of digital services. He was also Vice President of Technology at Fast Search & Transfer, a global enterprise search company that is now part of Microsoft Corporation.

Prior to Fast, Sid was Vice President of Engineering at Northern Light Technology, where he was responsible for production of the first enterprise version of the search engine. He also served as Director of Software Engineering at Freemark Communications, and a Principal Architect/System Manager at John Hancock Financial Services.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@MicroservicesExpo Stories
NHK, Japan Broadcasting, will feature the upcoming @ThingsExpo Silicon Valley in a special 'Internet of Things' and smart technology documentary that will be filmed on the expo floor between November 3 to 5, 2015, in Santa Clara. NHK is the sole public TV network in Japan equivalent to the BBC in the UK and the largest in Asia with many award-winning science and technology programs. Japanese TV is producing a documentary about IoT and Smart technology and will be covering @ThingsExpo Silicon Val...
Cloud Expo, Inc. has announced today that Aruna Ravichandran, vice president of DevOps Product and Solutions Marketing at CA Technologies, has been named co-conference chair of DevOps at Cloud Expo 2017. The @DevOpsSummit at Cloud Expo New York will take place on June 6-8, 2017, at the Javits Center in New York City, New York, and @DevOpsSummit at Cloud Expo Silicon Valley will take place Oct. 31-Nov. 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
Is your application too difficult to manage? Do changes take dozens of developers hundreds of hours to execute, and frequently result in downtime across all your site’s functions? It sounds like you have a monolith! A monolith is one of the three main software architectures that define most applications. Whether you’ve intentionally set out to create a monolith or not, it’s worth at least weighing the pros and cons of the different architectural approaches and deciding which one makes the most s...
Back in February of 2017, Andrew Clay Schafer of Pivotal tweeted the following: “seriously tho, the whole software industry is stuck on deployment when we desperately need architecture and telemetry.” Intrigue in a 140 characters. For me, I hear Andrew saying, “we’re jumping to step 5 before we’ve successfully completed steps 1-4.”
Cloud promises the agility required by today’s digital businesses. As organizations adopt cloud based infrastructures and services, their IT resources become increasingly dynamic and hybrid in nature. Managing these require modern IT operations and tools. In his session at 20th Cloud Expo, Raj Sundaram, Senior Principal Product Manager at CA Technologies, will discuss how to modernize your IT operations in order to proactively manage your hybrid cloud and IT environments. He will be sharing be...
Developers want to create better apps faster. Static clouds are giving way to scalable systems, with dynamic resource allocation and application monitoring. You won't hear that chant from users on any picket line, but helping developers to create better apps faster is the mission of Lee Atchison, principal cloud architect and advocate at New Relic Inc., based in San Francisco. His singular job is to understand and drive the industry in the areas of cloud architecture, microservices, scalability ...
This recent research on cloud computing from the Register delves a little deeper than many of the "We're all adopting cloud!" surveys we've seen. They found that meaningful cloud adoption and the idea of the cloud-first enterprise are still not reality for many businesses. The Register's stats also show a more gradual cloud deployment trend over the past five years, not any sort of explosion. One important takeaway is that coherence across internal and external clouds is essential for IT right n...
Enterprise architects are increasingly adopting multi-cloud strategies as they seek to utilize existing data center assets, leverage the advantages of cloud computing and avoid cloud vendor lock-in. This requires a globally aware traffic management strategy that can monitor infrastructure health across data centers and end-user experience globally, while responding to control changes and system specification at the speed of today’s DevOps teams. In his session at 20th Cloud Expo, Josh Gray, Chie...
To more closely examine the variety of ways in which IT departments around the world are integrating cloud services, and the effect hybrid IT has had on their organizations and IT job roles, SolarWinds recently released the SolarWinds IT Trends Report 2017: Portrait of a Hybrid Organization. This annual study consists of survey-based research that explores significant trends, developments, and movements related to and directly affecting IT and IT professionals.
Today we can collect lots and lots of performance data. We build beautiful dashboards and even have fancy query languages to access and transform the data. Still performance data is a secret language only a couple of people understand. The more business becomes digital the more stakeholders are interested in this data including how it relates to business. Some of these people have never used a monitoring tool before. They have a question on their mind like “How is my application doing” but no id...
In large enterprises, environment provisioning and server provisioning account for a significant portion of the operations team's time. This often leaves users frustrated while they wait for these services. For instance, server provisioning can take several days and sometimes even weeks. At the same time, digital transformation means the need for server and environment provisioning is constantly growing. Organizations are adopting agile methodologies and software teams are increasing the speed ...
In his session at 20th Cloud Expo, Scott Davis, CTO of Embotics, will discuss how automation can provide the dynamic management required to cost-effectively deliver microservices and container solutions at scale. He will discuss how flexible automation is the key to effectively bridging and seamlessly coordinating both IT and developer needs for component orchestration across disparate clouds – an increasingly important requirement at today’s multi-cloud enterprise.
Keeping pace with advancements in software delivery processes and tooling is taxing even for the most proficient organizations. Point tools, platforms, open source and the increasing adoption of private and public cloud services requires strong engineering rigor – all in the face of developer demands to use the tools of choice. As Agile has settled in as a mainstream practice, now DevOps has emerged as the next wave to improve software delivery speed and output. To make DevOps work, organization...
In his general session at 19th Cloud Expo, Manish Dixit, VP of Product and Engineering at Dice, discussed how Dice leverages data insights and tools to help both tech professionals and recruiters better understand how skills relate to each other and which skills are in high demand using interactive visualizations and salary indicator tools to maximize earning potential. Manish Dixit is VP of Product and Engineering at Dice. As the leader of the Product, Engineering and Data Sciences team at D...
Software as a service (SaaS), one of the earliest and most successful cloud services, has reached mainstream status. According to Cisco, by 2019 more than four-fifths (83 percent) of all data center traffic will be based in the cloud, up from 65 percent today. The majority of this traffic will be applications. Businesses of all sizes are adopting a variety of SaaS-based services – everything from collaboration tools to mission-critical commerce-oriented applications. The rise in SaaS usage has m...
The proper isolation of resources is essential for multi-tenant environments. The traditional approach to isolate resources is, however, rather heavyweight. In his session at 18th Cloud Expo, Igor Drobiazko, co-founder of elastic.io, drew upon his own experience with operating a Docker container-based infrastructure on a large scale and present a lightweight solution for resource isolation using microservices. He also discussed the implementation of microservices in data and application integrat...
We'd all like to fulfill that "find a job you love and you'll never work a day in your life" cliché. But in reality, every job (even if it's our dream job) comes with its downsides. For you, the constant fight against shadow IT might get on your last nerves. For your developer coworkers, infrastructure management is the roadblock that stands in the way of focusing on coding. As you watch more and more applications and processes move to the cloud, technology is coming to developers' rescue-most r...
2016 has been an amazing year for Docker and the container industry. We had 3 major releases of Docker engine this year , and tremendous increase in usage. The community has been following along and contributing amazing Docker resources to help you learn and get hands-on experience. Here’s some of the top read and viewed content for the year. Of course releases are always really popular, particularly when they fit requests we had from the community.
Even for the most seasoned IT pros, the cloud is complicated. It can be difficult just to wrap your head around the many terms and acronyms that make up the cloud dictionary-not to mention actually mastering the technology. Unfortunately, complicated cloud terms are often combined to the point that their meanings are lost in a sea of conflicting opinions. Two terms that are used interchangeably (but shouldn't be) are hybrid cloud and multicloud. If you want to be the cloud expert your company ne...
SYS-CON Events announced today that CollabNet, a global leader in enterprise software development, release automation and DevOps solutions, will be a Bronze Sponsor of SYS-CON's 20th International Cloud Expo®, taking place from June 6-8, 2017, at the Javits Center in New York City, NY. CollabNet offers a broad range of solutions with the mission of helping modern organizations deliver quality software at speed. The company’s latest innovation, the DevOps Lifecycle Manager (DLM), supports Value S...