Welcome!

Microservices Expo Authors: Elizabeth White, Pat Romanski, PagerDuty Blog, Daniel Khan, David Green

Related Topics: @BigDataExpo, Microservices Expo, Containers Expo Blog, Agile Computing, @CloudExpo, Apache

@BigDataExpo: Article

Babies, Big Data, and IT Analytics

Machine learning is a topic that has gone from obscure niche to mainstream visibility over the last few years

Machine learning and IT analytics can be just as beneficial to IT operations as it is for monitoring vital signs of premature babies to identify danger signs too subtle or abnormal to be detected by a human. But an enterprise must be willing to implement monitoring and instrumentation that gathers data and incorporates business activity across organizational silos in order to get meaningful results from machine learning.

Machine learning is a topic that has gone from obscure niche to mainstream visibility over the last few years. High profile software companies like Splunk have tapped into the Big Data "explosion" to highlight the benefits of building systems that use algorithms and data to make decisions and evolve over time.

One recent article on machine learning on the O'Reilly Radar blog that caught my attention made a connection between web operations and medical care for premature infants. "Operations, machine learning, and premature babies" by Mike Loukides describes how machine learning is used to analyze data streamed from dozens of monitors connected to each baby. The algorithms are able to detect dangerous infections a full day before any symptoms are noticeable to a human.

An interesting point from the article is that the machine learning system is not looking for spikes or irregularities in the data; it is actually looking for the opposite. Babies who are about to become sick stop exhibiting the normal variations in vital signs shown by healthy babies. It takes a machine learning system to detect changes in behavior too subtle for a human to notice.

Mike Loukides then wonders whether machine learning can be applied to web operations. Typical performance monitoring focuses on thresholds to identify a problem. "But what if crossing a threshold isn't what indicates trouble, but the disappearance (or diminution) of some regular pattern?" Machine learning could identify symptoms that a human fails to identify because he's just looking for thresholds to be crossed.

Mike's conclusion sums up much of the state of the IT industry concerning machine learning:

At most enterprises, operations have not taken the next step. Operations staff doesn't have the resources (neither computational nor human) to apply machine intelligence to our problems. We'd have to capture all the data coming off our servers for extended periods, not just the server logs that we capture now, but any every kind of data we can collect: network data, environmental data, I/O subsystem data, you name it.

As someone who works for a company that applies a form of machine learning (Behavior Learning for predictive analytics) to IT operations and application performance management, I read this with great interest. I didn't necessarily disagree with his conclusion but tried to pull apart the reasoning behind why more companies aren't applying algorithms to their IT data to look for problems.

There are at least three requirements for companies who want to move ahead in this area:

1. Establish maturity of one's monitoring infrastructure. This is the most fundamental point. If you want to apply machine intelligence to IT operations then you need to first add instrumentation and monitoring. Numerous monitoring products and approaches abound but you have to get the data before you can analyze it.

2. Coordinate multiple enterprise silos. Modern IT applications are increasingly complex and may cross multiple enterprise silos such as server virtualization, network, databases, application development, and other middleware components. Enterprises must be willing to coordinate between these multiple groups in gathering monitoring data and performing cross-functional troubleshooting when there are performance or uptime issues.

3. Incorporate business activity monitoring (BAM). Business activity data provides the "vital signs" of a business. Examples of retail business activity data include number of units sold, total gross sales, and total net sales for a time period. Knowing the true business impact of an application performance problem requires the correlation of business data. When an outage occurred for 20 minutes, how many fewer units were sold? What was the reduction in gross and net sales?

An organization that can fulfill these requirements is capable of achieving real benefits in IT operations and can successfully apply analytics. Gartner has established the ITScore Maturity Model for determining one's sophistication in availability and performance monitoring. Here is the description for level 5, which is the top tier:

Behavior Learning engines, embedded knowledge, advanced correlation, trend analysis, pattern matching, and integrated IT and business data from sources such as BAM provide IT operations with the ability to dynamically manage the IT infrastructure in line with business policy.

Applying machine learning to IT operations isn't easy. Most enterprises don't do it because they need to overcome organizational inertia and gather data from multiple groups scattered throughout the enterprise. For the organizations willing to do this, however, they will see tangible business benefits. Just as a hospital could algorithmically detect the failing health of a premature infant, an enterprise willing to use machine learning will visibly see how abnormal problems within IT operations can impact revenue.

More Stories By Richard Park

Richard Park is Director of Product Management at Netuitive. He currently leads Netuitive's efforts to integrate with application performance and cloud monitoring solutions. He has nearly 20 years of experience in network security, database programming, and systems engineering. Some past jobs include product management at Sourcefire and Computer Associates, network engineering and security at Booz Allen Hamilton, and systems engineering at UUNET Technologies (now part of Verizon). Richard has an MS in Computer Science from Johns Hopkins, an MBA from Harvard Business School, and a BA in Social Studies from Harvard University.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@MicroservicesExpo Stories
The 19th International Cloud Expo has announced that its Call for Papers is open. Cloud Expo, to be held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA, brings together Cloud Computing, Big Data, Internet of Things, DevOps, Digital Transformation, Microservices and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding business opportuni...
DevOps at Cloud Expo – being held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA – announces that its Call for Papers is open. Born out of proven success in agile development, cloud computing, and process automation, DevOps is a macro trend you cannot afford to miss. From showcase success stories from early adopters and web-scale businesses, DevOps is expanding to organizations of all sizes, including the world's largest enterprises – and delivering real results. Am...

Modern organizations face great challenges as they embrace innovation and integrate new tools and services. They begin to mature and move away from the complacency of maintaining traditional technologies and systems that only solve individual, siloed problems and work “well enough.” In order to build...

The post Gearing up for Digital Transformation appeared first on Aug. 24, 2016 11:45 AM EDT  Reads: 1,258

To leverage Continuous Delivery, enterprises must consider impacts that span functional silos, as well as applications that touch older, slower moving components. Managing the many dependencies can cause slowdowns. See how to achieve continuous delivery in the enterprise.
DevOps at Cloud Expo, taking place Nov 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 19th Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to wait for long dev...
The following fictional case study is a composite of actual horror stories I’ve heard over the years. Unfortunately, this scenario often occurs when in-house integration teams take on the complexities of DevOps and ALM integration with an enterprise service bus (ESB) or custom integration. It is written from the perspective of an enterprise architect tasked with leading an organization’s effort to adopt Agile to become more competitive. The company has turned to Scaled Agile Framework (SAFe) as ...
SYS-CON Events announced today that eCube Systems, a leading provider of middleware modernization, integration, and management solutions, will exhibit at @DevOpsSummit at 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. eCube Systems offers a family of middleware evolution products and services that maximize return on technology investment by leveraging existing technical equity to meet evolving business needs. ...
It's been a busy time for tech's ongoing infatuation with containers. Amazon just announced EC2 Container Registry to simply container management. The new Azure container service taps into Microsoft's partnership with Docker and Mesosphere. You know when there's a standard for containers on the table there's money on the table, too. Everyone is talking containers because they reduce a ton of development-related challenges and make it much easier to move across production and testing environm...
As the world moves toward more DevOps and Microservices, application deployment to the cloud ought to become a lot simpler. The Microservices architecture, which is the basis of many new age distributed systems such as OpenStack, NetFlix and so on, is at the heart of Cloud Foundry - a complete developer-oriented Platform as a Service (PaaS) that is IaaS agnostic and supports vCloud, OpenStack and AWS. Serverless computing is revolutionizing computing. In his session at 19th Cloud Expo, Raghav...
Internet of @ThingsExpo, taking place November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 19th Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The Internet of Things (IoT) is the most profound change in personal and enterprise IT since the creation of the Worldwide Web more than 20 years ago. All major researchers estimate there will be tens of billions devices - comp...
Using new techniques of information modeling, indexing, and processing, new cloud-based systems can support cloud-based workloads previously not possible for high-throughput insurance, banking, and case-based applications. In his session at 18th Cloud Expo, John Newton, CTO, Founder and Chairman of Alfresco, described how to scale cloud-based content management repositories to store, manage, and retrieve billions of documents and related information with fast and linear scalability. He addres...
SYS-CON Events announced today that 910Telecom will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Housed in the classic Denver Gas & Electric Building, 910 15th St., 910Telecom is a carrier-neutral telecom hotel located in the heart of Denver. Adjacent to CenturyLink, AT&T, and Denver Main, 910Telecom offers connectivity to all major carriers, Internet service providers, Internet backbones and ...
19th Cloud Expo, taking place November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA, will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud strategy. Meanwhile, 94% of enterpri...
Akana has announced the availability of version 8 of its API Management solution. The Akana Platform provides an end-to-end API Management solution for designing, implementing, securing, managing, monitoring, and publishing APIs. It is available as a SaaS platform, on-premises, and as a hybrid deployment. Version 8 introduces a lot of new functionality, all aimed at offering customers the richest API Management capabilities in a way that is easier than ever for API and app developers to use.
The burgeoning trends around DevOps are translating into new types of IT infrastructure that both developers and operators can take advantage of. The next BriefingsDirect Voice of the Customer thought leadership discussion focuses on the burgeoning trends around DevOps and how that’s translating into new types of IT infrastructure that both developers and operators can take advantage of.
With so much going on in this space you could be forgiven for thinking you were always working with yesterday’s technologies. So much change, so quickly. What do you do if you have to build a solution from the ground up that is expected to live in the field for at least 5-10 years? This is the challenge we faced when we looked to refresh our existing 10-year-old custom hardware stack to measure the fullness of trash cans and compactors.
SYS-CON Events announced today that Isomorphic Software will exhibit at DevOps Summit at 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Isomorphic Software provides the SmartClient HTML5/AJAX platform, the most advanced technology for building rich, cutting-edge enterprise web applications for desktop and mobile. SmartClient combines the productivity and performance of traditional desktop software with the simp...
The emerging Internet of Everything creates tremendous new opportunities for customer engagement and business model innovation. However, enterprises must overcome a number of critical challenges to bring these new solutions to market. In his session at @ThingsExpo, Michael Martin, CTO/CIO at nfrastructure, outlined these key challenges and recommended approaches for overcoming them to achieve speed and agility in the design, development and implementation of Internet of Everything solutions wi...
Internet of @ThingsExpo, taking place November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with the 19th International Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world and ThingsExpo Silicon Valley Call for Papers is now open.
Thomas Bitman of Gartner wrote a blog post last year about why OpenStack projects fail. In that article, he outlined three particular metrics which together cause 60% of OpenStack projects to fall short of expectations: Wrong people (31% of failures): a successful cloud needs commitment both from the operations team as well as from "anchor" tenants. Wrong processes (19% of failures): a successful cloud automates across silos in the software development lifecycle, not just within silos.