Welcome!

Microservices Expo Authors: Elizabeth White, Carmen Gonzalez, Pat Romanski, Liz McMillan, Sematext Blog

Related Topics: @BigDataExpo, Microservices Expo, IBM Cloud, Agile Computing, @CloudExpo, Cloud Security, SDN Journal

@BigDataExpo: Article

Innovative Analytics - Changing the IT Landscape

How log analysis is transforming how products are monitored

A common theme heard in the IT marketplace is that innovative technologies are enabling new insights to be harvested from data. Cloud computing provides the necessary infrastructure that allows smaller organizations to enjoy the value from these insights. As cloud computing becomes mainstream, analytics driven by new sources are leveraged by formation of new partnerships. GE and Google recently announced a partnership for the utility business that integrates Google map functions with GE's geospatial analytical tools to improve visualization. Any way you look at it, integrating consumer and industrial assets with speed and ease is game changing.

Recently, I attended the Strata and IBM Pulse 2013 Conferences that were relatively different but that both incorporated common analytics themes. The Strata Conference is independently organized and focused on data analytics while IBM Pulse, run by the company's Tivoli organization, primarily targets systems management.

Pulse 2013 focused more attention on analytics than last year's event.  While the Strata Conference focused on start-ups that demonstrated interesting analytic products, IBM is looking to leverage analytics as a core competency by embedding those processes into current solutions. Speaking at IBM's recent investor meeting, CEO Ginni Rometty projected $20B in analytics and big data revenues by 2015. To meet this goal, all IBM brands seem to be adding analytic capabilities.

One analytics area I found interesting at both Strata and IBM Pulse were solutions focused on system logs. Computer data logging is a common practice and supports the goal of keeping a history of administrator activities and problem diagnoses.  After an outage, logs are thoroughly analyzed by technicians who scour for the root cause then take appropriate steps to fix malfunctioning components.

The number of inter-connected components in a cloud computing infrastructure significantly increases the volume and variety of log data produced, as well as its importance. Traditional log file solutions leveraged legacy database technologies that are unable to process data being created at the variety and velocity today in a timely and cost effective manner. At the Strata Conference, Glassbeam and Splunk offered interesting solutions; and at Pulse, IBM released a new product capability in their SmartCloud portfolio that analyzes logs, events and metrics in an integrated manner. Let's discuss each in turn.

  • Glassbeam's goal is to provide a platform to help companies get intelligence from machine data. Their partnerships with LogiXML, OpSource (now part of Dimension Data) and Vertica (now part of HP) give them the capability to quickly assemble an innovative solution. Capabilities that focus on customer intelligence, product engineering and service revenue will be of interest to any organization looking to leverage intelligence and value from log data.
  • Splunk surprised a lot of folks when their stock price doubled on the first day of trading after they went public, offering insight to investors' thinking regarding Splunk's capabilities and the value of log analytics to customers. While Glassbeam is a niche and vertical solution, Splunk is a broad solution that covers the gamut of logs, from call centers to click streams.
  • IBM is a late entrant into developing a core product focused on log analytics leveraging new technologies. Their new product is promising as it integrates competencies from multiple software divisions as well as the recent Vivisimo acquisition. This combination of products brings the challenge of integration and installation, while allowing the new product to pull strengths of multiple best-of-breed products. By integrating systems manuals as an additional source to identify specific problems, this new workload analytics capability will be a good addition for existing and new IBM Tivoli customers.

Application logs have offered ways to diagnose system problems for a long time. Analyzing logs from multiple operational processes is a new challenge, and new technologies like Hadoop provide an excellent way to proactively act on intelligence gained. As technologies mature, technology giants will be forced to introduce new products to meet customer needs.

The ability to create new products rapidly is demonstrated by newcomers like Glassbeam and Splunk, changing the IT landscape. Pulse 2013 exhibited legacy product improvements along with new products bundling multiple capabilities from a diverse portfolio to address customer demand. Existing technology vendors need to quickly build their own products or acquire companies that will allow them to fill in this void in their portfolios. Predictive analytics from log data is a good example of how businesses can add significant value to their customers.

This post was first published on Robustcloud.com. Republished with permission.

More Stories By Larry Carvalho

Larry Carvalho runs Robust Cloud LLC, an advisory services company helping various ecosystem players develop a strategy to take advantage of cloud computing. As the 2010-12 Instructor of Cloud Expo's popular Cloud Computing Bootcamp, he has already led the bootcamp in New York, Silicon Valley, and Prague, receiving strong positive feedback from attendees about the value gained at these events. Carvalho has facilitated all-day sessions at customer locations to set a clear roadmap and gain consensus among attendees on strategy and product direction. He has participated in multiple discussion panels focused on cloud computing trends at information technology events, and he has delivered all-day cloud computing training to customers in conjunction with CloudCamps. To date, his role has taken him to clients in three continents.

@MicroservicesExpo Stories
Between 2005 and 2020, data volumes will grow by a factor of 300 – enough data to stack CDs from the earth to the moon 162 times. This has come to be known as the ‘big data’ phenomenon. Unfortunately, traditional approaches to handling, storing and analyzing data aren’t adequate at this scale: they’re too costly, slow and physically cumbersome to keep up. Fortunately, in response a new breed of technology has emerged that is cheaper, faster and more scalable. Yet, in meeting these new needs they...
@DevOpsSummit taking place June 6-8, 2017 at Javits Center, New York City, is co-located with the 20th International Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. @DevOpsSummit at Cloud Expo New York Call for Papers is now open.
In his session at 19th Cloud Expo, Claude Remillard, Principal Program Manager in Developer Division at Microsoft, contrasted how his team used config as code and immutable patterns for continuous delivery of microservices and apps to the cloud. He showed how the immutable patterns helps developers do away with most of the complexity of config as code-enabling scenarios such as rollback, zero downtime upgrades with far greater simplicity. He also demoed building immutable pipelines in the cloud ...
Today we can collect lots and lots of performance data. We build beautiful dashboards and even have fancy query languages to access and transform the data. Still performance data is a secret language only a couple of people understand. The more business becomes digital the more stakeholders are interested in this data including how it relates to business. Some of these people have never used a monitoring tool before. They have a question on their mind like “How is my application doing” but no id...
In IT, we sometimes coin terms for things before we know exactly what they are and how they’ll be used. The resulting terms may capture a common set of aspirations and goals – as “cloud” did broadly for on-demand, self-service, and flexible computing. But such a term can also lump together diverse and even competing practices, technologies, and priorities to the point where important distinctions are glossed over and lost.
Information technology is an industry that has always experienced change, and the dramatic change sweeping across the industry today could not be truthfully described as the first time we've seen such widespread change impacting customer investments. However, the rate of the change, and the potential outcomes from today's digital transformation has the distinct potential to separate the industry into two camps: Organizations that see the change coming, embrace it, and successful leverage it; and...
Monitoring of Docker environments is challenging. Why? Because each container typically runs a single process, has its own environment, utilizes virtual networks, or has various methods of managing storage. Traditional monitoring solutions take metrics from each server and applications they run. These servers and applications running on them are typically very static, with very long uptimes. Docker deployments are different: a set of containers may run many applications, all sharing the resource...
Join Impiger for their featured webinar: ‘Cloud Computing: A Roadmap to Modern Software Delivery’ on November 10, 2016, at 12:00 pm CST. Very few companies have not experienced some impact to their IT delivery due to the evolution of cloud computing. This webinar is not about deciding whether you should entertain moving some or all of your IT to the cloud, but rather, a detailed look under the hood to help IT professionals understand how cloud adoption has evolved and what trends will impact th...
Without lifecycle traceability and visibility across the tool chain, stakeholders from Planning-to-Ops have limited insight and answers to who, what, when, why and how across the DevOps lifecycle. This impacts the ability to deliver high quality software at the needed velocity to drive positive business outcomes. In his session at @DevOpsSummit 19th Cloud Expo, Eric Robertson, General Manager at CollabNet, showed how customers are able to achieve a level of transparency that enables everyone fro...
Internet of @ThingsExpo, taking place June 6-8, 2017 at the Javits Center in New York City, New York, is co-located with the 20th International Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. @ThingsExpo New York Call for Papers is now open.
The 20th International Cloud Expo has announced that its Call for Papers is open. Cloud Expo, to be held June 6-8, 2017, at the Javits Center in New York City, brings together Cloud Computing, Big Data, Internet of Things, DevOps, Containers, Microservices and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding business opportunity. Submit your speaking proposal ...
You have great SaaS business app ideas. You want to turn your idea quickly into a functional and engaging proof of concept. You need to be able to modify it to meet customers' needs, and you need to deliver a complete and secure SaaS application. How could you achieve all the above and yet avoid unforeseen IT requirements that add unnecessary cost and complexity? You also want your app to be responsive in any device at any time. In his session at 19th Cloud Expo, Mark Allen, General Manager of...
"Dice has been around for the last 20 years. We have been helping tech professionals find new jobs and career opportunities," explained Manish Dixit, VP of Product and Engineering at Dice, in this SYS-CON.tv interview at 19th Cloud Expo, held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA.
Rapid innovation, changing business landscapes, and new IT demands force businesses to make changes quickly. In the eyes of many, containers are at the brink of becoming a pervasive technology in enterprise IT to accelerate application delivery. In this presentation, attendees learned about the: The transformation of IT to a DevOps, microservices, and container-based architecture What are containers and how DevOps practices can operate in a container-based environment A demonstration of how ...
Application transformation and DevOps practices are two sides of the same coin. Enterprises that want to capture value faster, need to deliver value faster – time value of money principle. To do that enterprises need to build cloud-native apps as microservices by empowering teams to build, ship, and run in production. In his session at @DevOpsSummit at 19th Cloud Expo, Neil Gehani, senior product manager at HPE, discussed what every business should plan for how to structure their teams to delive...
As we enter the final week before the 19th International Cloud Expo | @ThingsExpo in Santa Clara, CA, it's time for me to reflect on six big topics that will be important during the show. Hybrid Cloud This general-purpose term seems to provide a comfort zone for many enterprise IT managers. It sounds reassuring to be able to work with one of the major public-cloud providers like AWS or Microsoft Azure while still maintaining an on-site presence.
Without lifecycle traceability and visibility across the tool chain, stakeholders from Planning-to-Ops have limited insight and answers to who, what, when, why and how across the DevOps lifecycle. This impacts the ability to deliver high quality software at the needed velocity to drive positive business outcomes. In his general session at @DevOpsSummit at 19th Cloud Expo, Phil Hombledal, Solution Architect at CollabNet, discussed how customers are able to achieve a level of transparency that e...
Much of the value of DevOps comes from a (renewed) focus on measurement, sharing, and continuous feedback loops. In increasingly complex DevOps workflows and environments, and especially in larger, regulated, or more crystallized organizations, these core concepts become even more critical. In his session at @DevOpsSummit at 18th Cloud Expo, Andi Mann, Chief Technology Advocate at Splunk, showed how, by focusing on 'metrics that matter,' you can provide objective, transparent, and meaningful f...
Logs are continuous digital records of events generated by all components of your software stack – and they’re everywhere – your networks, servers, applications, containers and cloud infrastructure just to name a few. The data logs provide are like an X-ray for your IT infrastructure. Without logs, this lack of visibility creates operational challenges for managing modern applications that drive today’s digital businesses.
Keeping pace with advancements in software delivery processes and tooling is taxing even for the most proficient organizations. Point tools, platforms, open source and the increasing adoption of private and public cloud services requires strong engineering rigor – all in the face of developer demands to use the tools of choice. As Agile has settled in as a mainstream practice, now DevOps has emerged as the next wave to improve software delivery speed and output. To make DevOps work, organization...