|By Gorka Sadowski||
|November 16, 2012 07:45 AM EST||
The debate between data and information has been going on for quite some time. When people say "knowledge is power", are they referring to data or information? Is knowledge different still? And how about "intelligence" where does that fit?
How can we go from data to information to knowledge to intelligence?
The answer is simple. By understanding the animated nature of data evolution and transformation, and acting upon this understanding.
And this is brought to light by logs from your Information Systems.
Understand this and unleash the Power of Logs.
Figure 1 - Data to Information to Knowledge to Intelligence, and the role of logs as metadata
Data seems mainly one-dimensional.
Consult any data base or data warehouse, perform even complex queries on these and you will get a "flat" answer.
The fact that you get a single answer will make you think that data is absolute, where in fact it's not.
In fact, data is relative. And it has a very high frequency of change!
For example data is relative in time, and today's flat answer may not be yesterday's flat answer and probably not tomorrow's flat answer.
Figure 2 describes the different iterations of flat data, and how these snapshots and iterations tie together to form the movie of the ever-evolving nature of data.
Figure 2 - Flat, one-dimensional data and multi-dimensional time-based data evolution
And understanding the ever-changing and evolving nature of data is key to transforming data into information. Indeed one has to understand the cycles of change of data in order to pretend to gain information.
Our brain is a superior machine for this transformation as we are able to subconsciously and automatically model, infer, expand, project and otherwise manipulate data as well as recognize patterns, differences, similarities and other properties of these data, allowing us a higher level of understanding and abstraction.
This is how our brain transforms huge amounts of data into information, by processing it, by transforming it.
The data about how to transform data, data about data, is called metadata.
Data is the raw material, the metadata is the way to manipulate the data, and information is the result of this manipulation, much like the fruit is the raw material and the juice is the result of its manipulation.
Figure 3 - Metadata as key component to understand data evolution
We are dealing here with very complex concepts and mechanisms in information technology. This is a fairly new science; one of the oldest theoretical studies of data was Shannon's entropy theorem. Shannon studied how much information was and could be contained in data.
Again, data about data, metadata.
Metadata is key to transforming data into information.
Figure 4 - Detail on metadata as key component from one iteration to another of flat data
Although our brains are expert systems in managing metadata and we are using heuristics so effective that we touch the very own nature of the magic of human kind, when transposed to Information Technology we have very clear and very practical limitations.
For example, it is simply not practical to keep, along with the data, all the previous versions of that data along with the reasons why this data had to change and generally all the context surrounding that change, and this for every iteration and version of the data. This is impractical because it would imply:
- An immense storage space
- An Artificial Intelligence system that is expert at sorting through the universe of metadata and extracting the one used to reconstruct in real time the lifecycle of the data
Too bad, as this is really what could eliminate data flatness and put back into perspective the animated nature of data in all its glory.
Okay, so what is it that we could do then? What is the minimum metadata that is absolutely key to understanding a little bit how this data came about?
Stay tuned for next installment!!
[video] Logging and Monitoring with @Sematext Founder @OtisG | @DevOpsSummit #DevOps #Logging #Monitoring
"We got started as search consultants. On the services side of the business we have help organizations save time and save money when they hit issues that everyone more or less hits when their data grows," noted Otis Gospodnetić, Founder of Sematext, in this SYS-CON.tv interview at @DevOpsSummit, held June 9-11, 2015, at the Javits Center in New York City.
Jul. 28, 2015 10:45 PM EDT Reads: 998
"We've just seen a huge influx of new partners coming into our ecosystem, and partners building unique offerings on top of our API set," explained Seth Bostock, Chief Executive Officer at IndependenceIT, in this SYS-CON.tv interview at 16th Cloud Expo, held June 9-11, 2015, at the Javits Center in New York City.
Jul. 28, 2015 08:00 PM EDT Reads: 581
[slides] Storage for Docker Containers By @OnModulus | @DevOpsSummit #DevOps #Docker #Containers #Microservices
Learn how to solve the problem of keeping files in sync between multiple Docker containers. In his session at 16th Cloud Expo, Aaron Brongersma, Senior Infrastructure Engineer at Modulus, discussed using rsync, GlusterFS, EBS and Bit Torrent Sync. He broke down the tools that are needed to help create a seamless user experience. In the end, can we have an environment where we can easily move Docker containers, servers, and volumes without impacting our applications? He shared his results so yo...
Jul. 28, 2015 07:15 PM EDT Reads: 711
[slides] A New Architecture for the Internet of Things By @JKirklan | @ThingsExpo @RedHatNews #IoT #M2M #InternetOfThings
Explosive growth in connected devices. Enormous amounts of data for collection and analysis. Critical use of data for split-second decision making and actionable information. All three are factors in making the Internet of Things a reality. Yet, any one factor would have an IT organization pondering its infrastructure strategy. How should your organization enhance its IT framework to enable an Internet of Things implementation? In his session at @ThingsExpo, James Kirkland, Red Hat's Chief Arch...
Jul. 28, 2015 06:30 PM EDT Reads: 1,369
Modern DevOps Tool Kit By @Logentries and @NewRelic | @DevOpsSummit #DevOps #Containers #Microservices
Auto-scaling environments, micro-service architectures and globally-distributed teams are just three common examples of why organizations today need automation and interoperability more than ever. But is interoperability something we simply start doing, or does it require a reexamination of our processes? And can we really improve our processes without first making interoperability a requirement for how we choose our tools?
Jul. 28, 2015 06:15 PM EDT Reads: 287
Microservices are individual units of executable code that work within a limited framework. They are extremely useful when placed within an architecture of numerous microservices. On June 24th, 2015 I attended a webinar titled “How to Share Share-Nothing Microservices,” hosted by Jason Bloomberg, the President of Intellyx, and Scott Edwards, Director Product Marketing for Service Virtualization at CA Technologies. The webinar explained how to use microservices to your advantage in order to deliv...
Jul. 28, 2015 06:00 PM EDT Reads: 919
Approved this February by the Internet Engineering Task Force (IETF), HTTP/2 is the first major update to HTTP since 1999, when HTTP/1.1 was standardized. Designed with performance in mind, one of the biggest goals of HTTP/2 implementation is to decrease latency while maintaining a high-level compatibility with HTTP/1.1. Though not all testing activities will be impacted by the new protocol, it's important for testers to be aware of any changes moving forward.
Jul. 28, 2015 04:00 PM EDT Reads: 136
[slides] Workloads and Public Cloud at @CloudExpo By @utollwi | @ProfitBricksUSA #DevOps #Containers #Microservices
Public Cloud IaaS started its life in the developer and startup communities and has grown rapidly to a $20B+ industry, but it still pales in comparison to how much is spent worldwide on IT: $3.6 trillion. In fact, there are 8.6 million data centers worldwide, the reality is many small and medium sized business have server closets and colocation footprints filled with servers and storage gear. While on-premise environment virtualization may have peaked at 75%, the Public Cloud has lagged in adop...
Jul. 28, 2015 04:00 PM EDT Reads: 2,178
How do you securely enable access to your applications in AWS without exposing any attack surfaces? The answer is usually very complicated because application environments morph over time in response to growing requirements from your employee base, your partners and your customers. In his session at @DevOpsSummit, Haseeb Budhani, CEO and Co-founder of Soha, shared five common approaches that DevOps teams follow to secure access to applications deployed in AWS, Azure, etc., and the friction an...
Jul. 28, 2015 03:30 PM EDT Reads: 480
The Software Defined Data Center (SDDC), which enables organizations to seamlessly run in a hybrid cloud model (public + private cloud), is here to stay. IDC estimates that the software-defined networking market will be valued at $3.7 billion by 2016. Security is a key component and benefit of the SDDC, and offers an opportunity to build security 'from the ground up' and weave it into the environment from day one. In his session at 16th Cloud Expo, Reuven Harrison, CTO and Co-Founder of Tufin,...
Jul. 28, 2015 03:00 PM EDT Reads: 457
Take the Long View with Digital Transformation By @IoT2040 | @ThingsExpo #IoT #M2M #API #Microservices #InternetOfThings
Digital Transformation is the ultimate goal of cloud computing and related initiatives. The phrase is certainly not a precise one, and as subject to hand-waving and distortion as any high-falutin' terminology in the world of information technology. Yet it is an excellent choice of words to describe what enterprise IT—and by extension, organizations in general—should be working to achieve. Digital Transformation means: handling all the data types being found and created in the organizat...
Jul. 28, 2015 03:00 PM EDT Reads: 1,060
Jul. 28, 2015 02:00 PM EDT Reads: 272
Discussions about cloud computing are evolving into discussions about enterprise IT in general. As enterprises increasingly migrate toward their own unique clouds, new issues such as the use of containers and microservices emerge to keep things interesting. In this Power Panel at 16th Cloud Expo, moderated by Conference Chair Roger Strukhoff, panelists addressed the state of cloud computing today, and what enterprise IT professionals need to know about how the latest topics and trends affect t...
Jul. 28, 2015 02:00 PM EDT Reads: 1,156
This week, I joined SOASTA as Senior Vice President of Performance Analytics. Given my background in cloud computing and distributed systems operations — you may have read my blogs on CNET or GigaOm — this may surprise you, but I want to explain why this is the perfect time to take on this opportunity with this team. In fact, that’s probably the best way to break this down. To explain why I’d leave the world of infrastructure and code for the world of data and analytics, let’s explore the timing...
Jul. 28, 2015 01:45 PM EDT Reads: 326
[video] Infrastructure as a Toolbox By @SoftLayer at @CloudExpo New York | #IoT #API #Containers #Microservices
Countless business models have spawned from the IaaS industry. Resell Web hosting, blogs, public cloud, and on and on. With the overwhelming amount of tools available to us, it's sometimes easy to overlook that many of them are just new skins of resources we've had for a long time. In his General Session at 16th Cloud Expo, Phil Jackson, Lead Technology Evangelist at SoftLayer, broke down what we've got to work with and discuss the benefits and pitfalls to discover how we can best use them to d...
Jul. 28, 2015 01:00 PM EDT Reads: 1,938
[session] The Container New World By @KeGilpin | @DevOpsSummit #DevOps #Docker #Containers #Microservices
Containers are changing the security landscape for software development and deployment. As with any security solutions, security approaches that work for developers, operations personnel and security professionals is a requirement. In his session at DevOps Summit, Kevin Gilpin, CTO and Co-Founder of Conjur, will discuss various security considerations for container-based infrastructure and related DevOps workflows.
Jul. 28, 2015 01:00 PM EDT Reads: 1,046
Microservices Total Cost of Ownership: Too Soon? By @Aruna13 | @DevOpsSummit #DevOps #Docker #Containers #Microservices
Microservices are hot. And for good reason. To compete in today’s fast-moving application economy, it makes sense to break large, monolithic applications down into discrete functional units. Such an approach makes it easier to update and add functionalities (text-messaging a customer, calculating sales tax for a specific geography, etc.) and get those updates / adds into production fast. In fact, some would argue that microservices are a prerequisite for true continuous delivery. But is it too...
Jul. 28, 2015 01:00 PM EDT Reads: 660
You often hear the two titles of "DevOps" and "Immutable Infrastructure" used independently. In his session at DevOps Summit, John Willis, Technical Evangelist for Docker, covered the union between the two topics and why this is important. He provided an overview of Immutable Infrastructure then showed how an Immutable Continuous Delivery pipeline can be applied as a best practice for "DevOps." He ended the session with some interesting case study examples.
Jul. 28, 2015 12:15 PM EDT Reads: 109
SYS-CON Events announced today that HPM Networks will exhibit at the 17th International Cloud Expo®, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. For 20 years, HPM Networks has been integrating technology solutions that solve complex business challenges. HPM Networks has designed solutions for both SMB and enterprise customers throughout the San Francisco Bay Area.
Jul. 28, 2015 11:45 AM EDT Reads: 403
Puppet Labs has published their annual State of DevOps report and it is loaded with interesting information as always. Last year’s report brought home the point that DevOps was becoming widely accepted in the enterprise. This year’s report further validates that point and provides us with some interesting insights from surveying a wide variety of companies in different phases of their DevOps journey.
Jul. 28, 2015 10:00 AM EDT Reads: 147