Welcome!

Microservices Expo Authors: Elizabeth White, Liz McMillan, Pat Romanski, Craig Lowell, Aruna Ravichandran

Related Topics: Containers Expo Blog, Java IoT, Microservices Expo, Microsoft Cloud, @CloudExpo, @BigDataExpo, SDN Journal

Containers Expo Blog: Blog Feed Post

Bare Metal Blog. When Things Go Wrong.

It’s about hardware

If you’re like most people, when you notice something really odd about your body, the first thought to enter your brain is not “I need to call the doctor.” Of course let me clarify, if you look down after a fall, and think “My arm really shouldn’t bend like that”, then yeah, you call a doctor right away. But if you didn’t fall and go “there’s this tingle in my right arm”, the first thing you do is attempt to analyze. Then, if you want to know all of the really worst things in the world it might be, you research online at a site like WebMD. If, after reasonable thought and possible research, you cannot place a reason for your problem, you might go see your doctor. Or you might try some aspirin, depending upon the problem, and your level of discomfort.

If you decide to go see a doctor, you don’t want him to take an XRay, glance at a blurry image, and pronounce that you have two days to live. You’re going to want him to do a thorough job of examining anything more than “you slept on it wrong, it’ll be better tomorrow”. Because you want to – and you want your doctor to – work from an informed position.

But do you give your hardware the same opportunity?

Just like your body shows odd symptoms or even has a system failure, so too for the hardware in your datacenter. While it would be wonderful if a device could last forever, see my Mean Time Between Failures post in the Bare Metal Series for more reasonable expectations.

<Disclaimer> As always, I am an F5 employee, and I know F5 gear better than other vendors. From here on, I will talk primarily about what is available to diagnose problems on an F5 device. Vendor support for these tools varies, check with your vendor to find out if/how you can achieve the same ends. </Disclaimer>

It’s about hardware.
Somewhere on the system, there is going to be a hardware diagnostics tool. In the case of F5 gear, it is called End User Diagnostics (EUD), and it provides you with a solid battery of self-diagnostics that can be used to see if the hardware is functioning well. Here is the main menu from the tool:

Notice that it can test the RAM, the LCD display, SFPs, the indicator lights on the chassis, overall system, and system sensors (like temperature). But it can go beyond these tests, checking the internal path packets traverse through the hardware, the hardware-used memory (PVA), SSL processing (since SSL is offloaded to specialized hardware), FIPS processing, compression, disk drives, file systems on the drives… It’s a pretty solid picture of what might be wrong with the system. While we hope you never need it, reality is that hardware wears down, gets dirty power, or on occasion fails in spite of burn-in. So for those times, you have the tool available.

Notice that EUD doesn’t have an option to quit without rebooting, and that’s not the only caveat. While I could give you the other details, like you need to disconnect network cables while running EUD, I’ll just point out that http://ask.f5.com has a lot of information about the tool if you have an account, and F5 offers training in using it also. Again, we do our best to make sure you’ll never need it but know it does happen, so want you prepared. It is strongly recommended that you download the latest version and read the release notes also.

But it’s software too!
No complex piece of computing machinery runs on straight hardware anymore. Whether you recognize it as software or not, all computer systems – including all ADCs – use software to accomplish some goals. In F5 gear, a fair share of processing is either shared hardware/software or straight software, and as you might imagine, the software can have issues from configuration on that can cause odd behavior.

For that bit of the puzzle, F5 has long had the tool for the job… QKView* runs on the machine and collects a ton of data. The results of QKView can be sent to technical support upon request, but also can be uploaded to a user diagnostics site. More on that in a moment. qkview runs across the system, picking up important (but non security-related) information and puts it all together in a tarball. “What good is that?” a bunch of you must be asking. And that’s the great part, since normally that would be a valid question. The logs, configs, error dumps are all available to you on the device, so what use is making them less available in a tarball? That’s where the next part comes in…

I cannot stress strongly enough, if you are an F5 customer considering using qkview, please go to ask.f5.com and download the latest version. Improvements in performance, what data is gathered, even organization of data inside the tarball are happening pretty regularly, and using the newest version will help insure that you have the most relevant data in the most efficient form.

But it’s really complex software…
F5 gear is a marriage of blazingly fast, bullet proof hardware with highly optimized software. To create a system that is not only that complex, but adds in features like the ability to store multiple versions of the software and boot the one of choice at any given time, and pluggable software modules that do a variety of application delivery and application security functions for you, well, that takes a lot of software. Never fear, all of our software has rigorous QC applied, just like our hardware does, but there’s a lot of it interacting, and I have never met the device whose designers knew before hand the array of uses that customers will find to put it to. Every network is different, every application architecture is different, and thus the usage of every single ADC deployed is different. Well, not every single one, since most customers use clustering sooner or later, but more than half of them, for certain.

That is why QkView output is a tar file There’s a bunch of information about how all the various software and hardware parts are communicating in those files, what’s gone wrong, how the device is configured… Just a ton of information. In fact, with versioning differences (if software changed, often what it reports changes), it was difficult to offer up a cohesive application on the BIG-IP to analyze these files.

Enter iHealth, a free (registration required to keep it to people with legitimate uses) qkview analyzer.  There are a large variety of reasons that F5 chose to go to a centralized online analyzer over a standalone tool. I’ll hit on a couple of them for you, they’re the ones I think you’ll care the most about.

1. The online tool offers manipulable graphical output. In short, you can navigate data organized in a natural way, look at what’s important to you, and get back to fixing problems faster. Generated charts are also great tools for management presentation to point out problem areas or talk up how much traffic the device is handling.

2.  The online tool can utilize the information from thousands of deployed devices to show you where you’ve made common configuration errors or point out potential future problems. It’s like chatting with thousands of your widespread peers about qkview output and getting free advice.

3.  The heuristics database that checks configurations and offers advice/tells you how to resolve issues is always up to date. You don’t have to update it before checking a qkview file.

But as always, a picture is worth a thousand words, so I’ll offer you a couple thousand words’ worth.

When the qkview file is uploaded and analyzed, you get the iHealth summary page:

This serves as a starting point to explore in more detail, and offers totals for how many devices have been defined, what add-on modules are licensed, version information, etc.

Next let’s take a look at the diagnostics section, the one that will interest most users (some users utilize iHealth to performance tune their network, and for those customers, diagnostics is far less used):

Notice how it' has issues divided up by severity? And it offers links to how to fix them. Useful when there’s trouble and you’re in a hurry.

It builds this handy list from information stored in the online app – information that can be updated as needed. That means the app is more responsive to your needs than an on-device tool might be.

In the end, it’s about serving traffic. Reliably.

All of these tools – End User Diagnostics, qkview, and iHealth are out to help with one thing… Helping you (and F5 tech support when necessary) figure out what’s really wrong and fix it, and helping you proactively fix things that might be wrong for the future. And all of that is to simply support the need to keep applications on-line and performing well. While they are not much use if your ADC is a doorstop, they’re invaluable if the ADC is a cornerstone of your datacenter, and cut hours, in some cases days off of troubleshooting and repair timelines.

And remember, all are free tools for you to use, just one part of the overall quality plan at F5.

Read the original blog entry...

More Stories By Don MacVittie

Don MacVittie is founder of Ingrained Technology, A technical advocacy and software development consultancy. He has experience in application development, architecture, infrastructure, technical writing,DevOps, and IT management. MacVittie holds a B.S. in Computer Science from Northern Michigan University, and an M.S. in Computer Science from Nova Southeastern University.

@MicroservicesExpo Stories
SYS-CON Events announced today that Super Micro Computer, Inc., a global leader in Embedded and IoT solutions, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 7-9, 2017, at the Javits Center in New York City, NY. Supermicro (NASDAQ: SMCI), the leading innovator in high-performance, high-efficiency server technology, is a premier provider of advanced server Building Block Solutions® for Data Center, Cloud Computing, Enterprise IT, Hadoop/Big Data, HPC and E...
SYS-CON Events announced today that Linux Academy, the foremost online Linux and cloud training platform and community, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Linux Academy was founded on the belief that providing high-quality, in-depth training should be available at an affordable price. Industry leaders in quality training, provided services, and student certification passes, its goal is to c...
Synthetic monitoring is hardly a new technology. It’s been around almost as long as the commercial World Wide Web has. But the importance of monitoring the performance and availability of a web application by simulating users’ interactions with that application, from around the globe, has never been more important. We’ve seen prominent vendors in the broad APM space add this technology with new development or partnerships just in the last 18 months.
The unique combination of Amazon Web Services and Cloud Raxak, a Gartner Cool Vendor in IT Automation, provides a seamless and cost-effective way of securely moving on-premise IT workloads to Amazon Web Services. Any enterprise can now leverage the cloud, manage risk, and maintain continuous security compliance. Forrester's analysis shows that enterprises need automated security to lower security risk and decrease IT operational costs. Through the seamless integration into Amazon Web Services, ...
All organizations that did not originate this moment have a pre-existing culture as well as legacy technology and processes that can be more or less amenable to DevOps implementation. That organizational culture is influenced by the personalities and management styles of Executive Management, the wider culture in which the organization is situated, and the personalities of key team members at all levels of the organization. This culture and entrenched interests usually throw a wrench in the work...
A lot of time, resources and energy has been invested over the past few years on de-siloing development and operations. And with good reason. DevOps is enabling organizations to more aggressively increase their digital agility, while at the same time reducing digital costs and risks. But as 2017 approaches, the hottest trends in DevOps aren’t specifically about dev or ops. They’re about testing, security, and metrics.
Containers have changed the mind of IT in DevOps. They enable developers to work with dev, test, stage and production environments identically. Containers provide the right abstraction for microservices and many cloud platforms have integrated them into deployment pipelines. DevOps and containers together help companies achieve their business goals faster and more effectively. In his session at DevOps Summit, Ruslan Synytsky, CEO and Co-founder of Jelastic, reviewed the current landscape of Dev...
True Story. Over the past few years, Fannie Mae transformed the way in which they delivered software. Deploys increased from 1,200/month to 15,000/month. At the same time, productivity increased by 28% while reducing costs by 30%. But, how did they do it? During the All Day DevOps conference, over 13,500 practitioners from around the world to learn from their peers in the industry. Barry Snyder, Senior Manager of DevOps at Fannie Mae, was one of 57 practitioners who shared his real world journe...
"Plutora provides release and testing environment capabilities to the enterprise," explained Dalibor Siroky, Director and Co-founder of Plutora, in this SYS-CON.tv interview at @DevOpsSummit, held June 9-11, 2015, at the Javits Center in New York City.
Docker containers have brought great opportunities to shorten the deployment process through continuous integration and the delivery of applications and microservices. This applies equally to enterprise data centers as well as the cloud. In his session at 20th Cloud Expo, Jari Kolehmainen, founder and CTO of Kontena, will discuss solutions and benefits of a deeply integrated deployment pipeline using technologies such as container management platforms, Docker containers, and the drone.io Cl tool...
In his session at 19th Cloud Expo, Claude Remillard, Principal Program Manager in Developer Division at Microsoft, contrasted how his team used config as code and immutable patterns for continuous delivery of microservices and apps to the cloud. He showed how the immutable patterns helps developers do away with most of the complexity of config as code-enabling scenarios such as rollback, zero downtime upgrades with far greater simplicity. He also demoed building immutable pipelines in the cloud ...
Internet of @ThingsExpo, taking place June 6-8, 2017 at the Javits Center in New York City, New York, is co-located with the 20th International Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. @ThingsExpo New York Call for Papers is now open.
DevOps is being widely accepted (if not fully adopted) as essential in enterprise IT. But as Enterprise DevOps gains maturity, expands scope, and increases velocity, the need for data-driven decisions across teams becomes more acute. DevOps teams in any modern business must wrangle the ‘digital exhaust’ from the delivery toolchain, "pervasive" and "cognitive" computing, APIs and services, mobile devices and applications, the Internet of Things, and now even blockchain. In this power panel at @...
SYS-CON Events announced today that Catchpoint Systems, Inc., a provider of innovative web and infrastructure monitoring solutions, has been named “Silver Sponsor” of SYS-CON's DevOps Summit at 18th Cloud Expo New York, which will take place June 7-9, 2016, at the Javits Center in New York City, NY. Catchpoint is a leading Digital Performance Analytics company that provides unparalleled insight into customer-critical services to help consistently deliver an amazing customer experience. Designed ...
As 2016 approaches its end, the time to prepare for the year ahead is now! Following our own advice, we sat down with three XebiaLabs thought leaders–Andrew Phillips, Tim Buntel, and TJ Randall–and asked what they think the future has in store for the DevOps world. In 2017, we’ll see a new wave of “next gen platform” projects focused on container orchestration frameworks such as Kubernetes, and re-tooled PaaS platforms such as OpenShift or Cloud Foundry. Acceptance of the need for a cross-machi...
When building DevOps or continuous delivery practices you can learn a great deal from others. What choices did they make, what practices did they put in place, and how did they connect the dots? At Sonatype, we pulled together a set of 21 reference architectures for folks building continuous delivery and DevOps practices using Docker. Why? After 3,000 DevOps professionals attended our webinar on "Continuous Integration using Docker" discussing just one reference architecture example, we recogn...
When you focus on a journey from up-close, you look at your own technical and cultural history and how you changed it for the benefit of the customer. This was our starting point: too many integration issues, 13 SWP days and very long cycles. It was evident that in this fast-paced industry we could no longer afford this reality. We needed something that would take us beyond reducing the development lifecycles, CI and Agile methodologies. We made a fundamental difference, even changed our culture...
The proper isolation of resources is essential for multi-tenant environments. The traditional approach to isolate resources is, however, rather heavyweight. In his session at 18th Cloud Expo, Igor Drobiazko, co-founder of elastic.io, drew upon his own experience with operating a Docker container-based infrastructure on a large scale and present a lightweight solution for resource isolation using microservices. He also discussed the implementation of microservices in data and application integrat...
In his General Session at DevOps Summit, Asaf Yigal, Co-Founder & VP of Product at Logz.io, will explore the value of Kibana 4 for log analysis and will give a real live, hands-on tutorial on how to set up Kibana 4 and get the most out of Apache log files. He will examine three use cases: IT operations, business intelligence, and security and compliance. This is a hands-on session that will require participants to bring their own laptops, and we will provide the rest.
"We're bringing out a new application monitoring system to the DevOps space. It manages large enterprise applications that are distributed throughout a node in many enterprises and we manage them as one collective," explained Kevin Barnes, President of eCube Systems, in this SYS-CON.tv interview at DevOps at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York City, NY.