Click here to close now.


Microservices Expo Authors: Pat Romanski, AppDynamics Blog, Liz McMillan, XebiaLabs Blog, Elizabeth White

Related Topics: Cloud Security, Java IoT, Microservices Expo, Linux Containers, Open Source Cloud, Containers Expo Blog

Cloud Security: Article

The Definitive Guide to Heartbleed and Enterprise Mobility

There has been a lot of media attention on Heartbleed and as always that means a lot of bizarre and often conflicting advice

By Peter Rogers, Principal Mobility Architect, Cognizant

There has been a lot of media attention on Heartbleed and as always that means a lot of bizarre and often conflicting advice.  I sat down (for a very long time) to find the truth of the matter and bring it together in one single article for those involved in enterprise mobility.

What is Heartbleed?
"The [Heartbleed] bug is in the OpenSSL's implementation of the TLS/DTLS (transport layer security protocols) heartbeat extension (RFC6520). When it is exploited it leads to the leak of memory contents from the server to the client, and from the client to the server. The bug was introduced to OpenSSL in December 2011 and has been out in the wild since OpenSSL release 1.0.1 on 14th of March 2012. OpenSSL 1.0.1g released on 7th of April 2014 fixes the bug."

"An attacker can retrieve a block of memory of the server up to 64kb. There is no limit on the number of attacks that can be performed. The attacker has no control over the memory region where the block is read from. Sensitive information that can be obtained is: SSL private keys; Basic authorization strings (username / password combinations); and source code [most important also cookies]. This bug affects both sides of the connection. Not only will client certificates not save you from having to update your server certificate, they can be read from the client (along with your username, password etc.) by any server you connect to."

"The Heartbleed bug is an example of Buffer Over-Read and that means that the offending OpenSSL heartbeat code does not check that this length is the actual length sent in the heartbeat request, allowing the request to ask for more data than it should be able to retrieve. The code then copies the amount of data indicated by the length from the incoming message to the outgoing message. If the length is longer than the incoming message, the software just keeps copying data past the end of the message. Since the length variable is 16 bits, you can request up to 65,535 bytes from memory. The data that lives past the end of the incoming message is from a kind of no-man's land that the program should not be accessing and may contain data left behind from other parts of OpenSSL."

The non-technical explanation from XKCD is slightly easier to follow (

Here is a very good FAQ and an actual screen shot of a Heartbleed attack from Fox-IT.

The important thing to note is that not only can a client potentially read data from a server but the converse is also true. This impacts mobile clients too and although Apple claim they are safe there were allegedly concerns around Android 4.1.1. If a client application is vulnerable on a mobile or desktop device then equally the server can act as a honey trap to steal information from the client. Web browser software and IM clients are the biggest concern, so please make sure you don't use obscure and vulnerable software.

Who is stealing what and why?

The important question is how we got into this mess. The first thing to point out is that this is not a floor in SSL/TSP, instead it is an implementation problem in the open source OpenSSL library.  A major contributing factor has been that TLS versions 1.2 came with OpenSSL 1.0.1 and after the BEAST attack everyone rushed to implement TLS 1.2. The current theory doing the rounds is that the NSA had already ‘found ways around' SSL some time ago and this could have been by taking advantage of the Heartbleed bug. It is however noted that Man in the Middle (MITM) techniques are far easier, especially as some large companies act as their own Certificate Authority (CA). Edward Snowden leaked the existence of BULLRUN (a highly classified decryption program run by the NSA) and subsequently Edgehill (a similar program run by GCHQ) which subsequently prompted Google to double the size of their encryptions keys. We know that a number of companies knew about the vulnerability before it went public because they stated they had applied the patches at least a week ago. If you think about it a minute then OpenSSL is open source code which meant anybody could have worked it out and kept it a secret.

They say the exploit leaves no trace and so nobody actually knows if they have been targeted. Actually that statement isn't strictly true. A single heartbeat can potentially grab 64K of memory however an attacker can either keep reconnecting or during an active TLS connection keep requesting arbitrary number of 64 kilobyte chunks of memory content. There are now companies who are configuring their network monitoring software to watch for abnormal behaviour such as Fox-IT, who have developed a set of Snort signatures to detect successful exploitation of the bug.

The most important thing is what happens next and most hackers will be after what classes as primary key material.

"These are the crown jewels, the encryption keys themselves. Leaked secret keys allows the attacker to decrypt any past and future traffic to the protected services and to impersonate the service at will. Any protection given by the encryption and the signatures in the X.509 certificates can be bypassed. Recovery from this leak requires patching the vulnerability, revocation of the compromised keys and reissuing and redistributing new keys. Even doing all this will still leave any traffic intercepted by the attacker in the past still vulnerable to decryption. All this has to be done by the owners of the services."

Let's imagine that a service provider has been using a vulnerable version of Open SSL 1.0.1 and the types of hacker that may target them based on time ranges:

  1. Soon after the vulnerability was released (14th of March 2012)
  2. Two or three weeks before the announcement was made public but was still being discussed in hushed circles
  3. Ever since the announcement (April 7th)
  4. Ever since the Heartbleed proof of concepts went live (April 8th)

The first group are likely to be very serious hackers or even possibly Government sponsored activities which are nothing to worry about unless you are of significant danger to anyone. The second group are likely to be serious hackers or service providers testing out vulnerabilities in their systems (and potentially partner solutions) after they learn of the issues. The third group is the rest of the world and depends on the general availability of OpenSSL vulnerability scanning software at that date. The fourth group covers hobbyist hackers because the proof of concepts are open source and highly available.

Unfortunately as of April 8th then anyone in the world can now test for this vulnerability using this widely shared Python Script ( and that puts it in the domain of hobbyists. Anyone can now dump a bit of RAM from a vulnerable server. It would definitely take a professional who can write their own script to make repeated heartbeats attacks and then got lucky with the timing of a server restart on a vulnerable operating system to be able to pull out a private encryption key. However, even an amateur can use Session Hijacking which means you look for cookies containing session IDs. In a single grab of memory then you are equally as likely to obtain a session ID or a username and password combination, and this is slightly harder to track for abnormal behaviour.

An example of which is running the Python proof of concept against a vulnerable JIRA ticketing system and pulling out a JSESSIONID (which is JIRA's way of tracking your HTTP session). If the system requires authentication then you can just insert the stolen cookie into the browser and become that user on the JIRA installation. (

Can you actually steal a private key?

The Heartbleed bug is an example of buffer over-read and here is the offending code for the programmers amongst us:

p = &s->s3->[0]


hbtype = *p++;
n2s(p, payload);
pl = p;


buffer = OPENSSL_malloc(1 + 2 + payload + padding);
bp = buffer;


memcpy(bp, pl, payload);

In NGINX, the keys are loaded immediately when the process is started, which puts the keys very low in the memory space. This makes it unlikely that incoming requests will be allocated with a lower address space. If NGINX is reloaded, it starts a new process and loads the keys right away, putting them at a low address. On NGINX you can therefore retrieve user credentials, cookies and even public parts of the certificate but so far not the private part. It all depends therefore on the operating system and how often the servers are restarted. CloudFlare have even issued a challenge by leaving a server running nginx-1.5.13 linked against OpenSSL 1.0.1.f on Ubuntu 13.10 x86_6

That said there appears to be a proof of concept working on Apache for a first request which corresponds with the server being rebooted.

The Certificate Revocation Tsunami
There is a genuine importance to the CloudFlare challenge and that is that if private keys can be stolen then there will be a whole lot of companies requesting certificates all at once leading to a so called Certificate Revocation Tsunami. Certificate Authorities are supposed to revoke certificates within 24 hours if there is any evidence of a key compromise. However, even if all of the affected certificates were to be revoked, contemporary web browser software handles certificate revocation poorly. For example, some browsers only perform OCSP revocation checks for Extended Validation certificates, while others ignore certificate revocation lists completely. The other concern is the rush to create new certificates will lead to incorrect certificates being generated which in turn leads to Man In The Middle (MITM) exploits down the line.

"The certificate authority infrastructure was never built to do a mass revocation of this many certificates and because of the way the infrastructure is built, if you did do a mass revocation of millions of certificates it would significantly slow down the performance of the Internet itself, which is potentially very, very bad."

"Such is the haste to fix the fallout of the Heartbleed bug, some certificate authorities and website administrators have been making careless mistakes. PayPal's Hosted Message Applications, such as the one at, are now using Extended Validation certificates issued by VeriSign on 10 April 2014. The CAB Forum requires certificate authorities to adhere to a stringent set of guidelines [pdf] when issuing EV certificates, and it is the CA's responsibility to verify the accuracy of the information in the certificate. In particular, they must verify that the legal name of the subject in an EV certificate matches the name which appears on official government records. However, this verification does not appear to have been performed correctly in the case of these certificates."

How can I protect myself?

Here is a list of all the password changing advice from the various companies and the results of a vulnerable scan performed on April 8, 12:00 UTC.

"Only 30,000 of the 500,000+ SSL certificates affected by the Heartbleed bug have been reissued up until today, and even fewer certificates have been revoked. Some of the first sites to deploy newly issued certificates in response to the OpenSSL vulnerability included Yahoo, Adobe, CloudFlare, DuckDuckGo, GitHub, Reddit , Launchpad, PayPal, Netflix and Amazon's CloudFront content delivery network."

Fox-IT have actually listed a set of vulnerability tests, along with advice of the next steps and have developed a set of Snort signatures to detect successful exploitation of the bug.

Vendors are recommended to perform the following operations:

  • Upgrade the OpenSSL version to 1.0.1g (you can also recompile your OpenSSL 1.0.1 with the compile time option -DOPENSSL_NO_HEARTBEATS)
  • Request revocation of the current SSL certificate
  • Regenerate your private key
  • Request and replace the SSL certificate
  • Restart your services

Upgrading to OpenSSL 1.0.1g  does not automatically restart your services and likewise certificate revocation and replacement is a whole different step from ‘patching your server'.

  • If a vendor does not restart their services then their sessions are still vulnerable
  • Even if a vendor does revoke their certificates then some web browsers ignore the revocation
  • If a vendor does not replace their certificates then if their private key was stolen then they are wide open

If we look at the situation with WordPress then it appears they applied the patches but may not have replaced their certificates yet. I can also see smaller companies forgetting to restart their services after apply the OpenSSL 1.0.1g upgrade.

Should I change my password?

If you look at this list of vendors and their statements then the following rules apply:

  • You are safe if a vendor never used OpenSSL (remember SSL isn't broken itself, just one implementation of it including heartbeats)
  • You are safe if a vendor used an earlier version of OpenSSL (0.9.8 / 1.0.0)
  • You are moderately safe if a vendor upgraded to OpenSSL 1.0.1g, restarted their services and issued new certificates, before the public announcement on April 7th (apart from in the case of serious hackers)
  • You are at risk if a vendor upgraded to OpenSSL 1.0.1g, restarted their services and issued new certificates, sometime after April 7th (the longer the gap the higher the risk) and should change your passwords
  • You are at risk if a vendor upgraded to OpenSSL 1.0.1g, restarted their services and has not issued new certificates. You have a personal decision to make regarding changing passwords.

o   There is a small risk a private key could have been stolen and so even if you change your password then the site could still be vulnerable and you just gave your new password away

o   That said if the key did not get stolen then changing passwords prevents a hacker using your stolen details in the future.

  • You are at risk if a vendor either did not upgrade to OpenSSL 1.0.1g across all their services or forgot to restart their server. It is probably pretty pointless doing anything yet if that is the case.
  • You are at risk if a vendor made a nebulous statement like "We patched our servers" but was not drawn about restarting their services or issuing new certificates and should look to Internet forums
  • You are risk if a vendor has revoked their certificate but not issues a new certificate yet. Your web browser may ignore the certificate revocation and be open to MITM attacks.
  • You are at risk if you used client side software using a vulnerable version of OpenSSL and could have leaked confidential information from your desktop or mobile device

o   Double check the application that you use on Internet forums especially web browsers and IM clients

  • Most vendors are going to tell you to change your passwords anyway but check the details as above
  • Making your password really long is good practise but is no deterrent whatsoever from Heartbleed and you run the obvious risk of forgetting it or revealing it to hackers waiting for all the password resets

Imagine a hacker did target Vendor X, based on watching the charts of companies requesting new certificates and targeting those who haven't yet. Let us presume they were running Operating System X (all we know is that NGINX probably isn't vulnerable and Apache potentially is - but it would require specific timing) and just as a server rebooted the private key was added to the top of the OpenSSL memory stack and at that precise moment a hacker somehow managed to make a first request with an OpenSSL heartbeat and steal the private key. If we imagine that Vendor X are rebooting their servers regularly and maybe there was an insider involved then it becomes all the more probable. Let us now imagine the servers have been upgraded to OpenSSL 1.0.1g and all the services restarted but that the old certificates are still up there. That means if you reset your passwords then it makes no difference whatsoever because the hacker can decrypt all the traffic using their stolen private key. Changing your password actually means that the hacker now has your new password and you think you are actually safe. The hacker may have already decrypted everything on the site using their private key and have all your confidential information but let's assume they haven't. Vendor X then issue new certificates and does not bother going back through the last two years of logs to look for Heartbleed attacks. You have absolutely no idea that the hacker stole your new password and you carry on using the service and update it with your new credit card are completely surprised that your account has been hacked even though Vendor X patched their servers.

That said, I would actually be more worried by the news that the majority of the traffic performing heartbeat vulnerability scans is apparently coming from various Government (thankfully not ours) sponsored cybercrime activity around the world.

I would like to end by thanking all the Blogs that I have referenced in bringing together this hopefully definitive guide to Heartbleed. I am happy to hear your views on the matter so please feel free to email me.



Kevin Benedict Senior Analyst, Digital Transformation, EBA, Center for the Future of Work Cognizant View my profile on LinkedIn Learn about mobile strategies at Follow me on Twitter @krbenedict Browse the Mobile Solution Directory Join the Linkedin Group Strategic Enterprise Mobility Join the Google+ Community Mobile Enterprise Strategies Recommended Strategy Book Code Halos Recommended iPad App Code Halos for iPads

***Full Disclosure: These are my personal opinions. No company is silly enough to claim them. I am a mobility and digital transformation analyst, consultant and writer. I work with and have worked with many of the companies mentioned in my articles.

More Stories By Kevin Benedict

Kevin Benedict is the Senior Analyst for Digital Transformation at Cognizant, a writer, speaker and SAP Mentor Alumnus. Follow him on Twitter @krbenedict. He is a popular speaker around the world on the topic of digital transformation and enterprise mobility. He maintains a busy schedule researching, writing and speaking at events in North America, Asia and Europe. He has over 25 years of experience working in the enterprise IT solutions industry.

@MicroservicesExpo Stories
SYS-CON Events announced today that G2G3 will exhibit at SYS-CON's @DevOpsSummit Silicon Valley, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. Based on a collective appreciation for user experience, design, and technology, G2G3 is uniquely qualified and motivated to redefine how organizations and people engage in an increasingly digital world.
If you are new to Python, you might be confused about the different versions that are available. Although Python 3 is the latest generation of the language, many programmers still use Python 2.7, the final update to Python 2, which was released in 2010. There is currently no clear-cut answer to the question of which version of Python you should use; the decision depends on what you want to achieve. While Python 3 is clearly the future of the language, some programmers choose to remain with Py...
“All our customers are looking at the cloud ecosystem as an important part of their overall product strategy. Some see it evolve as a multi-cloud / hybrid cloud strategy, while others are embracing all forms of cloud offerings like PaaS, IaaS and SaaS in their solutions,” noted Suhas Joshi, Vice President – Technology, at Harbinger Group, in this exclusive Q&A with Cloud Expo Conference Chair Roger Strukhoff.
Opinions on how best to package and deliver applications are legion and, like many other aspects of the software world, are subject to recurring trend cycles. On the server-side, the current favorite is container delivery: a “full stack” approach in which your application and everything it needs to run are specified in a container definition. That definition is then “compiled” down to a container image and deployed by retrieving the image and passing it to a container runtime to create a running...
Clearly the way forward is to move to cloud be it bare metal, VMs or containers. One aspect of the current public clouds that is slowing this cloud migration is cloud lock-in. Every cloud vendor is trying to make it very difficult to move out once a customer has chosen their cloud. In his session at 17th Cloud Expo, Naveen Nimmu, CEO of Clouber, Inc., will advocate that making the inter-cloud migration as simple as changing airlines would help the entire industry to quickly adopt the cloud wit...
As the world moves towards more DevOps and microservices, application deployment to the cloud ought to become a lot simpler. The microservices architecture, which is the basis of many new age distributed systems such as OpenStack, NetFlix and so on, is at the heart of Cloud Foundry - a complete developer-oriented Platform as a Service (PaaS) that is IaaS agnostic and supports vCloud, OpenStack and AWS. In his session at 17th Cloud Expo, Raghavan "Rags" Srinivas, an Architect/Developer Evangeli...
Culture is the most important ingredient of DevOps. The challenge for most organizations is defining and communicating a vision of beneficial DevOps culture for their organizations, and then facilitating the changes needed to achieve that. Often this comes down to an ability to provide true leadership. As a CIO, are your direct reports IT managers or are they IT leaders? The hard truth is that many IT managers have risen through the ranks based on their technical skills, not their leadership ab...
Apps and devices shouldn't stop working when there's limited or no network connectivity. Learn how to bring data stored in a cloud database to the edge of the network (and back again) whenever an Internet connection is available. In his session at 17th Cloud Expo, Bradley Holt, Developer Advocate at IBM Cloud Data Services, will demonstrate techniques for replicating cloud databases with devices in order to build offline-first mobile or Internet of Things (IoT) apps that can provide a better, ...
Despite all the talk about public cloud services and DevOps, you would think the move to cloud for enterprises is clear and simple. But in a survey of almost 1,600 IT decision makers across the USA and Europe, the state of the cloud in enterprise today is still fraught with considerable frustration. The business case for apps in the real world cloud is hybrid, bimodal, multi-platform, and difficult. Download this report commissioned by NTT Communications to see the insightful findings – registra...
Application availability is not just the measure of “being up”. Many apps can claim that status. Technically they are running and responding to requests, but at a rate which users would certainly interpret as being down. That’s because excessive load times can (and will be) interpreted as “not available.” That’s why it’s important to view ensuring application availability as requiring attention to all its composite parts: scalability, performance, and security.
SYS-CON Events announced today that HPM Networks will exhibit at the 17th International Cloud Expo®, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. For 20 years, HPM Networks has been integrating technology solutions that solve complex business challenges. HPM Networks has designed solutions for both SMB and enterprise customers throughout the San Francisco Bay Area.
There once was a time when testers operated on their own, in isolation. They’d huddle as a group around the harsh glow of dozens of CRT monitors, clicking through GUIs and recording results. Anxiously, they’d wait for the developers in the other room to fix the bugs they found, yet they’d frequently leave the office disappointed as issues were filed away as non-critical. These teams would rarely interact, save for those scarce moments when a coder would wander in needing to reproduce a particula...
All we need to do is have our teams self-organize, and behold! Emergent design and/or architecture springs up out of the nothingness! If only it were that easy, right? I follow in the footsteps of so many people who have long wondered at the meanings of such simple words, as though they were dogma from on high. Emerge? Self-organizing? Profound, to be sure. But what do we really make of this sentence?
As we increasingly rely on technology to improve the quality and efficiency of our personal and professional lives, software has become the key business differentiator. Organizations must release software faster, as well as ensure the safety, security, and reliability of their applications. The option to make trade-offs between time and quality no longer exists—software teams must deliver quality and speed. To meet these expectations, businesses have shifted from more traditional approaches of d...
Information overload has infiltrated our lives. From the amount of news available and at our fingertips 24/7, to the endless choices we have when making a simple purchase, to the quantity of emails we receive on a given day, it’s increasingly difficult to sift out the details that really matter. When you envision your cloud monitoring system, the same thinking applies. We receive a lot of useless data that gets fed into the system, and the reality is no one in IT or DevOps has the time to manu...
Last month, my partners in crime – Carmen DeArdo from Nationwide, Lee Reid, my colleague from IBM and I wrote a 3-part series of blog posts on We titled our posts the Simple Math, Calculus and Art of DevOps. I would venture to say these are must-reads for any organization adopting DevOps. We examined all three ascpects – the Cultural, Automation and Process improvement side of DevOps. One of the key underlying themes of the three posts was the need for Cultural change – things like t...
It is with great pleasure that I am able to announce that Jesse Proudman, Blue Box CTO, has been appointed to the position of IBM Distinguished Engineer. Jesse is the first employee at Blue Box to receive this honor, and I’m quite confident there will be more to follow given the amazing talent at Blue Box with whom I have had the pleasure to collaborate. I’d like to provide an overview of what it means to become an IBM Distinguished Engineer.
I’ve been thinking a bit about microservices (μServices) recently. My immediate reaction is to think: “Isn’t this just yet another new term for the same stuff, Web Services->SOA->APIs->Microservices?” Followed shortly by the thought, “well yes it is, but there are some important differences/distinguishing factors.” Microservices is an evolutionary paradigm born out of the need for simplicity (i.e., get away from the ESB) and alignment with agile (think DevOps) and scalable (think Containerizati...
The cloud has reached mainstream IT. Those 18.7 million data centers out there (server closets to corporate data centers to colocation deployments) are moving to the cloud. In his session at 17th Cloud Expo, Achim Weiss, CEO & co-founder of ProfitBricks, will share how two companies – one in the U.S. and one in Germany – are achieving their goals with cloud infrastructure. More than a case study, he will share the details of how they prioritized their cloud computing infrastructure deployments ...
DevOps Summit at Cloud Expo 2014 Silicon Valley was a terrific event for us. The Qubell booth was crowded on all three days. We ran demos every 30 minutes with folks lining up to get a seat and usually standing around. It was great to meet and talk to over 500 people! My keynote was well received and so was Stan's joint presentation with RingCentral on Devops for BigData. I also participated in two Power Panels – ‘Women in Technology’ and ‘Why DevOps Is Even More Important than You Think,’ both ...