Microservices Expo Authors: Liz McMillan, Kareen Kircher, Andreas Grabner, Roger Strukhoff, Carmen Gonzalez

Related Topics: Cloud Security, Java IoT, Microservices Expo, Linux Containers, Open Source Cloud, Containers Expo Blog

Cloud Security: Article

The Definitive Guide to Heartbleed and Enterprise Mobility

There has been a lot of media attention on Heartbleed and as always that means a lot of bizarre and often conflicting advice

By Peter Rogers, Principal Mobility Architect, Cognizant

There has been a lot of media attention on Heartbleed and as always that means a lot of bizarre and often conflicting advice.  I sat down (for a very long time) to find the truth of the matter and bring it together in one single article for those involved in enterprise mobility.

What is Heartbleed?
"The [Heartbleed] bug is in the OpenSSL's implementation of the TLS/DTLS (transport layer security protocols) heartbeat extension (RFC6520). When it is exploited it leads to the leak of memory contents from the server to the client, and from the client to the server. The bug was introduced to OpenSSL in December 2011 and has been out in the wild since OpenSSL release 1.0.1 on 14th of March 2012. OpenSSL 1.0.1g released on 7th of April 2014 fixes the bug."

"An attacker can retrieve a block of memory of the server up to 64kb. There is no limit on the number of attacks that can be performed. The attacker has no control over the memory region where the block is read from. Sensitive information that can be obtained is: SSL private keys; Basic authorization strings (username / password combinations); and source code [most important also cookies]. This bug affects both sides of the connection. Not only will client certificates not save you from having to update your server certificate, they can be read from the client (along with your username, password etc.) by any server you connect to."

"The Heartbleed bug is an example of Buffer Over-Read and that means that the offending OpenSSL heartbeat code does not check that this length is the actual length sent in the heartbeat request, allowing the request to ask for more data than it should be able to retrieve. The code then copies the amount of data indicated by the length from the incoming message to the outgoing message. If the length is longer than the incoming message, the software just keeps copying data past the end of the message. Since the length variable is 16 bits, you can request up to 65,535 bytes from memory. The data that lives past the end of the incoming message is from a kind of no-man's land that the program should not be accessing and may contain data left behind from other parts of OpenSSL."

The non-technical explanation from XKCD is slightly easier to follow (http://xkcd.com/1354/).

Here is a very good FAQ and an actual screen shot of a Heartbleed attack from Fox-IT.

The important thing to note is that not only can a client potentially read data from a server but the converse is also true. This impacts mobile clients too and although Apple claim they are safe there were allegedly concerns around Android 4.1.1. If a client application is vulnerable on a mobile or desktop device then equally the server can act as a honey trap to steal information from the client. Web browser software and IM clients are the biggest concern, so please make sure you don't use obscure and vulnerable software.

Who is stealing what and why?

The important question is how we got into this mess. The first thing to point out is that this is not a floor in SSL/TSP, instead it is an implementation problem in the open source OpenSSL library.  A major contributing factor has been that TLS versions 1.2 came with OpenSSL 1.0.1 and after the BEAST attack everyone rushed to implement TLS 1.2. The current theory doing the rounds is that the NSA had already ‘found ways around' SSL some time ago and this could have been by taking advantage of the Heartbleed bug. It is however noted that Man in the Middle (MITM) techniques are far easier, especially as some large companies act as their own Certificate Authority (CA). Edward Snowden leaked the existence of BULLRUN (a highly classified decryption program run by the NSA) and subsequently Edgehill (a similar program run by GCHQ) which subsequently prompted Google to double the size of their encryptions keys. We know that a number of companies knew about the vulnerability before it went public because they stated they had applied the patches at least a week ago. If you think about it a minute then OpenSSL is open source code which meant anybody could have worked it out and kept it a secret.

They say the exploit leaves no trace and so nobody actually knows if they have been targeted. Actually that statement isn't strictly true. A single heartbeat can potentially grab 64K of memory however an attacker can either keep reconnecting or during an active TLS connection keep requesting arbitrary number of 64 kilobyte chunks of memory content. There are now companies who are configuring their network monitoring software to watch for abnormal behaviour such as Fox-IT, who have developed a set of Snort signatures to detect successful exploitation of the bug.

The most important thing is what happens next and most hackers will be after what Heartbleed.com classes as primary key material.

"These are the crown jewels, the encryption keys themselves. Leaked secret keys allows the attacker to decrypt any past and future traffic to the protected services and to impersonate the service at will. Any protection given by the encryption and the signatures in the X.509 certificates can be bypassed. Recovery from this leak requires patching the vulnerability, revocation of the compromised keys and reissuing and redistributing new keys. Even doing all this will still leave any traffic intercepted by the attacker in the past still vulnerable to decryption. All this has to be done by the owners of the services."

Let's imagine that a service provider has been using a vulnerable version of Open SSL 1.0.1 and the types of hacker that may target them based on time ranges:

  1. Soon after the vulnerability was released (14th of March 2012)
  2. Two or three weeks before the announcement was made public but was still being discussed in hushed circles
  3. Ever since the announcement (April 7th)
  4. Ever since the Heartbleed proof of concepts went live (April 8th)

The first group are likely to be very serious hackers or even possibly Government sponsored activities which are nothing to worry about unless you are of significant danger to anyone. The second group are likely to be serious hackers or service providers testing out vulnerabilities in their systems (and potentially partner solutions) after they learn of the issues. The third group is the rest of the world and depends on the general availability of OpenSSL vulnerability scanning software at that date. The fourth group covers hobbyist hackers because the proof of concepts are open source and highly available.

Unfortunately as of April 8th then anyone in the world can now test for this vulnerability using this widely shared Python Script (https://gist.github.com/takeshixx/10107280) and that puts it in the domain of hobbyists. Anyone can now dump a bit of RAM from a vulnerable server. It would definitely take a professional who can write their own script to make repeated heartbeats attacks and then got lucky with the timing of a server restart on a vulnerable operating system to be able to pull out a private encryption key. However, even an amateur can use Session Hijacking which means you look for cookies containing session IDs. In a single grab of memory then you are equally as likely to obtain a session ID or a username and password combination, and this is slightly harder to track for abnormal behaviour.

An example of which is running the Python proof of concept against a vulnerable JIRA ticketing system and pulling out a JSESSIONID (which is JIRA's way of tracking your HTTP session). If the system requires authentication then you can just insert the stolen cookie into the browser and become that user on the JIRA installation. (https://www.mattslifebytes.com/?p=533)

Can you actually steal a private key?

The Heartbleed bug is an example of buffer over-read and here is the offending code for the programmers amongst us:

p = &s->s3->rrec.data[0]


hbtype = *p++;
n2s(p, payload);
pl = p;


buffer = OPENSSL_malloc(1 + 2 + payload + padding);
bp = buffer;


memcpy(bp, pl, payload);

In NGINX, the keys are loaded immediately when the process is started, which puts the keys very low in the memory space. This makes it unlikely that incoming requests will be allocated with a lower address space. If NGINX is reloaded, it starts a new process and loads the keys right away, putting them at a low address. On NGINX you can therefore retrieve user credentials, cookies and even public parts of the certificate but so far not the private part. It all depends therefore on the operating system and how often the servers are restarted. CloudFlare have even issued a challenge by leaving a server running nginx-1.5.13 linked against OpenSSL 1.0.1.f on Ubuntu 13.10 x86_6

That said there appears to be a proof of concept working on Apache for a first request which corresponds with the server being rebooted.

The Certificate Revocation Tsunami
There is a genuine importance to the CloudFlare challenge and that is that if private keys can be stolen then there will be a whole lot of companies requesting certificates all at once leading to a so called Certificate Revocation Tsunami. Certificate Authorities are supposed to revoke certificates within 24 hours if there is any evidence of a key compromise. However, even if all of the affected certificates were to be revoked, contemporary web browser software handles certificate revocation poorly. For example, some browsers only perform OCSP revocation checks for Extended Validation certificates, while others ignore certificate revocation lists completely. The other concern is the rush to create new certificates will lead to incorrect certificates being generated which in turn leads to Man In The Middle (MITM) exploits down the line.

"The certificate authority infrastructure was never built to do a mass revocation of this many certificates and because of the way the infrastructure is built, if you did do a mass revocation of millions of certificates it would significantly slow down the performance of the Internet itself, which is potentially very, very bad."

"Such is the haste to fix the fallout of the Heartbleed bug, some certificate authorities and website administrators have been making careless mistakes. PayPal's Hosted Message Applications, such as the one at https://view.paypal-communication.com, are now using Extended Validation certificates issued by VeriSign on 10 April 2014. The CAB Forum requires certificate authorities to adhere to a stringent set of guidelines [pdf] when issuing EV certificates, and it is the CA's responsibility to verify the accuracy of the information in the certificate. In particular, they must verify that the legal name of the subject in an EV certificate matches the name which appears on official government records. However, this verification does not appear to have been performed correctly in the case of these certificates."

How can I protect myself?

Here is a list of all the password changing advice from the various companies and the results of a vulnerable scan performed on April 8, 12:00 UTC.

"Only 30,000 of the 500,000+ SSL certificates affected by the Heartbleed bug have been reissued up until today, and even fewer certificates have been revoked. Some of the first sites to deploy newly issued certificates in response to the OpenSSL vulnerability included Yahoo, Adobe, CloudFlare, DuckDuckGo, GitHub, Reddit , Launchpad, PayPal, Netflix and Amazon's CloudFront content delivery network."

Fox-IT have actually listed a set of vulnerability tests, along with advice of the next steps and have developed a set of Snort signatures to detect successful exploitation of the bug.

Vendors are recommended to perform the following operations:

  • Upgrade the OpenSSL version to 1.0.1g (you can also recompile your OpenSSL 1.0.1 with the compile time option -DOPENSSL_NO_HEARTBEATS)
  • Request revocation of the current SSL certificate
  • Regenerate your private key
  • Request and replace the SSL certificate
  • Restart your services

Upgrading to OpenSSL 1.0.1g  does not automatically restart your services and likewise certificate revocation and replacement is a whole different step from ‘patching your server'.

  • If a vendor does not restart their services then their sessions are still vulnerable
  • Even if a vendor does revoke their certificates then some web browsers ignore the revocation
  • If a vendor does not replace their certificates then if their private key was stolen then they are wide open

If we look at the situation with WordPress then it appears they applied the patches but may not have replaced their certificates yet. I can also see smaller companies forgetting to restart their services after apply the OpenSSL 1.0.1g upgrade.

Should I change my password?

If you look at this list of vendors and their statements then the following rules apply:

  • You are safe if a vendor never used OpenSSL (remember SSL isn't broken itself, just one implementation of it including heartbeats)
  • You are safe if a vendor used an earlier version of OpenSSL (0.9.8 / 1.0.0)
  • You are moderately safe if a vendor upgraded to OpenSSL 1.0.1g, restarted their services and issued new certificates, before the public announcement on April 7th (apart from in the case of serious hackers)
  • You are at risk if a vendor upgraded to OpenSSL 1.0.1g, restarted their services and issued new certificates, sometime after April 7th (the longer the gap the higher the risk) and should change your passwords
  • You are at risk if a vendor upgraded to OpenSSL 1.0.1g, restarted their services and has not issued new certificates. You have a personal decision to make regarding changing passwords.

o   There is a small risk a private key could have been stolen and so even if you change your password then the site could still be vulnerable and you just gave your new password away

o   That said if the key did not get stolen then changing passwords prevents a hacker using your stolen details in the future.

  • You are at risk if a vendor either did not upgrade to OpenSSL 1.0.1g across all their services or forgot to restart their server. It is probably pretty pointless doing anything yet if that is the case.
  • You are at risk if a vendor made a nebulous statement like "We patched our servers" but was not drawn about restarting their services or issuing new certificates and should look to Internet forums
  • You are risk if a vendor has revoked their certificate but not issues a new certificate yet. Your web browser may ignore the certificate revocation and be open to MITM attacks.
  • You are at risk if you used client side software using a vulnerable version of OpenSSL and could have leaked confidential information from your desktop or mobile device

o   Double check the application that you use on Internet forums especially web browsers and IM clients

  • Most vendors are going to tell you to change your passwords anyway but check the details as above
  • Making your password really long is good practise but is no deterrent whatsoever from Heartbleed and you run the obvious risk of forgetting it or revealing it to hackers waiting for all the password resets

Imagine a hacker did target Vendor X, based on watching the charts of companies requesting new certificates and targeting those who haven't yet. Let us presume they were running Operating System X (all we know is that NGINX probably isn't vulnerable and Apache potentially is - but it would require specific timing) and just as a server rebooted the private key was added to the top of the OpenSSL memory stack and at that precise moment a hacker somehow managed to make a first request with an OpenSSL heartbeat and steal the private key. If we imagine that Vendor X are rebooting their servers regularly and maybe there was an insider involved then it becomes all the more probable. Let us now imagine the servers have been upgraded to OpenSSL 1.0.1g and all the services restarted but that the old certificates are still up there. That means if you reset your passwords then it makes no difference whatsoever because the hacker can decrypt all the traffic using their stolen private key. Changing your password actually means that the hacker now has your new password and you think you are actually safe. The hacker may have already decrypted everything on the site using their private key and have all your confidential information but let's assume they haven't. Vendor X then issue new certificates and does not bother going back through the last two years of logs to look for Heartbleed attacks. You have absolutely no idea that the hacker stole your new password and you carry on using the service and update it with your new credit card details...you are completely surprised that your account has been hacked even though Vendor X patched their servers.

That said, I would actually be more worried by the news that the majority of the traffic performing heartbeat vulnerability scans is apparently coming from various Government (thankfully not ours) sponsored cybercrime activity around the world.

I would like to end by thanking all the Blogs that I have referenced in bringing together this hopefully definitive guide to Heartbleed. I am happy to hear your views on the matter so please feel free to email me.



Kevin Benedict Senior Analyst, Digital Transformation, EBA, Center for the Future of Work Cognizant View my profile on LinkedIn Learn about mobile strategies at MobileEnterpriseStrategies.com Follow me on Twitter @krbenedict Browse the Mobile Solution Directory Join the Linkedin Group Strategic Enterprise Mobility Join the Google+ Community Mobile Enterprise Strategies Recommended Strategy Book Code Halos Recommended iPad App Code Halos for iPads

***Full Disclosure: These are my personal opinions. No company is silly enough to claim them. I am a mobility and digital transformation analyst, consultant and writer. I work with and have worked with many of the companies mentioned in my articles.

More Stories By Kevin Benedict

Kevin Benedict is an opinionated Senior Analyst at Cognizant's Center for the Future of Work, SAP Mentor Alumnus, speaker, writer, and mobile and digital strategies expert. He is a popular keynote speaker, and in the past three years he has shared his insights into mobile and digital strategies with companies in 17 different countries. He has over 30 years of experience working with enterprise applications, and he is a veteran mobile industry executive. He wrote the Forward to SAP Press' bestselling book on enterprise mobility titled Mobilizing Your Enterprise with SAP, and he has written over 3,000 articles.

@MicroservicesExpo Stories
DevOps is a term that comes full of controversy. A lot of people are on the bandwagon, while others are waiting for the term to jump the shark, and eventually go back to business as usual. Regardless of where you are along the specturm of loving or hating the term DevOps, one thing is certain. More and more people are using it to describe a system administrator who uses scripts, or tools like, Chef, Puppet or Ansible, in order to provision infrastructure. There is also usually an expectation of...
DevOps is speeding towards the IT world like a freight train and the hype around it is deafening. There is no reason to be afraid of this change as it is the natural reaction to the agile movement that revolutionized development just a few years ago. By definition, DevOps is the natural alignment of IT performance to business profitability. The relevance of this has yet to be quantified but it has been suggested that the route to the CEO’s chair will come from the IT leaders that successfully ma...
As we enter the final week before the 19th International Cloud Expo | @ThingsExpo in Santa Clara, CA, it's time for me to reflect on six big topics that will be important during the show. Hybrid Cloud This general-purpose term seems to provide a comfort zone for many enterprise IT managers. It sounds reassuring to be able to work with one of the major public-cloud providers like AWS or Microsoft Azure while still maintaining an on-site presence.
The best way to leverage your Cloud Expo presence as a sponsor and exhibitor is to plan your news announcements around our events. The press covering Cloud Expo and @ThingsExpo will have access to these releases and will amplify your news announcements. More than two dozen Cloud companies either set deals at our shows or have announced their mergers and acquisitions at Cloud Expo. Product announcements during our show provide your company with the most reach through our targeted audiences.
This is a no-hype, pragmatic post about why I think you should consider architecting your next project the way SOA and/or microservices suggest. No matter if it’s a greenfield approach or if you’re in dire need of refactoring. Please note: considering still keeps open the option of not taking that approach. After reading this, you will have a better idea about whether building multiple small components instead of a single, large component makes sense for your project. This post assumes that you...
As software becomes more and more complex, we, as software developers, have been splitting up our code into smaller and smaller components. This is also true for the environment in which we run our code: going from bare metal, to VMs to the modern-day Cloud Native world of containers, schedulers and microservices. While we have figured out how to run containerized applications in the cloud using schedulers, we've yet to come up with a good solution to bridge the gap between getting your conta...
DevOps is being widely accepted (if not fully adopted) as essential in enterprise IT. But as Enterprise DevOps gains maturity, expands scope, and increases velocity, the need for data-driven decisions across teams becomes more acute. DevOps teams in any modern business must wrangle the ‘digital exhaust’ from the delivery toolchain, "pervasive" and "cognitive" computing, APIs and services, mobile devices and applications, the Internet of Things, and now even blockchain. In this power panel at @...
DevOps theory promotes a culture of continuous improvement built on collaboration, empowerment, systems thinking, and feedback loops. But how do you collaborate effectively across the traditional silos? How can you make decisions without system-wide visibility? How can you see the whole system when it is spread across teams and locations? How do you close feedback loops across teams and activities delivering complex multi-tier, cloud, container, serverless, and/or API-based services?
SYS-CON Events announced today that SoftNet Solutions will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. SoftNet Solutions specializes in Enterprise Solutions for Hadoop and Big Data. It offers customers the most open, robust, and value-conscious portfolio of solutions, services, and tools for the shortest route to success with Big Data. The unique differentiator is the ability to architect and ...
Today every business relies on software to drive the innovation necessary for a competitive edge in the Application Economy. This is why collaboration between development and operations, or DevOps, has become IT’s number one priority. Whether you are in Dev or Ops, understanding how to implement a DevOps strategy can deliver faster development cycles, improved software quality, reduced deployment times and overall better experiences for your customers.
Without lifecycle traceability and visibility across the tool chain, stakeholders from Planning-to-Ops have limited insight and answers to who, what, when, why and how across the DevOps lifecycle. This impacts the ability to deliver high quality software at the needed velocity to drive positive business outcomes. In his session at @DevOpsSummit 19th Cloud Expo, Eric Robertson, General Manager at CollabNet, will show how customers are able to achieve a level of transparency that enables everyon...
At its core DevOps is all about collaboration. The lines of communication must be opened and it takes some effort to ensure that they stay that way. It’s easy to pay lip service to trends and talk about implementing new methodologies, but without action, real benefits cannot be realized. Success requires planning, advocates empowered to effect change, and, of course, the right tooling. To bring about a cultural shift it’s important to share challenges. In simple terms, ensuring that everyone k...
What do dependency resolution, situational awareness, and superheroes have in common? Meet Chris Corriere, a DevOps/Software Engineer at Autotrader, speaking on creative ways to maximize usage of all of the above. Mark Miller, Community Advocate and senior storyteller at Sonatype, caught up with Chris to learn more about what his team is up to.
@DevOpsSummit has been named the ‘Top DevOps Influencer' by iTrend. iTrend processes millions of conversations, tweets, interactions, news articles, press releases, blog posts - and extract meaning form them and analyzes mobile and desktop software platforms used to communicate, various metadata (such as geo location), and automation tools. In overall placement, @DevOpsSummit ranked as the number one ‘DevOps Influencer' followed by @CloudExpo at third, and @MicroservicesE at 24th.
SYS-CON Events announced today that Super Micro Computer, Inc., a global leader in Embedded and IoT solutions, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 7-9, 2017, at the Javits Center in New York City, NY. Supermicro (NASDAQ: SMCI), the leading innovator in high-performance, high-efficiency server technology, is a premier provider of advanced server Building Block Solutions® for Data Center, Cloud Computing, Enterprise IT, Hadoop/Big Data, HPC and ...
JetBlue Airways uses virtual environments to reduce software development costs, centralize performance testing, and create a climate for continuous integration and real-time monitoring of mobile applications. The next BriefingsDirect Voice of the Customer performance engineering case study discussion examines how JetBlue Airways in New York uses virtual environments to reduce software development costs, centralize performance testing, and create a climate for continuous integration and real-tim...
Without lifecycle traceability and visibility across the tool chain, stakeholders from Planning-to-Ops have limited insight and answers to who, what, when, why and how across the DevOps lifecycle. This impacts the ability to deliver high quality software at the needed velocity to drive positive business outcomes. In his general session at @DevOpsSummit at 19th Cloud Expo, Eric Robertson, General Manager at CollabNet, will discuss how customers are able to achieve a level of transparency that e...
So you think you are a DevOps warrior, huh? Put your money (not really, it’s free) where your metrics are and prove it by taking The Ultimate DevOps Geek Quiz Challenge, sponsored by DevOps Summit. Battle through the set of tough questions created by industry thought leaders to earn your bragging rights and win some cool prizes.
SYS-CON Events announced today that LeaseWeb USA, a cloud Infrastructure-as-a-Service (IaaS) provider, will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. LeaseWeb is one of the world's largest hosting brands. The company helps customers define, develop and deploy IT infrastructure tailored to their exact business needs, by combining various kinds cloud solutions.
“Being able to take needless work out of the system is more important than being able to put more work into the system.” This is one of my favorite quotes from Gene Kim’s book, The Phoenix Project, and it plays directly into why we're announcing the DevOps Express initiative today. Tracing the Steps. For years now, I have witnessed needless work being performed across the DevOps industry. No, not within our clients DevOps and continuous delivery practices. I have seen it in the buyer’s journe...