Welcome!

Microservices Expo Authors: Elizabeth White, Pat Romanski, Liz McMillan, Anders Wallgren, AppDynamics Blog

Related Topics: Cloud Security, Java IoT, Microservices Expo, Linux Containers, Open Source Cloud, Containers Expo Blog

Cloud Security: Article

The Definitive Guide to Heartbleed and Enterprise Mobility

There has been a lot of media attention on Heartbleed and as always that means a lot of bizarre and often conflicting advice

By Peter Rogers, Principal Mobility Architect, Cognizant

There has been a lot of media attention on Heartbleed and as always that means a lot of bizarre and often conflicting advice.  I sat down (for a very long time) to find the truth of the matter and bring it together in one single article for those involved in enterprise mobility.

What is Heartbleed?
"The [Heartbleed] bug is in the OpenSSL's implementation of the TLS/DTLS (transport layer security protocols) heartbeat extension (RFC6520). When it is exploited it leads to the leak of memory contents from the server to the client, and from the client to the server. The bug was introduced to OpenSSL in December 2011 and has been out in the wild since OpenSSL release 1.0.1 on 14th of March 2012. OpenSSL 1.0.1g released on 7th of April 2014 fixes the bug."
[http://www.heartbeat.com/]

"An attacker can retrieve a block of memory of the server up to 64kb. There is no limit on the number of attacks that can be performed. The attacker has no control over the memory region where the block is read from. Sensitive information that can be obtained is: SSL private keys; Basic authorization strings (username / password combinations); and source code [most important also cookies]. This bug affects both sides of the connection. Not only will client certificates not save you from having to update your server certificate, they can be read from the client (along with your username, password etc.) by any server you connect to."
[http://blog.fox-it.com/2014/04/08/openssl-heartbleed-bug-live-blog/]

"The Heartbleed bug is an example of Buffer Over-Read and that means that the offending OpenSSL heartbeat code does not check that this length is the actual length sent in the heartbeat request, allowing the request to ask for more data than it should be able to retrieve. The code then copies the amount of data indicated by the length from the incoming message to the outgoing message. If the length is longer than the incoming message, the software just keeps copying data past the end of the message. Since the length variable is 16 bits, you can request up to 65,535 bytes from memory. The data that lives past the end of the incoming message is from a kind of no-man's land that the program should not be accessing and may contain data left behind from other parts of OpenSSL."
[http://blog.cloudflare.com/answering-the-critical-question-can-you-get-private-ssl-keys-using-heartbleed]

The non-technical explanation from XKCD is slightly easier to follow (http://xkcd.com/1354/).

Here is a very good FAQ and an actual screen shot of a Heartbleed attack from Fox-IT.

The important thing to note is that not only can a client potentially read data from a server but the converse is also true. This impacts mobile clients too and although Apple claim they are safe there were allegedly concerns around Android 4.1.1. If a client application is vulnerable on a mobile or desktop device then equally the server can act as a honey trap to steal information from the client. Web browser software and IM clients are the biggest concern, so please make sure you don't use obscure and vulnerable software.

Who is stealing what and why?

The important question is how we got into this mess. The first thing to point out is that this is not a floor in SSL/TSP, instead it is an implementation problem in the open source OpenSSL library.  A major contributing factor has been that TLS versions 1.2 came with OpenSSL 1.0.1 and after the BEAST attack everyone rushed to implement TLS 1.2. The current theory doing the rounds is that the NSA had already ‘found ways around' SSL some time ago and this could have been by taking advantage of the Heartbleed bug. It is however noted that Man in the Middle (MITM) techniques are far easier, especially as some large companies act as their own Certificate Authority (CA). Edward Snowden leaked the existence of BULLRUN (a highly classified decryption program run by the NSA) and subsequently Edgehill (a similar program run by GCHQ) which subsequently prompted Google to double the size of their encryptions keys. We know that a number of companies knew about the vulnerability before it went public because they stated they had applied the patches at least a week ago. If you think about it a minute then OpenSSL is open source code which meant anybody could have worked it out and kept it a secret.

They say the exploit leaves no trace and so nobody actually knows if they have been targeted. Actually that statement isn't strictly true. A single heartbeat can potentially grab 64K of memory however an attacker can either keep reconnecting or during an active TLS connection keep requesting arbitrary number of 64 kilobyte chunks of memory content. There are now companies who are configuring their network monitoring software to watch for abnormal behaviour such as Fox-IT, who have developed a set of Snort signatures to detect successful exploitation of the bug.

The most important thing is what happens next and most hackers will be after what Heartbleed.com classes as primary key material.

"These are the crown jewels, the encryption keys themselves. Leaked secret keys allows the attacker to decrypt any past and future traffic to the protected services and to impersonate the service at will. Any protection given by the encryption and the signatures in the X.509 certificates can be bypassed. Recovery from this leak requires patching the vulnerability, revocation of the compromised keys and reissuing and redistributing new keys. Even doing all this will still leave any traffic intercepted by the attacker in the past still vulnerable to decryption. All this has to be done by the owners of the services."
[http://www.heartbeat.com/]

Let's imagine that a service provider has been using a vulnerable version of Open SSL 1.0.1 and the types of hacker that may target them based on time ranges:

  1. Soon after the vulnerability was released (14th of March 2012)
  2. Two or three weeks before the announcement was made public but was still being discussed in hushed circles
  3. Ever since the announcement (April 7th)
  4. Ever since the Heartbleed proof of concepts went live (April 8th)

The first group are likely to be very serious hackers or even possibly Government sponsored activities which are nothing to worry about unless you are of significant danger to anyone. The second group are likely to be serious hackers or service providers testing out vulnerabilities in their systems (and potentially partner solutions) after they learn of the issues. The third group is the rest of the world and depends on the general availability of OpenSSL vulnerability scanning software at that date. The fourth group covers hobbyist hackers because the proof of concepts are open source and highly available.

Unfortunately as of April 8th then anyone in the world can now test for this vulnerability using this widely shared Python Script (https://gist.github.com/takeshixx/10107280) and that puts it in the domain of hobbyists. Anyone can now dump a bit of RAM from a vulnerable server. It would definitely take a professional who can write their own script to make repeated heartbeats attacks and then got lucky with the timing of a server restart on a vulnerable operating system to be able to pull out a private encryption key. However, even an amateur can use Session Hijacking which means you look for cookies containing session IDs. In a single grab of memory then you are equally as likely to obtain a session ID or a username and password combination, and this is slightly harder to track for abnormal behaviour.

An example of which is running the Python proof of concept against a vulnerable JIRA ticketing system and pulling out a JSESSIONID (which is JIRA's way of tracking your HTTP session). If the system requires authentication then you can just insert the stolen cookie into the browser and become that user on the JIRA installation. (https://www.mattslifebytes.com/?p=533)

Can you actually steal a private key?

The Heartbleed bug is an example of buffer over-read and here is the offending code for the programmers amongst us:

p = &s->s3->rrec.data[0]

[...]

hbtype = *p++;
n2s(p, payload);
pl = p;

[...]

buffer = OPENSSL_malloc(1 + 2 + payload + padding);
bp = buffer;

[...]

memcpy(bp, pl, payload);

In NGINX, the keys are loaded immediately when the process is started, which puts the keys very low in the memory space. This makes it unlikely that incoming requests will be allocated with a lower address space. If NGINX is reloaded, it starts a new process and loads the keys right away, putting them at a low address. On NGINX you can therefore retrieve user credentials, cookies and even public parts of the certificate but so far not the private part. It all depends therefore on the operating system and how often the servers are restarted. CloudFlare have even issued a challenge by leaving a server running nginx-1.5.13 linked against OpenSSL 1.0.1.f on Ubuntu 13.10 x86_6

That said there appears to be a proof of concept working on Apache for a first request which corresponds with the server being rebooted.

The Certificate Revocation Tsunami
There is a genuine importance to the CloudFlare challenge and that is that if private keys can be stolen then there will be a whole lot of companies requesting certificates all at once leading to a so called Certificate Revocation Tsunami. Certificate Authorities are supposed to revoke certificates within 24 hours if there is any evidence of a key compromise. However, even if all of the affected certificates were to be revoked, contemporary web browser software handles certificate revocation poorly. For example, some browsers only perform OCSP revocation checks for Extended Validation certificates, while others ignore certificate revocation lists completely. The other concern is the rush to create new certificates will lead to incorrect certificates being generated which in turn leads to Man In The Middle (MITM) exploits down the line.

"The certificate authority infrastructure was never built to do a mass revocation of this many certificates and because of the way the infrastructure is built, if you did do a mass revocation of millions of certificates it would significantly slow down the performance of the Internet itself, which is potentially very, very bad."
[http://www.washingtonpost.com/blogs/the-switch/wp/2014/04/11/no-joke-this-company-wants-to-be-hacked-with-heartbleed/]

"Such is the haste to fix the fallout of the Heartbleed bug, some certificate authorities and website administrators have been making careless mistakes. PayPal's Hosted Message Applications, such as the one at https://view.paypal-communication.com, are now using Extended Validation certificates issued by VeriSign on 10 April 2014. The CAB Forum requires certificate authorities to adhere to a stringent set of guidelines [pdf] when issuing EV certificates, and it is the CA's responsibility to verify the accuracy of the information in the certificate. In particular, they must verify that the legal name of the subject in an EV certificate matches the name which appears on official government records. However, this verification does not appear to have been performed correctly in the case of these certificates."
[http://news.netcraft.com/archives/2014/04/11/heartbleed-certificate-revocation-tsunami-yet-to-arrive.html]

How can I protect myself?

Here is a list of all the password changing advice from the various companies and the results of a vulnerable scan performed on April 8, 12:00 UTC.

"Only 30,000 of the 500,000+ SSL certificates affected by the Heartbleed bug have been reissued up until today, and even fewer certificates have been revoked. Some of the first sites to deploy newly issued certificates in response to the OpenSSL vulnerability included Yahoo, Adobe, CloudFlare, DuckDuckGo, GitHub, Reddit , Launchpad, PayPal, Netflix and Amazon's CloudFront content delivery network."
[http://news.netcraft.com/archives/2014/04/11/heartbleed-certificate-revocation-tsunami-yet-to-arrive.html]

Fox-IT have actually listed a set of vulnerability tests, along with advice of the next steps and have developed a set of Snort signatures to detect successful exploitation of the bug.

Vendors are recommended to perform the following operations:

  • Upgrade the OpenSSL version to 1.0.1g (you can also recompile your OpenSSL 1.0.1 with the compile time option -DOPENSSL_NO_HEARTBEATS)
  • Request revocation of the current SSL certificate
  • Regenerate your private key
  • Request and replace the SSL certificate
  • Restart your services

Upgrading to OpenSSL 1.0.1g  does not automatically restart your services and likewise certificate revocation and replacement is a whole different step from ‘patching your server'.

  • If a vendor does not restart their services then their sessions are still vulnerable
  • Even if a vendor does revoke their certificates then some web browsers ignore the revocation
  • If a vendor does not replace their certificates then if their private key was stolen then they are wide open

If we look at the situation with WordPress then it appears they applied the patches but may not have replaced their certificates yet. I can also see smaller companies forgetting to restart their services after apply the OpenSSL 1.0.1g upgrade.

Should I change my password?

If you look at this list of vendors and their statements then the following rules apply:

  • You are safe if a vendor never used OpenSSL (remember SSL isn't broken itself, just one implementation of it including heartbeats)
  • You are safe if a vendor used an earlier version of OpenSSL (0.9.8 / 1.0.0)
  • You are moderately safe if a vendor upgraded to OpenSSL 1.0.1g, restarted their services and issued new certificates, before the public announcement on April 7th (apart from in the case of serious hackers)
  • You are at risk if a vendor upgraded to OpenSSL 1.0.1g, restarted their services and issued new certificates, sometime after April 7th (the longer the gap the higher the risk) and should change your passwords
  • You are at risk if a vendor upgraded to OpenSSL 1.0.1g, restarted their services and has not issued new certificates. You have a personal decision to make regarding changing passwords.

o   There is a small risk a private key could have been stolen and so even if you change your password then the site could still be vulnerable and you just gave your new password away

o   That said if the key did not get stolen then changing passwords prevents a hacker using your stolen details in the future.

  • You are at risk if a vendor either did not upgrade to OpenSSL 1.0.1g across all their services or forgot to restart their server. It is probably pretty pointless doing anything yet if that is the case.
  • You are at risk if a vendor made a nebulous statement like "We patched our servers" but was not drawn about restarting their services or issuing new certificates and should look to Internet forums
  • You are risk if a vendor has revoked their certificate but not issues a new certificate yet. Your web browser may ignore the certificate revocation and be open to MITM attacks.
  • You are at risk if you used client side software using a vulnerable version of OpenSSL and could have leaked confidential information from your desktop or mobile device

o   Double check the application that you use on Internet forums especially web browsers and IM clients

  • Most vendors are going to tell you to change your passwords anyway but check the details as above
  • Making your password really long is good practise but is no deterrent whatsoever from Heartbleed and you run the obvious risk of forgetting it or revealing it to hackers waiting for all the password resets

Imagine a hacker did target Vendor X, based on watching the charts of companies requesting new certificates and targeting those who haven't yet. Let us presume they were running Operating System X (all we know is that NGINX probably isn't vulnerable and Apache potentially is - but it would require specific timing) and just as a server rebooted the private key was added to the top of the OpenSSL memory stack and at that precise moment a hacker somehow managed to make a first request with an OpenSSL heartbeat and steal the private key. If we imagine that Vendor X are rebooting their servers regularly and maybe there was an insider involved then it becomes all the more probable. Let us now imagine the servers have been upgraded to OpenSSL 1.0.1g and all the services restarted but that the old certificates are still up there. That means if you reset your passwords then it makes no difference whatsoever because the hacker can decrypt all the traffic using their stolen private key. Changing your password actually means that the hacker now has your new password and you think you are actually safe. The hacker may have already decrypted everything on the site using their private key and have all your confidential information but let's assume they haven't. Vendor X then issue new certificates and does not bother going back through the last two years of logs to look for Heartbleed attacks. You have absolutely no idea that the hacker stole your new password and you carry on using the service and update it with your new credit card details...you are completely surprised that your account has been hacked even though Vendor X patched their servers.

That said, I would actually be more worried by the news that the majority of the traffic performing heartbeat vulnerability scans is apparently coming from various Government (thankfully not ours) sponsored cybercrime activity around the world.

I would like to end by thanking all the Blogs that I have referenced in bringing together this hopefully definitive guide to Heartbleed. I am happy to hear your views on the matter so please feel free to email me.

Peter

************************************************************************

Kevin Benedict Senior Analyst, Digital Transformation, EBA, Center for the Future of Work Cognizant View my profile on LinkedIn Learn about mobile strategies at MobileEnterpriseStrategies.com Follow me on Twitter @krbenedict Browse the Mobile Solution Directory Join the Linkedin Group Strategic Enterprise Mobility Join the Google+ Community Mobile Enterprise Strategies Recommended Strategy Book Code Halos Recommended iPad App Code Halos for iPads

***Full Disclosure: These are my personal opinions. No company is silly enough to claim them. I am a mobility and digital transformation analyst, consultant and writer. I work with and have worked with many of the companies mentioned in my articles.

More Stories By Kevin Benedict

Kevin Benedict is the Senior Analyst for Digital Transformation at Cognizant, a writer, speaker and SAP Mentor Alumnus. Follow him on Twitter @krbenedict. He is a popular speaker around the world on the topic of digital transformation and enterprise mobility. He maintains a busy schedule researching, writing and speaking at events in North America, Asia and Europe. He has over 25 years of experience working in the enterprise IT solutions industry.

@MicroservicesExpo Stories
Let’s face it, embracing new storage technologies, capabilities and upgrading to new hardware often adds complexity and increases costs. In his session at 18th Cloud Expo, Seth Oxenhorn, Vice President of Business Development & Alliances at FalconStor, will discuss how a truly heterogeneous software-defined storage approach can add value to legacy platforms and heterogeneous environments. The result reduces complexity, significantly lowers cost, and provides IT organizations with improved effi...
Data-as-a-Service is the complete package for the transformation of raw data into meaningful data assets and the delivery of those data assets. In her session at 18th Cloud Expo, Lakshmi Randall, an industry expert, analyst and strategist, will address: What is DaaS (Data-as-a-Service)? Challenges addressed by DaaS Vendors that are enabling DaaS Architecture options for DaaS
One of the bewildering things about DevOps is integrating the massive toolchain including the dozens of new tools that seem to crop up every year. Part of DevOps is Continuous Delivery and having a complex toolchain can add additional integration and setup to your developer environment. In his session at @DevOpsSummit at 18th Cloud Expo, Miko Matsumura, Chief Marketing Officer of Gradle Inc., will discuss which tools to use in a developer stack, how to provision the toolchain to minimize onboa...
SYS-CON Events announced today that Catchpoint Systems, Inc., a provider of innovative web and infrastructure monitoring solutions, has been named “Silver Sponsor” of SYS-CON's DevOps Summit at 18th Cloud Expo New York, which will take place June 7-9, 2016, at the Javits Center in New York City, NY. Catchpoint is a leading Digital Performance Analytics company that provides unparalleled insight into customer-critical services to help consistently deliver an amazing customer experience. Designed...
With the proliferation of both SQL and NoSQL databases, organizations can now target specific fit-for-purpose database tools for their different application needs regarding scalability, ease of use, ACID support, etc. Platform as a Service offerings make this even easier now, enabling developers to roll out their own database infrastructure in minutes with minimal management overhead. However, this same amount of flexibility also comes with the challenges of picking the right tool, on the right ...
SYS-CON Events announced today that Interoute, owner-operator of one of Europe's largest networks and a global cloud services platform, has been named “Bronze Sponsor” of SYS-CON's 18th Cloud Expo, which will take place on June 7-9, 2015 at the Javits Center in New York, New York. Interoute is the owner-operator of one of Europe's largest networks and a global cloud services platform which encompasses 12 data centers, 14 virtual data centers and 31 colocation centers, with connections to 195 ad...
Microservices are all the rage right now — and the industry is still learning, experimenting, and developing patterns, for successfully designing, deploying and managing Microservices in the real world. Are you considering jumping on the Microservices-wagon? Do Microservices make sense for your particular use case? What are some of the “gotchas” you should be aware of? This morning on #c9d9 we had experts from popular chat app Kik, SMB SaaS platform Yodle and hosted CI solution Semaphore sha...
SYS-CON Events announced today that Commvault, a global leader in enterprise data protection and information management, has been named “Bronze Sponsor” of SYS-CON's 18th International Cloud Expo, which will take place on June 7–9, 2016, at the Javits Center in New York City, NY, and the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Commvault is a leading provider of data protection and information management...
Microservices are a type of software architecture where large applications are made up of small, self-contained units working together through APIs that are not dependent on a specific language. Each service has a limited scope, concentrates on a specific task and is highly independent. This setup allows IT managers and developers to build systems in a modular way. In his book, “Building Microservices,” Sam Newman said microservices are small, focused components built to do a single thing very w...
With an estimated 50 billion devices connected to the Internet by 2020, several industries will begin to expand their capabilities for retaining end point data at the edge to better utilize the range of data types and sheer volume of M2M data generated by the Internet of Things. In his session at @ThingsExpo, Don DeLoach, CEO and President of Infobright, will discuss the infrastructures businesses will need to implement to handle this explosion of data by providing specific use cases for filte...
SYS-CON Events announced today that Avere Systems, a leading provider of enterprise storage for the hybrid cloud, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. Avere delivers a more modern architectural approach to storage that doesn’t require the overprovisioning of storage capacity to achieve performance, overspending on expensive storage media for inactive data or the overbuilding of data centers ...
CIOs and those charged with running IT Operations are challenged to deliver secure, audited, and reliable compute environments for the applications and data for the business. Behind the scenes these tasks are often accomplished by following onerous time-consuming processes and often the management of these environments and processes will be outsourced to multiple IT service providers. In addition, the division of work is often siloed into traditional "towers" that are not well integrated for cro...
SYS-CON Events announced today that Alert Logic, Inc., the leading provider of Security-as-a-Service solutions for the cloud, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. Alert Logic, Inc., provides Security-as-a-Service for on-premises, cloud, and hybrid infrastructures, delivering deep security insight and continuous protection for customers at a lower cost than traditional security solutions. Ful...
In most cases, it is convenient to have some human interaction with a web (micro-)service, no matter how small it is. A traditional approach would be to create an HTTP interface, where user requests will be dispatched and HTML/CSS pages must be served. This approach is indeed very traditional for a web site, but not really convenient for a web service, which is not intended to be good looking, 24x7 up and running and UX-optimized. Instead, talking to a web service in a chat-bot mode would be muc...
More and more companies are looking to microservices as an architectural pattern for breaking apart applications into more manageable pieces so that agile teams can deliver new features quicker and more effectively. What this pattern has done more than anything to date is spark organizational transformations, setting the foundation for future application development. In practice, however, there are a number of considerations to make that go beyond simply “build, ship, and run,” which changes ho...
SYS-CON Events announced today that AppNeta, the leader in performance insight for business-critical web applications, will exhibit and present at SYS-CON's @DevOpsSummit at Cloud Expo New York, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. AppNeta is the only application performance monitoring (APM) company to provide solutions for all applications – applications you develop internally, business-critical SaaS applications you use and the networks that deli...
WebSocket is effectively a persistent and fat pipe that is compatible with a standard web infrastructure; a "TCP for the Web." If you think of WebSocket in this light, there are other more hugely interesting applications of WebSocket than just simply sending data to a browser. In his session at 18th Cloud Expo, Frank Greco, Director of Technology for Kaazing Corporation, will compare other modern web connectivity methods such as HTTP/2, HTTP Streaming, Server-Sent Events and new W3C event APIs ...
SYS-CON Events announced today that Men & Mice, the leading global provider of DNS, DHCP and IP address management overlay solutions, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. The Men & Mice Suite overlay solution is already known for its powerful application in heterogeneous operating environments, enabling enterprises to scale without fuss. Building on a solid range of diverse platform support,...
The (re?)emergence of Microservices was especially prominent in this week’s news. What are they good for? do they make sense for your application? should you take the plunge? and what do Microservices mean for your DevOps and Continuous Delivery efforts? Continue reading for more on Microservices, containers, DevOps culture, and more top news from the past week. As always, stay tuned to all the news coming from@ElectricCloud on DevOps and Continuous Delivery throughout the week and retweet/favo...
How is your DevOps transformation coming along? How do you measure Agility? Reliability? Efficiency? Quality? Success?! How do you optimize your processes? This morning on #c9d9 we talked about some of the metrics that matter for the different stakeholders throughout the software delivery pipeline. Our panelists shared their best practices.