While patching our systems for the recent Heartbleed vulnerability, we found that some sites (including huge web properties), which had patched their servers were still vulnerable to a variant of the attack that we're calling "reverse heartbleed." This is a post about how the vulnerability works, what we found with our public tester tool, and how you might be affected.

What's Heartbleed and "reverse" Heartbleed?

The Heartbleed vulnerability in OpenSSL allows a malicious TLS implementation to extract random chunks of memory from an unpatched peer. If you're not up to speed on Heartbleed, check out the excellent documentation on that site and check your servers ASAP to see if you might be vulnerable.

Most of the attention around the Heartbleed attack has focused on the simplest and most obvious scenario: a malicious client attacking an HTTPS server to steal cookies, private keys, and other secrets. But this isn't the only attack possible: a malicious server can also send bad heartbeat packets to a client that uses OpenSSL and extract data from that client. The TLS heartbeats used in this attack are symmetric: they can be initiated by either the "client" or the "server" in a TLS connection, and both endpoints use the same vulnerable parsing code.

What kinds of clients are vulnerable?

Anything that speaks TLS using OpenSSL is potentially vulnerable, but there are two main classes of client apps that are worth mentioning:

1) Traditional clients are things like web browsers, apps that use HTTP APIs (everything from Dropbox to Microsoft Office), and of course many mobile apps on both iOS and Android. It might be easy to direct one of these clients to connect to a malicious server (as in the case of a web browser) or it might require a man-in-the-middle (MITM) attack to redirect a client to an evil endpoint.

2) Open agents are clients that can be driven by an attacker but don't reside on an attacker's machine. If you can direct some remote application to fetch a URL on your behalf, then you could theoretically attack that application. The web is full of applications that accept URLs and do something with them; any of these have the potential to be vulnerable:

* Social networks that do smart things with URLs; e.g. Facebook, which
  fetches any URL that you type in to a status update in order to
  generate a preview of that URL.
* File sharing apps like image thumbnailers, image hosters, Gravatar,
  and anything else that can "upload" an image or other user-supplied
  data via a URL.
* Web spiders like the Googlebot that can stumble on a URL and index it
  – they can be directed to a malicious server just by linking to it.
* API consumers that allow integrations across websites. For example,
  Redbooth integrates with Dropbox to allow users to upload files to
  projects. If I can convince the Redbooth servers via MITM to send
  their Dropbox requests to my server, I can potentially exploit them.
* Identity federation protocols, such as OpenID and WebFinger, allow
  low-trust users to direct high-trust servers to arbitrary URLs that
  the user can control. The StackOverflow login page prompts the user
  for a URL that can be used to log in with OpenID – therefore, the
  code that StackOverflow uses to fetch that URL must not be
  vulnerable.
* Webhooks, which allow a user to register interest in a certain event
  happening and get a callback. I can tell Github that I'd like to be
  notified at a URL I control whenever someone pushes to a repository,
  and Github's agent will connect to that URL over TLS if specified.

The surface of exposed clients is potentially very broad – any code in an application that makes outbound HTTP requests must be checked against reverse Heartbleed attacks. The traditional Heartbleed attack goes after servers that are at the "edges" of a company's security perimeter and therefore may be more hardened, but these open agents are usually deep within an organization's infrastructure.

We found exactly this problem in Meldium's architecture. We run hosts that can make HTTP requests to user-defined URLs. These hosts also have extremely sensitive data in their process memory. While we were not vulnerable to reverse heartbleed, this exercise has pointed out that there needs to be a wide "gap" between anything that touches the outside web and our secure hosts, regardless of whether the network connections are inbound or outbound.

What did we find?

After we developed this tool but before we made it public, we ran some ad-hoc tests against a number of major web properties. These ad-hoc tests found three sites that had patched against Heartbleed on their perimeter hosts, but had not patched their agents and thus could be exploited:

* An unnamed top 5 social network (we're waiting for confirmation of
  their fix) that fetched our URL to generate a preview. The memory we
  extracted from their agent included results from internal API calls
  and snippets of python source code.
* Reddit, which can use a URL to suggest a name for a new post, used a
  vulnerable agent that they've now patched. The memory we were able to
  extract from this agent was less sensitive, but we didn't get as many
  samples because they patched so quickly (nice work!).
* We registered a webhook to our malicious URL at rubygems.org to
  notify us whenever a gem was published. Within a few minutes, we
  captured chunks of S3 API calls that the Rubygems servers were
  making. After the disclosure, they quickly updated OpenSSL and are
  now protected (really nice work, especially from an all-volunteer
  staff!).

Is this harder to exploit than Heartbleed?

At a high-level, the exploit code is very similar to the client-initiated attack, but there are a couple of factors that make "reverse Heartbleed" a little harder to pull off:

* If a client is using certificate pinning, then the window for
  exploitation is much smaller, because as soon as the client realizes
  the server certificate doesn't match, they'll cut off the TLS
  connection.
* Some clients that support TLS heartbeats will not advertise that
  support when negotiating a handshake. In our exploit code, we removed
  the OpenSSL checks that determined whether or not heartbeat requests
  were allowed based on the negotiated and just sent them anyway.
* Server-initiated heartbeat requests can only be sent once the TLS
  connection has been established, so they need to be properly
  encrypted and MACed. We ended up building our exploit on top of a
  modified version of OpenSSL instead of writing one from scratch to
  keep the code simpler.
* Most clients are trying to complete an HTTP request and then close
  the channel, so our server uses a few tricks to keep the connection
  open as long as possible and maximize the number of heartbeat
  requests we can make. We slowly bleed out the HTTP response headers,
  we stream the HTTP response body, and we send malformed heartbeat
  requests interleved throughout. We've found that vulnerable clients
  can actually be made to send hundreds of 16kb chunks of memory back,
  making it much easier to explore the client's memory space.

What can I do?

The mitigation steps are the same as for the regular Heartbleed attack: don't use vulnerable versions of OpenSSL. The important takeaway is that it's not enough to patch your perimeter hosts - you need to purge bad OpenSSL versions from your entire infrastructure. And you should keep a healthy distance between agent code that fetches user-provided URLs and sensitive parts of your systems.

Try the reverse heartbleed tester now on your sites, and please practice responsible disclosure if you find vulnerabilities. We'll continue to work to improve the tester to cover more cases, and we hope to release the source code soon.