Infrastructure vulnerabilities present a unique challenge for a large enterprise. These vulnerabilities are often widespread and affect many systems that are key infrastructure components. Fixing these kinds of issues is not as simple as clicking “yes” on an auto-update prompt and rebooting. It requires planning, coordination and a well-tested patching mechanism that takes uptime and performance into account.

The Internet is beginning to adopt HTTP/2 on a large scale. It only takes a handful of high traffic sites and one or two browsers to add support to make a large change in the adoption rate. However HTTP/2 brings some challenges with it. If you’re not familiar with the HTTP/2 protocol it is best summarized by the official page linked above:

“The focus of the protocol is on performance; specifically, end-user perceived latency, network and server resource usage. One major goal is to allow the use of a single connection from browsers to a Web site.”

HTTP/2 is a binary protocol that improves the performance of web-based sessions by keeping connections open, allowing for multiple exchanges per connection, and offering compression (HPACK) of a rather bloated ASCII based protocol (HTTP/1.1). It’s a great step forward for the web in general. Following the standardization and initial implementations of HTTP/2, the Yahoo Pentest Team began bug hunting in hopes of finding security vulnerabilities before they were widely deployed. This resulted in the development of an internal HTTP/2 fuzzer. Stuart Larsen wrote the first one in Go over the course of a few days and it immediately resulted in some great bugs.

To understand the fuzzer we built, you have to know a little bit about the protocol. HTTP/2 is very similar to HTTP/1.1 at its core. It still uses verb methods (GET/PUT/POST etc), and has the same HTTP headers we are used to (Content-Type, Origin, Referer). One of the key differences between these two protocols is how the requests are sent. Multiple binary requests are made over a single TCP session and they are often multiplexed.

Within a single TCP connection, messages called “frames” are exchanged between the client and the server. Frames manage things like headers, requesting data, setting priorities, and terminating connections. In total there are 10 different frame types: DATA, HEADERS, PRIORITY, RST_STREAM, SETTINGS, PUSH_PROMISE, PING, GOAWAY, WINDOW_UPDATE, CONTINUATION

The initial fuzzer design was to split outputs into 12 different strategies, 10 of them are for each of the frames, one for raw frames, and one for completely random data. Each fuzzing strategy manages a single TCP connection, and fuzzes that particular frame-type. The fuzzer then monitors the connection and restarts it as soon as the connection is dropped. On the server side we attach a debugger to the process and monitor for unexpected behavior such as segmentation faults.

Our first target for this fuzzer was Apache Traffic Server, a reverse caching proxy that Yahoo created and uses extensively. The idea behind targeting ATS was to discover security vulnerabilities before HTTP/2 was widely deployed. Below is our analysis of two vulnerabilities that the fuzzer discovered. Both of these issues have been patched upstream and are credited to Stuart Larsen. They have been assigned CVE-2015-3249. If you can’t patch ATS right now you can disable HTTP2 support mitigate these issues.

#1 Out Of Bounds Function Pointer Array Access

This vulnerability may allow for arbitrary code execution, but is highly dependent on the process memory layout. The issue is on line 637 of Http2ConnectionState.cc

[637] case HTTP2_SESSION_EVENT_RECV: { [638] Http2Frame frame = (Http2Frame )edata; [639] Http2StreamId last_streamid = frame->header().streamid; [640] Http2ErrorCode error; [641] [642] // Implementations MUST ignore and discard any frame that has a type that is unknown. [643] ink_assert(frame->header().type < countof(frame_handlers)); [644] if (frame->header().type > countof(frame_handlers)) { [645] return 0; [646] } [647] [648] if (frame_handlers[frame->header().type]) { [649] error = frame_handlersframe->header().type;

On line 644 an if statement is used to validate the value of type which is provided by the untrusted HTTP/2 frame. The frame_handlers array holds 9 function pointers to various functions for handling HTTP/2 frames. However it is declared with a size of HTTP2_FRAME_TYPE_MAX(10). The enum for indexing this array is named Http2FrameType and the last member is HTTP2_FRAME_TYPE_MAX (10). Providing a type value of 10 will satisfy the check on line 644 because countof(frame_handlers) properly returns the value 10, despite the array only holding 9 valid function pointers.

On line 649 frame_handlers is indexed with type and the value at that index in frame_handlers is treated as a function pointer and called. This array is static and thus located in global memory in the process. If an attacker can control the bytes just beyond this array in memory then this vulnerability can be used for arbitrary code execution.

#2 Out Of Bounds Vector Access Leads To Type Confusion / Incorrect delete

This vulnerability may allow for arbitrary code execution via arbitrary read and write primitives. The set_dynamic_table_size function allows an HTTP/2 client to control the value of the _settings_dynamic_table_size member variable. There is no constraint on the minimum or maximum size of this value.

The default table size, _settings_dynamic_table_size, is 4096. If new_size (attacker controllable) is smaller than old_size then the while loop on line 217 is entered. This incorrectly checks _settings_dynamic_table_size against new_size, instead of the current_size member, which should be updated anytime the table size changes.

[214] Http2DynamicTable::set_dynamic_table_size(uint32_t new_size) [215] { [216] uint32_t old_size = _settings_dynamic_table_size; [217] while (old_size > new_size) { [218] int last_name_len, last_value_len; [219] MIMEField *last_field = _headers.last(); [220] [221] last_field->name_get(&last_name_len); [222] last_field->value_get(&last_value_len); [223] old_size -= ADDITIONAL_OCTETS + last_name_len + last_value_len; [224] [225] _headers.remove_index(_headers.length() - 1); [226] _mhdr->field_delete(last_field, false); [227] } [228] [229] _settings_dynamic_table_size = new_size; [230] }

On line 219 the last member function of the _headers vector is called.

C & last() const { return v[n - 1]; }

If the vector contains no elements (this is the default as the Vec constructor initializes its member variables to 0) then n is 0 and the call to last returns a reference to an object that is not a member of the vector. In this particular case it returns a pointer to memory that is interpreted as a MIMEField object. Several values are retrieved from this object with the non-virtual calls to name_get and value_get. On line 225 a call to remove_index is made which results in an out-of-bounds write via a call to memmove.

Vec<C, A, S>::remove_index(int index) { if (n > 1) memmove(&v[index], &v[index + 1], (n - 1 - index) * sizeof(v[0])); n--; if (n <= 0) v = e; }

After this function completes the field_delete function is called with a pointer to the fake object which results in a number of different exploitable write primitives.

The Yahoo Paranoids are working hard to increase the security of critical Internet infrastructure that we all rely on. For a minimal investment in time and effort we were able to discover multiple vulnerabilities. This sort of payoff is the ultimate goal of fuzzing - minimal effort for maximum gain. We will continue running and tweaking our fuzzer and hope to uncover more bugs in other implementations soon. We hope you enjoyed this post!