The nghttp2 library itself is still h2-14. To experiment with the
implementations to require h2-16 to test new features (e.g.,
prioritization), nghttp, nghttpx, nghttpd and h2load now support h2-16
as well as h2-14. Cleartext HTTP Upgrade is still limited to h2-14
however.
Previously when requests are issued to HTTP/2 downstream connection,
but it turns out that connection is down, handlers of those requests
are deleted. In some situations, we only know connection is down when
we write something to network, so we'd like to handle this kind of
situation in more robust manner. In this change, certain seconds
passed after last network activity, we first issue PING frame to
downstream connection before issuing new HTTP request. If writing
PING frame is failed, it means connection was lost. In this case,
instead of deleting handler, pending requests are migrated to new
HTTP2/ downstream connection, so that it can continue without
affecting upstream connection.
nghttp2_mem structure is introduced to hold custom memory allocator
functions and user supplied pointer. nghttp2_mem object can be passed
to nghttp2_session_client_new3(), nghttp2_session_server_new3(),
nghttp2_hd_deflate_new2() and nghttp2_hd_inflate_new2() to replace
standard malloc(), free(), calloc() and realloc(). nghttp2_mem
structure has user supplied pointer mem_user_data which can be used as
per session/object memory pool.
For HTTP/1 backend, -b option can be used several times to specify
multiple backend address. HTTP/2 backend does not support multiple
addresses and only uses first address even if multiple addresses are
specified.
This commit limits the number of concurrent HTTP/1 downstream
connections to same host. By defualt, it is limited to 8 connections.
--backend-connections-per-frontend option was replaced with
--backend-http1-connections-per-host, which changes the maximum number
of connections per host. This limitation only kicks in when h2 proxy
is used (-s option).
This is unfortunate but help2man behaves badly when there is indented
lines in help messages in commnad-line args. We removed indentations
to make help2man happy.
This commit adds functionality to customize access logging format in
nghttpx. The format variables are inspired by nginx. The default
format is combined format.
This is not obvious but it makes intermediaries flush and forward DATA
frame boundary without excessive buffering. Since we have different
TCP connections frontend and backend, this may not work. This is
still experimental.
Use the same behaviour the current Google server does: start with 1300
TLS record size and after transmitting 1MiB, change record size to
16384. After 1 second idle time, reset to 1300. Only applies to
HTTP/2 and SPDY upstream connections.
Now the number of -v option specifies verbosity level. Current all
verbose output are turned on for at lest one -v option, except for the
debug output for each data chunk which is only turned on more than one
-v options.
* Add NGHTTP2_HTTP_1_1_REQUIRED error code
* Allow transmission of WINDOW_UPDATE on reserved (remote)
* Allow reception of WINDOW_UPDATE on reserved (local)
* Treat frame larger than MAX_FRAME_SIZE as FRAME_SIZE_ERROR
ALPN identifier is still h2-14 to continue interop, since draft-14 and
-15 are binary compatible. The new error code was added in draft-15,
but HTTP/2 allows extensions can freely add new error code, so it is
not a problem.
This is simply programming error, but it is interesting that using
libstdc++ does not reveal this error. With clang++-libc++, we got
std::system_error: mutex lock faild: Invalid argument. This is
because we did not give a name to lock object, so it is immediately
destructed. I think this will fix the reported crash on Mac OSX.
We may run into race condition if execve is called at the same time
when fcntl is called. But we just does this for now to compile
nghttp2 applications under older kernel.
The libnghttp2_asio library is C++ library built on top of libnghttp2.
Currently, it has server API and easily create HTTP/2 server using
node.js like API calls. See the example server source code in
examples/asio-sv.cc. The library depends on Boost::ASIO library.
Previously read and write timeouts work independently. When we are
writing response to the client, read timeout still ticks (e.g., HTTP/2
or tunneled HTTPS connection). So read timeout may occur during long
download. This commit fixes this issue. This commit only fixes the
upstream part. We need similar fix for the downstream.
With the combination of HTTP/1 upstream and HTTP/2 downstream,
downstream tells SHRPX_NO_BUFFER while connecting to the backend
server. Previously, we did not call upstream resume_read and upload
was blocked. This commit now calls upstream resume_read to unblock.
This commit also remove pending output buffer size of Http2Session
when calculating downstream connection's buffer is full. This is
desirable since we only operate resume_read by stream basis.
Android does not have _Exit. We detect this and use _exit instead.
clang-3.4 has an issue around undefined reference to
__atomic_fetch_add_4, so we stick to gcc-4.8 for now.
By default, nghttp2 library only handles HTTP/2 frames and does not
recognize first 24 bytes of client connection preface. This design
choice is done due to the fact that server may want to detect the
application protocol based on first few bytes on clear text
communication. But for simple servers which only speak HTTP/2, it is
easier for developers if nghttp2 library takes care of client
connection preface.
If this option is used with nonzero val, nghttp2 library checks first
24 bytes client connection preface. If it is not a valid one,
nghttp2_session_recv() and nghttp2_session_mem_recv() will return
error NGHTTP2_ERR_BAD_PREFACE, which is fatal error.
Previously we empties request headers after they are sent to
downstream in order to free memory. But it turns out that we use
request headers when rewriting location header response field. Also
user reported that request headers are useful to add new features.
This commits defers the deletion of request headers to the point when
response headers are deleted (which is after response headers are sent
to upstream client).
Even after on_stream_close_callback, Http2DownstreamConnection is
still alive and upstream keeps sending response to the client. The
consumed bytes are processed normally (data_source_read_callback) and
also we have a code to consume all allocated bytes for
Http2DownstreamConnection object when it is deleted. This means that
we don't need to and should not consume response data in downstream
on_stream_close_callback. If we do, we may get assertion error in
Http2DownstreamConnection::resume_read().
Now it returns only stream's available remote window size, without
considering connection level window size. For connection-level window
size, nghttp2_session_get_remote_window_size() is added by this
commit. To get old behavior of
nghttp2_session_get_stream_remote_window_size() is use
min(nghttp2_session_get_stream_remote_window_size(),
nghttp2_session_get_remote_window_size()). The reason of this change
is that it is desirable to know just stream level window size without
taking into connection level window size. This is useful for
debugging purpose.
It is not used by library for a while. It could be used to pass
unsupported extension frames to application, but its interface
requires library to buffer entire frame, which we'd like to avoid.
For unsupported extension frames, we will add new callbacks which does
not require buffering if they are required.
h2-14 now allows extensions to define new error codes. To allow
application callback to access such error codes, we uses uint32_t as
error_code type for structs and function parameters. Previously we
treated unknown error code as INTERNAL_ERROR, but this change removes
this and unknown error code is passed to application callback as is.
To make it possible to add new callbacks without bumping so name, we
decided to hide details of nghttp2_session_callbacks. We provide
setter like functions to set individual callback function.
Previously we only update consumed flow control window when number of
bytes read in nghttp2 and spdylay callback is 0. Now we notify
nghttp2 library the consumed bytes even if number of bytes read > 0.
This change also uses newly added spdylay_session_consume() API, so we
require spdylay >= 1.3.0.
Android lacks /dev/stderr, so directly use /proc/self/fd/2 as default
errorlog-file. Android does not like O_APPEND for /proc/self/fd/1 and
/proc/self/fd/2, so omit the flag for these paths.
This option limits the number of backend connections per frontend.
This is meaningful for the combination of HTTP/2 and SPDY frontend and
HTTP/1 backend.
libnghttp2 will call on_stream_close callback when RST_STREAM is
received. So we can use on_stream_close callback to handle existing
stream, instead of on_frame_recv callback.
nghttpx supports hot deploy feature using signals. The host deploy in
nghttpx is multi step process. First send USR2 signal to nghttpx
process. It will do fork and execute new executable, using same
command-line arguments and environment variables. At this point, both
current and new processes can accept requests. To gracefully shutdown
current process, send QUIT signal to current nghttpx process. When
all existing frontend connections are done, the current process will
exit. At this point, only new nghttpx process exists and serves
incoming requests.
--no-location-rewrite option disallows location header rewrite on
--http2-bridge, --client and default mode. This option is useful when
connecting nghttpx proxy with --http2-bridge to backend nghttpx with
http2-proxy mode.
It might be useful to clean the unused stream out to make up the room
for new streams. On the other hand, proxy should maintain the
connection between upstream client and downstream server and they have
the timeout for their own. Proxy just reacts to their decision.
Reworked no automatic WINDOW_UPDATE feature. We added new API
nghttp2_session_consume() which tells the library how many bytes are
consumed by the application. Instead of submitting WINDOW_UPDATE by
the application, the library is now responsible to submit
WINDOW_UPDATE based on consumed bytes. This is more reliable method,
since it enables us to properly send WINDOW_UPDATE for stream and
connection individually. The previous implementation of nghttpx had
broken connection window management.
Now concatenating header values with 0x00 as delimiter is not
necessary because HPACK reference set is removed and the order of
header field fed into HPACK encoder is preserved when they are
decoded.
This change rewrites logging system of nghttpx. Previously access log
and error log are written to stderr or syslog and there was no option
to change stderr to something else. With this change, file path of
access log and error log can be configured separately and logging to
regular file is now added. To support rotating log, if SIGUSR1 signal
is received by nghttpx, it closes the current log files and reopen it
with the same name. The format of access log is changed and has same
look of apache's. But not all columns are not supported yet.
For now, if request has request body, we'll issue RST_STREAM to inform
the peer to stop sending body. RST_STREAM may be sent before error
page header or data, so peer may receive RST_STREAM only.
Previously we do not specify the number of requests each client has to
issue. The each client corresponds to 1 TCP connection. If
connection was not accepted by server or not TLS handshake is not
done, we effectively don't use that connection and the requests
supposed to be issued for those connections are done via other
established connections. If this occurs, servers which do not accept
all connections may gain good benchmark results since they don't have
to pay extra cost to handle all connections (e.g., SSL/TLS handshake).
This change explicitly set the number of requests each client has to
issue so that servers cannot *cheat*.
It looks like setting read-rate and read-burst to 0 makes busy loop.
It seems a bug. On the other hand, we most likely want per-thread
rate limit rather than per-connection. So we decided to drop them.
It seems that if readcb is not set before SSL/TLS handshake, the
incoming data already available when eventcb (BEV_EVENT_CONNECTED
event) is fired is not further notified after setting new readcb. We
knew this fact and call upstream->on_read() in eventcb, but it is
wrong for HTTP/2. We have to call upstream_http2_connhd_readcb to
check connection preface. Otherwise, we consume it by nghttp2 session
and it is treated as unknown frame and connection preface is not
detected properly.
Libevent Openssl filter is very inconvenient in various respect. The
most annoying thing is it somehow emits data when SSL_shutdown is
called. The reason we introduced this filter solution is drop
connection if TLS renegotiation is detected. This commit implements
renegotiation detection and drop connection without filtering.
Cipher suites are chosen by DHE and ECDHE ciphers + GCM (AEAD). Now
default cipher list is the one recommended by Mozilla web site. The
--honor-cipher-order option is removed and now it is always assumed.
ALTSVC and BLOCKED frames are now extension frames. To add new
extension frame without modifying nghttp2_frame union, which causes so
name bump, we separated extension frames from core frames.
nghttp2_frame includes generic nghttp2_extension. The payload member
of nghttp2_extension will point to the structure of extension frame
payload. The frame types of extension frames are defined in
nghttp2_ext_frame_type.
Previously, we use evbuffer_pullup(buf, -1) to linearize the memory
region and it may cause buffer copy. To avoid this, we use the return
value of evbuffer_get_contiguous_space() as 2nd parameter. According
to the libevent manual, by doing so evbuffer_pullup() will not copy or
modify any data in evbuffer.
It seems that specifyig '*' to node parameter in getaddrinfo() is
treated as specifying NULL, but it is not documented. So rather than
relying on this feature, we explicitly treat '*' as "wildcard" address
and specify NULL to node parameter in getaddrinfo().
Now '*,3000' is a default value of --frontend option. Specyfing '*'
binds all addresses including both IPv4 and IPv6.
It appears that some Android client does not like Connection: close in
200 CONNECT response. Browsers fine with this header field. It is
better to remove it. Squid does not emit it too.
Previously stream ID was assigned just before HEADERS or PUSH_PROMISE
was serialized and nghttp2_submit_{request, headers, push_promise} did
not return stream ID. The application has to check assigned stream ID
using before_frame_send_callback. Now it is apparent that priority is
meant to DATA transfer only. Also application can reorder the
requests if it wants. Therefore we can assign stream ID in
nghttp2_submit_* functions and return stream ID from them. With this
change, now application does not have to check stream ID using
before_frame_send_callback and its code will be simplified.
We inherited gzip compression API from spdylay codebase. In spdylay,
the cost of having such API is almost free because spdylay requires
zlib for header compression. nghttp2 no longer uses gzip to header
compression. zlib dependency exists just for gzip compression API,
which is not an essential. So we decided to move gzip code to under
src and remove zlib dependency from libnghttp2 itself. As nghttp2
package, we depend on zlib to compile tools under src.
Currently, nghttpd server only compresses files whose extensions are
one of .html, .js, .css and .txt. nghttp advertises its support of
per-frame compression in SETTINGS frame. To implement this feature,
we added 2 public API: nghttp2_session_get_remote_settings() and
nghttp2_gzip_inflate_finished().
Callback function invoked to adjust priority value for request
HEADERS.
Since the application doesn’t know stream ID when it submits
requests, it may not be able to add correct priority value to HEADERS
frame and forced to use follwing PRIORITY frame. The purpose of this
callback is give the chance to the application to adjust priority
value with the latest information it has just before transmission so
that correct priority is included in HEADERS frame and it doesn’t
have to send additional PRIORITY frame.
The library interface supports compressed DATA. The library does not
deflate nor inflate data payload. When sending data, an application
has to compress data and set NGHTTP2_DATA_FLAG_COMPRESSED to
data_flags parameter in nghttp2_data_source_read_callback. On
receiving, flags parameter in nghttp2_on_data_chunk_recv_callback
includes NGHTTP2_FLAG_COMPRESSED. An application should check the
flags and inflate data as necessary. Since compression context is per
frame, when DATA is seen in nghttp2_on_frame_recv_callback, an
application should reset compression context.
Previously h2load supports SPDY only for https URI. This is because
SPDY has no mechanism to negotiate its protocol version without NPN.
With this change, user can specify the exact protocol version to use
when http URI (without SSL/TLS) is used.
To make adding new option easier, we decided to make the details of
option struct private and hide it from public API. We provide
functions to set individual option value.
If SPDY or HTTP/2 ustream is used and HTTP/2 downstream is used, only
call {spdylay,nghttp2}_resume_data when complete DATA frame was read
in backend to avoid to transmit too small DATA frame to the upstream.
NGHTTP2_CLIENT_CONNECTION_PREFACE has the same content with
NGHTTP2_CLIENT_CONNECTION_HEADER, which is now obsoleted by
NGHTTP2_CLIENT_CONNECTION_PREFACE.
Supplying multiple URIs can simulate more real life situation on
server side. For example, we can supply URIs of html, css and js and
benchmark the server. The -m option is updated so that it defaults to
the number of supplied URIs.
The existing options --{read,write}-{rate,burst} are per connection.
The new options --worker-{read,write}-{rate,burst} are per worker
thread, which is overall rate limit of all connections worker handles.
The profiler and benchmarking showed that calling evbuffer_add()
repeatedly is very costly. To avoid this, we buffer up small writes
into one large chunk and call evbuffer_add() less times.
This function behaves like nghttp2_session_send(), but it does not
use nghttp2_send_callback to send data. Instead, it returns the
serialized data to trasmit and its length to the caller.
* Use 1 Huffman code table for both request and response
* Remove complicated deflater side table size management
* Add encoding context update
* Fix memory leak in inflater
Now previous padding options are removed and instead we added
select_padding_callback to select padding length for each frame
by application. If this callback is not implemented by application,
no padding is added.
This change also fixes the broken session_detect_idle_stream()
if stream_id is our side.
Previously, there is inconsistency when on_frame_recv_callback
is called between HEADERS/PUSH_PROMISE and the other frames.
For former case, it is called before header block, in latter
case, it is called after whole frame is received. To make it
consistent, we call on_frame_recv_callback for HEADERS/PUSH_PROMISE
after its frame is fully received. Since on_frame_recv_callback
can signal the end of header block, we replaced on_end_headers_callback
with on_begin_headers_callback, which is called when the reception
of the header block is started.
Since all headers are not always longer available on one
nghttp2_session_mem_recv call, received headers may be interleaved
with transmission log of the other frames. To make it clear that
each header belongs to which stream, each header is printed with
stream_id.
nghttp2_data is added to nghttp2_frame union. When DATA is
received, nghttp2_on_frame_recv_callback is called. When DATA is
sent, nghttp2_on_frame_send_callback is called.
This stream inflater can inflate incoming header block in streaming
fashion. Currently, we buffer up single name/value pair, but we chose
far more smaller buffer size than HTTP/2 frame size.
evdns_base uses /etc/resolve.conf for *nix like systems,
but all platforms don't have the file (e.g., android device).
For such platforms, address resolution fails.
To fix this problem we use getaddrinfo() directly.
4ed4efc does not disable TLS renegotiation at all, if client keeps
rengotiations without sending application data. In this change,
we intercept the raw incoming data from the client and if it is a
renegotiation, drop the connection immediately.
nghttp2 library itself now accept octet header/value pairs,
completely not restricted by HTTP/1 header name/value rule.
The applications may impose restriction about them using
validators.
Now, in nghttp2_on_frame_recv_callback, nva and nvlen in
HEADERS and PUSH_PROMISE frames are always NULL and 0 respectively.
The header name/value pairs are emitted successive
nghttp2_on_header_callback functions. The end of header fields are
signaled with nghttp2_on_end_headers_callback function.
Since NGHTTP2_ERR_PAUSE for nghttp2_on_frame_recv_callback is
introduced to handle header block, it is now deprecated.
Instead, nghttp2_on_header_callback can be paused using
NGHTTP2_ERR_PAUSE.
We thought that this kind of rewrite can be achieved by the configuration
of the backend severs, but in some configuration, however, it may get
complicated. So we decided to implement at least location rewrite in
nghttpx.
This commit also contains a fix to the bug which prevents the http2
backend request from concatenating header fields with the same value.
This is pathological case, but we may get RST_STREAM to the promised
stream while we are sending PUSH_PROMISE. To handle this case,
instantiate Request object before transmission.
The option syntax is <PATH>=<PUSH_PATH,...>.
Push resources PUSH_PATHs when PATH is requested. This option can be used
repeatedly to specify multiple push configurations. For example,
-p/=/foo.png -p/doc=/bar.css
PATH and PUSH_PATHs are relative to document root.
Tatsuhiro Tsujikawa
Kazuho Oku
Lucas Pardue
Kenny (kang-yen) Peng
Kenny Peng
Svante Signell
Alexis La Goutte
Nicholas Hurley
David Weekly