We now use priority queue per stream, which contains the stream which
has ready to send a frame, or one of its descendants have a frame to
send. We maintain invariant that if a stream is queued, then its
ancestors are also queued (except for root). When we re-schedule
stream after transmission, we re-schedule all ancestors, so that
streams on the other path can get a chance to send. This is basically
the same mechanism h2o project uses, but there are differences in the
details.
Previously, the number of stream in one dependency tree (not including
root) is limited to 120. This is due to the fact that we use
recursive calls to traverse trees. Now we replaced recursive calls
with loop, we can remove this limitation. Also now all streams are
descendant of root stream, rather than linked list of individual
subtree root.
After reviewing codebase, only queue for DATA frames requires
priorities. Other frames can be replaced multiple linear queues.
Replacing priority queue with linear queue allows us to simplify
codebase a bit; for example, now nghttp2_session.next_seq is gone.
Previously nghttp2_session_send() and nghttp2_session_mem_send() did
not send 24 bytes client magic byte string (MAGIC). We made
nghttp2_session_recv() and nghttp2_session_mem_recv() process MAGIC by
default, so it is natural to make library send MAGIC as well. This
commit makes nghttp2_session_send() and nghttp2_session_mem_send()
send MAGIC. This commit also replace "connection preface" with
"client magic", since we call MAGIC as "connection preface" but it is
just a part of connection preface. NGHTTP2_CLIENT_CONNECTION_PREFACE
macro was replaced with NGHTTP2_CLIENT_MAGIC. The already deprecated
NGHTTP2_CLIENT_CONNECTION_HEADER macro was removed permanently.
nghttp2_option_set_no_recv_client_preface() was renamed as
nghttp2_option_set_no_recv_client_magic(). NGHTTP2_ERR_BAD_PREFACE
was renamed as NGHTTP2_ERR_BAD_CLIENT_MAGIC.
Since HTTP/2 spec requires for client to send connection preface, it
is reasonable to make this option enabled by default. It is still a
use case to disable this, so replace this option with
nghttp2_option_set_no_recv_client_preface().
To avoid buffer copy in nghttp2_data_source_read_callback, this commit
introduces NGHTTP2_DATA_FLAG_NO_COPY and nghttp2_send_data_callback.
By using NGHTTP2_DATA_FLAG_NO_COPY in
nghttp2_data_source_read_callback, application can avoid to copy
application data to given buffer. Instead, application has to
implement nghttp2_send_data_callback to send complete DATA frame by
itself. We see noticeable performance increase in nghttpd and
tiny-nghttpd using this new feature. On the other hand, nghttpx does
not show such difference, probably because buffer copy is not
bottleneck. Using nghttp2_send_data_callback adds complexity, so it
is recommended to measure the performance to see whether this extra
complexity worth it.
After we sent SETTINGS including ENABLE_PUSH = 0, peer may already
issue PUSH_PROMISE before receiving our SETTINGS and react it to
SETTINGS ACK. Previously we accept this PUSH_PROMISE. In this
commit, we check the pending ENABLE_PUSH value and if it means
disabling push, we refuse PUSH_PROMISE with RST_STREAM of error
REFUSED_STREAM.
Previously we did not check HTTP semantics and it is left out for
application. Although checking is relatively easy, but they are
scattered and error prone. We have implemented these checks in our
applications and also feel they are tedious. To make application
development a bit easier, this commit adds basic HTTP semantics
validation to library code. We do following checks:
server:
* HEADERS is either request header or trailer header. Other type of
header is disallowed.
client:
* HEADERS is either zero or more non-final response header or final
response header or trailer header. Other type of header is
disallowed.
For both:
* Check mandatory pseudo header fields.
* Make sure that content-length matches the amount of DATA we
received.
If validation fails, RST_STREAM of type PROTOCOL_ERROR is issued.
Previously we treat stream in NGHTTP2_STREAM_RESERVED state specially,
that is we don't increment or decrement streams counts if stream is in
that state. Because of this, we don't change the stream state to
NGHTTP2_STREAM_CLOSING if stream is in NGHTTP2_STREAM_RESERVED. But
it turns out that it causes a problem. If client canceled pushed
stream before push response HEADERS, stream is still in
NGHTTP2_STREAM_RESERVED state. If push response HEADERS arrived in
this state, library happily accepts it and passed to application.
With this commit, this bug was corrected. We now change stream state
to NGHTTP2_STREAM_CLOSING even if it was in NGHTTP2_STREAM_RESERVED
state. We now use NGHTTP2_STREAM_FLAG_PUSH to determine whether we
have to increase/decrase stream count.
nghttp2_submit_shutdown_notice() is used to notify the client that
graceful shutdown is started. We expect that after this call, the
server application should send another GOAWAY using
nghttp2_submit_goaway() with appropriate last_stream_id. In this
commit, we also added nghttp2_session_get_last_proc_stream_id(), which
can be used as last_stream_id parameter.
This commit implements graceful shutdown in nghttpx. The integration
test for graceful shutdown is also added.
Previously we handle idle streams as closed streams. We only keeps
sum of closed streams and active streams under max concurrent streams
limit, idle streams gets deleted earlier than client expects.
In this change, idle streams are kept in separate list and not handled
as closed streams. To mitigate possible attack vector to make
unlimited idle streams, we cap the number of idle streams in a half of
max concurrent streams. This is arbitrary choice. It may be adjusted
in the future when we have interop experience.
nghttp2_mem structure is introduced to hold custom memory allocator
functions and user supplied pointer. nghttp2_mem object can be passed
to nghttp2_session_client_new3(), nghttp2_session_server_new3(),
nghttp2_hd_deflate_new2() and nghttp2_hd_inflate_new2() to replace
standard malloc(), free(), calloc() and realloc(). nghttp2_mem
structure has user supplied pointer mem_user_data which can be used as
per session/object memory pool.
This change will utilize last_stream_id in GOAWAY extensively. When
GOAWAY is received with a last_stream_id, library closes all outgoing
streams whose stream_id > received last_stream_id.
nghttp2_on_stream_callback is called for each stream to be closed.
When GOAWAY is sent with a last_stream_id, library closes all incoming
streams whose stream_id > sent last_stream_id.
nghttp2_on_stream_callback is called for each stream to be closed.
Allowing PRIORITY frame at anytime so that PRIORITY frame to idle
stream can create anchor node in dependency tree. In this change, we
open stream with new NGHTTP2_STREAM_IDLE state, which is linked in
session->closed_stream_head and is treated as if it is closed stream.
One difference is that if the stream is opened, we remove it from
linked list and change the state to the appropriate one. To O(1)
removal from linked list, we change session->closed_stream_head to
doubly linked list.
Previously when connection level remote flow control window gets 0, we
mark the stream having DATA frame with
NGHTTP2_STREAM_FLAG_DEFERRED_FLOW_CONTROL. When connection level
WINDOW_UPDATE is received, we checks all existing streams, including
closed ones, and call nghttp2_stream_resume_deferred_data(). The
profiler shows this is expensive.
Now we prepare dedicated priority queue for DATA frames. And we don't
mark stream with NGHTTP2_STREAM_FLAG_DEFERRED_FLOW_CONTROL when DATA
cannot be sent solely due to connection level flow control. Instead,
we just queue DATA item to queue. We won't pop DATA item from queue
when connection level remote window size is 0. This way, we avoid the
expensive operation for all streams when WINDOW_UPDATE is arrived.
By default, nghttp2 library only handles HTTP/2 frames and does not
recognize first 24 bytes of client connection preface. This design
choice is done due to the fact that server may want to detect the
application protocol based on first few bytes on clear text
communication. But for simple servers which only speak HTTP/2, it is
easier for developers if nghttp2 library takes care of client
connection preface.
If this option is used with nonzero val, nghttp2 library checks first
24 bytes client connection preface. If it is not a valid one,
nghttp2_session_recv() and nghttp2_session_mem_recv() will return
error NGHTTP2_ERR_BAD_PREFACE, which is fatal error.
h2-14 now allows extensions to define new error codes. To allow
application callback to access such error codes, we uses uint32_t as
error_code type for structs and function parameters. Previously we
treated unknown error code as INTERNAL_ERROR, but this change removes
this and unknown error code is passed to application callback as is.
To make it possible to add new callbacks without bumping so name, we
decided to hide details of nghttp2_session_callbacks. We provide
setter like functions to set individual callback function.
The application should be responsible for the size of incoming header
block size. Framing layer just passes everything (we have size limit
for one header/field though) to application.
Reworked no automatic WINDOW_UPDATE feature. We added new API
nghttp2_session_consume() which tells the library how many bytes are
consumed by the application. Instead of submitting WINDOW_UPDATE by
the application, the library is now responsible to submit
WINDOW_UPDATE based on consumed bytes. This is more reliable method,
since it enables us to properly send WINDOW_UPDATE for stream and
connection individually. The previous implementation of nghttpx had
broken connection window management.
Previously we just assumed that if same settings ID is found in
SETTINGS, it is enough to process last seen entry. But it turns out
it is not enough for SETTINGS_HEADER_TABLE_SIZE. If we have 0 and
4096 for SETTINGS_HEADER_TABLE_SIZE in one SETTINGS, we must first
shrink dynamic table to 0 and then enlarge it to 4096. This means
that we have to remember the minimum value and last value.
Since we do not call on_data_chunk_recv_callback for ignored DATA
chunk, if nghttp2_option_set_no_auto_connection_window_update is used,
application may not have a chance to send connection WINDOW_UPDATE.
To fix this, we accumulate those received bytes, and if it exceeds
certain number, we automatically send connection-level WINDOW_UPDATE.
Tatsuhiro Tsujikawa
Alexis La Goutte