diff --git a/doc/Makefile.am b/doc/Makefile.am index 46590104..7da84f7d 100644 --- a/doc/Makefile.am +++ b/doc/Makefile.am @@ -22,7 +22,7 @@ # WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. EXTRA_DIST = README.rst apiref-header.rst index.rst mkapiref.py \ - package_README.rst + package_README.rst tutorial-client.py tutorial-server.py # Makefile for Sphinx documentation # diff --git a/doc/index.rst b/doc/index.rst index ddb04d78..24da1503 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -18,6 +18,7 @@ Contents: package_README tutorial-client + tutorial-server apiref nghttp2.h nghttp2ver.h diff --git a/doc/tutorial-client.rst b/doc/tutorial-client.rst index 74769189..1a2b63b4 100644 --- a/doc/tutorial-client.rst +++ b/doc/tutorial-client.rst @@ -197,10 +197,10 @@ frame. The transmission of client connection header is done in } } -Here we specify max MAX_CONCURRENT_STREAMS to 100, which is really not -needed for this tiny example progoram, but we are demonstrating the -use of SETTINGS frame. To queue the SETTINGS frame for the -transmission, we use `nghttp2_submit_settings()`. Note that +Here we specify SETTINGS_MAX_CONCURRENT_STREAMS to 100, which is +really not needed for this tiny example progoram, but we are +demonstrating the use of SETTINGS frame. To queue the SETTINGS frame +for the transmission, we use `nghttp2_submit_settings()`. Note that `nghttp2_submit_settings()` function only queues the frame and not actually send it. All ``nghttp2_submit_*()`` family functions have this property. To actually send the frame, `nghttp2_session_send()` is diff --git a/doc/tutorial-server.rst b/doc/tutorial-server.rst new file mode 100644 index 00000000..5ede14a3 --- /dev/null +++ b/doc/tutorial-server.rst @@ -0,0 +1,597 @@ +Tutorial: HTTP/2.0 server +========================= + +In this tutorial, we are going to write single-threaded, event-based +HTTP/2.0 web server, which supports HTTPS. It can handle concurrent +multiple requests, but only GET method is supported. The complete +source code, `libevent-server.c`_, is attached at the end of this +page. It also resides in examples directory in the archive or +repository. + +This simple server takes 3 arguments, a port number to listen to, a +path to SSL/TLS private key file and certificate file. Its synopsis +is like this:: + + $ libevent-server PORT /path/to/server.key /path/to/server.crt + +We use libevent in this tutorial to handle networking I/O. Please +note that nghttp2 iteself does not depends on libevent. + +First we do some setup routine for libevent and OpenSSL library in +function ``main()`` and ``run()``, which is not so relevant to nghttp2 +library use. The one thing you should look at is setup NPN callback. +The NPN callback is used for the server to advertise the application +protocols the server supports to a client. In this example program, +when creating SSL_CTX object, we stores the application protocol name +in the wire format of NPN in statically allocated buffer. This is safe +because we only create 1 SSL_CTX object in the entire program life +time:: + + static unsigned char next_proto_list[256]; + static size_t next_proto_list_len; + + static int next_proto_cb(SSL *s, const unsigned char **data, unsigned int *len, + void *arg) + { + *data = next_proto_list; + *len = next_proto_list_len; + return SSL_TLSEXT_ERR_OK; + } + + + static SSL_CTX* create_ssl_ctx(const char *key_file, const char *cert_file) + { + SSL_CTX *ssl_ctx; + ssl_ctx = SSL_CTX_new(SSLv23_server_method()); + + ... + + next_proto_list[0] = NGHTTP2_PROTO_VERSION_ID_LEN; + memcpy(&next_proto_list[1], NGHTTP2_PROTO_VERSION_ID, + NGHTTP2_PROTO_VERSION_ID_LEN); + next_proto_list_len = 1 + NGHTTP2_PROTO_VERSION_ID_LEN; + + SSL_CTX_set_next_protos_advertised_cb(ssl_ctx, next_proto_cb, NULL); + return ssl_ctx; + } + +The wire format of NPN is array of length prefixed string. The exactly +one byte is used to specify the length of the protocol identifier. In +this tutorial, we advertise the HTTP/2.0 protocol the nghttp2 library +supports. We export its identifier in +:macro:`NGHTTP2_PROTO_VERSION_ID`. The ``next_proto_cb()`` function is +the server-side NPN callback. In OpenSSL implementation, We just +assign the pointer to the NPN buffers we filled earlier. The NPN +callback function is set to SSL_CTX object using +``SSL_CTX_set_next_protos_advertised_cb()``. + +We use ``app_content`` structure to store the application-wide data:: + + struct app_context { + SSL_CTX *ssl_ctx; + struct event_base *evbase; + }; + +We use ``http2_session_data`` structure to store the session-level +(which corresponds to 1 HTTP/2.0 connection) data:: + + typedef struct http2_session_data { + struct http2_stream_data root; + struct bufferevent *bev; + app_context *app_ctx; + nghttp2_session *session; + char *client_addr; + size_t handshake_leftlen; + } http2_session_data; + +We use ``http2_stream_data`` structure to store the stream-level +data:: + + typedef struct http2_stream_data { + struct http2_stream_data *prev, *next; + char *request_path; + int32_t stream_id; + int fd; + } http2_stream_data; + +1 HTTP/2.0 session can have multiple streams. We manage these +multiple streams by intrusive doubly linked list to add and remove the +object in O(1). The first element of this list is pointed by the +``root->next`` in ``http2_session_data``. Initially, ``root->next`` +is ``NULL``. The ``handshake_leftlen`` member of +``http2_session_data`` is used to track the number of bytes remaining +when receiving first 24 bytes magic values from the client. We use +libevent's bufferevent structure to perform network I/O. Notice that +bufferevent object is in ``http2_session_data`` and not in +``http2_stream_data``. This is because ``http2_stream_data`` is just a +logical stream multiplexed over the single connection managed by +bufferevent in ``http2_session_data``. + +We first create listener object to accept incoming connections. +We use libevent's ``struct evconnlistener`` for this purpose:: + + static void start_listen(struct event_base *evbase, const char *service, + app_context *app_ctx) + { + int rv; + struct addrinfo hints; + struct addrinfo *res, *rp; + + memset(&hints, 0, sizeof(hints)); + hints.ai_family = AF_UNSPEC; + hints.ai_socktype = SOCK_STREAM; + hints.ai_flags = AI_PASSIVE; + #ifdef AI_ADDRCONFIG + hints.ai_flags |= AI_ADDRCONFIG; + #endif // AI_ADDRCONFIG + + rv = getaddrinfo(NULL, service, &hints, &res); + if(rv != 0) { + errx(1, NULL); + } + for(rp = res; rp; rp = rp->ai_next) { + struct evconnlistener *listener; + listener = evconnlistener_new_bind(evbase, acceptcb, app_ctx, + LEV_OPT_CLOSE_ON_FREE | + LEV_OPT_REUSEABLE, -1, + rp->ai_addr, rp->ai_addrlen); + if(listener) { + return; + } + } + errx(1, "Could not start listener"); + } + +We specify ``acceptcb`` callback which is called when a new connection +is accepted:: + + static void acceptcb(struct evconnlistener *listener, int fd, + struct sockaddr *addr, int addrlen, void *arg) + { + app_context *app_ctx = (app_context*)arg; + http2_session_data *session_data; + + session_data = create_http2_session_data(app_ctx, fd, addr, addrlen); + bufferevent_setcb(session_data->bev, handshake_readcb, NULL, eventcb, + session_data); + } + +Here we create ``http2_session_data`` object. The bufferevent for this +connection is also initialized at this time. We specify 2 callbacks +for the bufferevent: ``handshake_readcb`` and ``eventcb``. + +The ``eventcb()`` is invoked by libevent event loop when an event +(e.g., connection established, timeout, etc) happens on the underlying +network socket:: + + static void eventcb(struct bufferevent *bev, short events, void *ptr) + { + http2_session_data *session_data = (http2_session_data*)ptr; + if(events & BEV_EVENT_CONNECTED) { + fprintf(stderr, "%s connected\n", session_data->client_addr); + return; + } + if(events & BEV_EVENT_EOF) { + fprintf(stderr, "%s disconnected\n", session_data->client_addr); + } else if(events & BEV_EVENT_ERROR) { + fprintf(stderr, "%s network error\n", session_data->client_addr); + } else if(events & BEV_EVENT_TIMEOUT) { + fprintf(stderr, "%s timeout\n", session_data->client_addr); + } + delete_http2_session_data(session_data); + } + +For ``BEV_EVENT_EOF``, ``BEV_EVENT_ERROR`` and ``BEV_EVENT_TIMEOUT`` +event, we just simply tear down the connection. The +``delete_http2_session_data()`` function destroys +``http2_session_data`` object and thus its bufferevent member. As a +result, the underlying connection is closed. The +``BEV_EVENT_CONNECTED`` event is invoked when SSL/TLS handshake is +finished successfully. + +The ``handshake_readcb()`` is a callback function to handle 24 bytes +magic byte string from a client, since nghttp2 library does not handle +it:: + + static void handshake_readcb(struct bufferevent *bev, void *ptr) + { + http2_session_data *session_data = (http2_session_data*)ptr; + uint8_t data[24]; + struct evbuffer *input = bufferevent_get_input(session_data->bev); + int readlen = evbuffer_remove(input, data, session_data->handshake_leftlen); + const char *conhead = NGHTTP2_CLIENT_CONNECTION_HEADER; + + if(memcmp(conhead + NGHTTP2_CLIENT_CONNECTION_HEADER_LEN + - session_data->handshake_leftlen, data, readlen) != 0) { + delete_http2_session_data(session_data); + return; + } + session_data->handshake_leftlen -= readlen; + if(session_data->handshake_leftlen == 0) { + bufferevent_setcb(session_data->bev, readcb, writecb, eventcb, ptr); + /* Process pending data in buffer since they are not notified + further */ + initialize_nghttp2_session(session_data); + if(send_server_connection_header(session_data) != 0) { + delete_http2_session_data(session_data); + return; + } + if(session_recv(session_data) != 0) { + delete_http2_session_data(session_data); + return; + } + } + } + +Nothing special here, we just compare the magic byte string received +and expected one :macro:`NGHTTP2_CLIENT_CONNECTION_HEADER`. When +whole magic byte string is received, the connection state is ready for +starting HTTP/2.0 communication. First we change the callback +functions for the bufferevent object. We use same ``eventcb`` as +before. But we specify new ``readcb`` and ``writecb`` function to +handle HTTP/2.0 communication. We describe these 2 functions later. + +We initialize nghttp2 session object which is done in +``initialize_nghttp2_session()``:: + + static void initialize_nghttp2_session(http2_session_data *session_data) + { + nghttp2_session_callbacks callbacks = {0}; + + callbacks.send_callback = send_callback; + callbacks.on_frame_recv_callback = on_frame_recv_callback; + callbacks.on_request_recv_callback = on_request_recv_callback; + callbacks.on_stream_close_callback = on_stream_close_callback; + nghttp2_session_server_new(&session_data->session, &callbacks, session_data); + } + +Since we are creating server, nghttp2 session object is created using +`nghttp2_session_server_new()` function. We registers 4 callbacks to +nghttp2 session object. We'll talk about these callbacks later. + +After initialization of nghttp2 session object, we are going to send +server connection header in ``send_server_connection_header()``:: + + static int send_server_connection_header(http2_session_data *session_data) + { + nghttp2_settings_entry iv[1] = { + { NGHTTP2_SETTINGS_MAX_CONCURRENT_STREAMS, 100 } + }; + int rv; + + rv = nghttp2_submit_settings(session_data->session, NGHTTP2_FLAG_NONE, + iv, ARRLEN(iv)); + if(rv != 0) { + warnx("Fatal error: %s", nghttp2_strerror(rv)); + return -1; + } + return 0; + } + +The server connection header is SETTINGS frame. We specify +SETTINGS_MAX_CONCURRENT_STREAMS to 100 in SETTINGS frame. To queue +the SETTINGS frame for the transmission, we use +`nghttp2_submit_settings()`. Note that `nghttp2_submit_settings()` +function only queues the frame and not actually send it. All +``nghttp2_submit_*()`` family functions have this property. To +actually send the frame, `nghttp2_session_send()` is used, which is +described about later. + +Since bufferevent may buffer more than first 24 bytes from the client, +we have to process them here since libevent won't invoke callback +functions for these pending data. To process received data, we call +``session_recv()`` function:: + + static int session_recv(http2_session_data *session_data) + { + int rv; + struct evbuffer *input = bufferevent_get_input(session_data->bev); + size_t datalen = evbuffer_get_length(input); + unsigned char *data = evbuffer_pullup(input, -1); + rv = nghttp2_session_mem_recv(session_data->session, data, datalen); + if(rv < 0) { + warnx("Fatal error: %s", nghttp2_strerror(rv)); + return -1; + } + evbuffer_drain(input, rv); + if(session_send(session_data) != 0) { + return -1; + } + return 0; + } + +In this function, we feed all unprocessed, received data to nghttp2 +session object using `nghttp2_session_mem_recv()` function. The +`nghttp2_session_mem_recv()` processes the received data and may +invoke nghttp2 callbacks and also queue frames. Since there may be +pending frames, we call ``session_send()`` function to send those +frames. The ``session_send()`` function is defined as follows:: + + static int session_send(http2_session_data *session_data) + { + int rv; + rv = nghttp2_session_send(session_data->session); + if(rv != 0) { + warnx("Fatal error: %s", nghttp2_strerror(rv)); + return -1; + } + return 0; + } + +The `nghttp2_session_send()` function serializes the frame into wire +format and call :member:`nghttp2_callbacks.nghttp2_send_callback` with +it. We set ``send_callback()`` function as +:member:`nghttp2_session_callbacks.send_callback` in +``initialize_nghttp2_session()`` function described earlier. It is +defined as follows:: + + static ssize_t send_callback(nghttp2_session *session, + const uint8_t *data, size_t length, + int flags, void *user_data) + { + http2_session_data *session_data = (http2_session_data*)user_data; + struct bufferevent *bev = session_data->bev; + /* Avoid excessive buffering in server side. */ + if(evbuffer_get_length(bufferevent_get_output(session_data->bev)) >= + OUTPUT_WOULDBLOCK_THRESHOLD) { + return NGHTTP2_ERR_WOULDBLOCK; + } + bufferevent_write(bev, data, length); + return length; + } + +Since we use bufferevent to abstract network I/O, we just write the +data to the bufferevent object. Note that `nghttp2_session_send()` +continues to write all frames queued so far. If we were writing the +data to the non-blocking socket directly using ``write()`` system call +in the :member:`nghttp2_session_callbacks.send_callback`, we will +surely get ``EAGAIN`` or ``EWOULDBLOCK`` since the socket has limited +send buffer. If that happens, we can return +:macro:`NGHTTP2_ERR_WOULDBLOCK` to signal the nghttp2 library to stop +sending further data. But writing to the bufferevent, we have to +regulate the amount data to be buffered by ourselves to avoid possible +huge memory consumption. To achieve this, we check the size of output +buffer and if it is more than or equal to +``OUTPUT_WOULDBLOCK_THRESHOLD`` bytes, we stop writing data and return +:macro:`NGHTTP2_ERR_WOULDBLOCK` to tell the library to stop calling +send_callback. + +The next bufferevent callback is ``readcb()``, which is invoked when +data is available to read in the bufferevent input buffer:: + + static void readcb(struct bufferevent *bev, void *ptr) + { + http2_session_data *session_data = (http2_session_data*)ptr; + if(session_recv(session_data) != 0) { + delete_http2_session_data(session_data); + return; + } + } + +In this function, we just call ``session_recv()`` to process incoming +data. + +The third bufferevent callback is ``writecb()``, which is invoked when +all data written in the bufferevent output buffer have been sent:: + + static void writecb(struct bufferevent *bev, void *ptr) + { + http2_session_data *session_data = (http2_session_data*)ptr; + if(evbuffer_get_length(bufferevent_get_output(bev)) > 0) { + return; + } + if(nghttp2_session_want_read(session_data->session) == 0 && + nghttp2_session_want_write(session_data->session) == 0) { + delete_http2_session_data(session_data); + return; + } + if(session_send(session_data) != 0) { + delete_http2_session_data(session_data); + return; + } + } + +First we check whether we should drop connection or not. The nghttp2 +session object keeps track of reception and transmission of GOAWAY +frame and other error conditions as well. Using these information, +nghttp2 session object will tell whether the connection should be +dropped or not. More specifically, both `nghttp2_session_want_read()` +and `nghttp2_session_want_write()` return 0, we have no business in +the connection. But since we have using bufferevent and its deferred +callback option, the bufferevent output buffer may contain the pending +data when the ``writecb()`` is called. To handle this situation, we +also check whether the output buffer is empty or not. If these +conditions are met, we drop connection. + +Otherwise, we call ``session_send()`` to process pending output +data. Remember that in ``send_callback()``, we may not write all data +to bufferevent to avoid excessive buffering. We continue process +pending data if output buffer becomes empty. + +We have already described about nghttp2 callback ``send_callback()``. +Let's describe remaining nghttp2 callbacks we setup in +``initialize_nghttp2_setup()`` function. + +The ``on_frame_recv_callback()`` function is invoked when a frame is +received from the remote peer:: + + static int on_frame_recv_callback(nghttp2_session *session, + const nghttp2_frame *frame, void *user_data) + { + http2_session_data *session_data = (http2_session_data*)user_data; + http2_stream_data *stream_data; + size_t i; + const char PATH[] = ":path"; + switch(frame->hd.type) { + case NGHTTP2_HEADERS: + if(frame->headers.cat != NGHTTP2_HCAT_REQUEST) { + break; + } + stream_data = create_http2_stream_data(session_data, frame->hd.stream_id); + nghttp2_session_set_stream_user_data(session, frame->hd.stream_id, + stream_data); + for(i = 0; i < frame->headers.nvlen; ++i) { + nghttp2_nv *nv = &frame->headers.nva[i]; + if(nv->namelen == sizeof(PATH) - 1 && + memcmp(PATH, nv->name, nv->namelen) == 0) { + size_t j; + for(j = 0; j < nv->valuelen && nv->value[j] != '?'; ++j); + stream_data->request_path = percent_decode(nv->value, j); + break; + } + } + break; + default: + break; + } + return 0; + } + +We only interested HEADERS frame in this function. Since HEADERS frame +has several roles in HTTP/2.0 protocol, we check that it is a request +HEADERS, which opens new stream. If frame is request HEADERS, then we +create ``http2_stream_data`` object to store stream related data. We +associate created ``http2_stream_data`` object to the stream in +nghttp2 session object using `nghttp2_set_stream_user_data()` in order +to get the object without searching through doubly linked list. + +In this example server, we want to server static file relative to the +current working directory the program was invoked. We search ``:path`` +header field in request headers and keep the requested path in +``http2_stream_data`` object. In this example program, we ignore +``:method`` header field and always treat the request as GET request. + +It is ok for the server to start sending response in this callback. In +this example, we defer it to ``on_request_recv_callback()`` function. + +The ``on_request_recv_callback()`` function is invoked when all HTTP +request, including entity body, was received:: + + static int on_request_recv_callback(nghttp2_session *session, + int32_t stream_id, void *user_data) + { + int fd; + http2_session_data *session_data = (http2_session_data*)user_data; + http2_stream_data *stream_data; + nghttp2_nv hdrs[] = { + MAKE_NV(":status", "200") + }; + char *rel_path; + + stream_data = (http2_stream_data*)nghttp2_session_get_stream_user_data + (session, stream_id); + if(!stream_data->request_path) { + if(error_reply(session, stream_data) != 0) { + return NGHTTP2_ERR_CALLBACK_FAILURE; + } + return 0; + } + fprintf(stderr, "%s GET %s\n", session_data->client_addr, + stream_data->request_path); + if(!check_path(stream_data->request_path)) { + if(error_reply(session, stream_data) != 0) { + return NGHTTP2_ERR_CALLBACK_FAILURE; + } + return 0; + } + for(rel_path = stream_data->request_path; *rel_path == '/'; ++rel_path); + fd = open(rel_path, O_RDONLY); + if(fd == -1) { + if(error_reply(session, stream_data) != 0) { + return NGHTTP2_ERR_CALLBACK_FAILURE; + } + return 0; + } + stream_data->fd = fd; + + if(send_response(session, stream_id, hdrs, ARRLEN(hdrs), fd) != 0) { + close(fd); + return NGHTTP2_ERR_CALLBACK_FAILURE; + } + return 0; + } + +First we retrieve ``http2_stream_data`` object associated to the +stream in ``on_frame_recv_callback()``. It is done using +`nghttp2_session_get_stream_user_data()`. If the requested path cannot +be served for some reasons (e.g., file is not found), we send 404 +response, which is done in ``error_reply()``. Otherwise, we open +requested file and send its content. We send 1 header field +``:status`` as a response header. + +Sending content of a file is done in ``send_response()`` function:: + + static int send_response(nghttp2_session *session, int32_t stream_id, + nghttp2_nv *nva, size_t nvlen, int fd) + { + int rv; + nghttp2_data_provider data_prd; + data_prd.source.fd = fd; + data_prd.read_callback = file_read_callback; + + rv = nghttp2_submit_response(session, stream_id, nva, nvlen, &data_prd); + if(rv != 0) { + warnx("Fatal error: %s", nghttp2_strerror(rv)); + return -1; + } + return 0; + } + +The nghttp2 library uses :type:`nghttp2_data_provider` structure to +send entity body to the remote peer. The ``source`` member of this +structure is a union and it can be either void pointer or int which is +intended to be used as file descriptor. In this example server, we use +file descriptor. We also set ``file_read_callback()`` callback +function to read content of the file:: + + static ssize_t file_read_callback + (nghttp2_session *session, int32_t stream_id, + uint8_t *buf, size_t length, int *eof, + nghttp2_data_source *source, void *user_data) + { + int fd = source->fd; + ssize_t r; + while((r = read(fd, buf, length)) == -1 && errno == EINTR); + if(r == -1) { + return NGHTTP2_ERR_TEMPORAL_CALLBACK_FAILURE; + } + if(r == 0) { + *eof = 1; + } + return r; + } + +If error happens while reading file, we return +:macro:`NGHTTP2_ERR_TEMPORAL_CALLBACK_FAILURE`. This tells the library +to send RST_STREAM to the stream. When all data is read, set 1 to +``*eof`` to tell the nghttp2 library that we have finished reading +file. + +The `nghttp2_submit_response()` is used to send response to the remote +peer. + +The ``on_stream_close_callback()`` function is invoked when the stream +is about to close:: + + static int on_stream_close_callback(nghttp2_session *session, + int32_t stream_id, + nghttp2_error_code error_code, + void *user_data) + { + http2_session_data *session_data = (http2_session_data*)user_data; + http2_stream_data *stream_data; + + stream_data = nghttp2_session_get_stream_user_data(session, stream_id); + remove_stream(session_data, stream_data); + delete_http2_stream_data(stream_data); + return 0; + } + +We destroy ``http2_stream_data`` object in this function since the +stream is about to close and we no longer to use that object. + + +libevent-server.c +----------------- + +.. literalinclude:: ../examples/libevent-server.c