HTTP: The Protocol Every Web Developer Must Know Part 1

The status code is important and tells the client how to interpret the server response.

• Via header is used in a TRACE message and updated by all intermittent proxies and gateways
• Pragma is considered a custom header and may be used to include implementation-specific headers. The most commonly used pragma-directive is Pragma: no-cache, which really is Cache-Control: no-cache under HTTP/1.1. This will be covered in Part 2 of the article.
• The Date header field is used to timestamp the request/response message
• Upgrade is used to switch protocols and allow a smooth transition to a newer protocol.
• Transfer-Encoding is generally used to break the response into smaller parts with the Transfer-Encoding: chunked value. This is a new header in HTTP/1.1 and allows for streaming of response to the client instead of one big payload.

Entity headers

Request and Response messages may also include entity headers to provide meta-information about the the content (aka Message Body or Entity). These headers include:

entity-header  = Allow                                   | Content-Encoding                 | Content-Language                 | Content-Length                   | Content-Location                 | Content-MD5                      | Content-Range                    | Content-Type                     | Expires                          | Last-Modified  

All of the Content- prefixed headers provide information about the structure, encoding and size of the message body. Some of these headers need to be present if the entity is part of the message.

The Expires header indicates a timestamp of whent he entity expires. Interestingly, a “never expires” entity is sent with a timestamp of one year into the future. The Last-Modified header indicates the last modification timestamp for the entity.

Custom headers can also be created and sent by the client; they will be treated as entity headers by the HTTP protocol.

This is really an extension mechanism, and some client-server implementations may choose to communicate specifically over these extension headers. Although HTTP supports custom headers, what it really looks for are the request and response headers, which we cover next.

Request Format

The request message has the same generic structure as above, except for the request line which looks like:

Request-Line = Method SP URI SP HTTP-Version CRLFMethod = "OPTIONS"       | "HEAD"         | "GET"         | "POST"         | "PUT"         | "DELETE"         | "TRACE"

SP is the space separator between the tokens. HTTP-Version is specified as “HTTP/1.1″ and then followed by a new line. Thus, a typical request message might look like:

GET /articles/http-basics HTTP/1.1Host: www.articles.comConnection: keep-aliveCache-Control: no-cachePragma: no-cacheAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

Note the request line followed by many request headers. The Host header is mandatory for HTTP/1.1 clients. GET requests do not have a message body, but POST requests can contain the post data in the body.

The request headers act as modifiers of the request message. The complete list of known request headers is not too long, and is provided below. Unknown headers are treated as entity-header fields.

request-header = Accept                                  | Accept-Charset                   | Accept-Encoding                  | Accept-Language                  | Authorization                    | Expect                           | From                             | Host                             | If-Match                         | If-Modified-Since                | If-None-Match                    | If-Range                         | If-Unmodified-Since               | Max-Forwards                      | Proxy-Authorization               | Range                             | Referer                           | TE                                | User-Agent

The Accept prefixed headers indicate the acceptable media-types, languages and character sets on the client. From, Host, Referer and User-Agent identify details about the client that initiated the request. The If- prefixed headers are used to make a request more conditional, and the server returns the resource only if the condition matches. Otherwise, it returns a 304 Not Modified. The condition can be based on a timestamp or an ETag (a hash of the entity).

Response Format

The response format is similar to the request message, except for the status line and headers. The status line has the following structure:

Status-Line = HTTP-Version SP Status-Code SP Reason-Phrase CRLF

• HTTP-Version is sent as HTTP/1.1
• The Status-Code is one of the many statuses discussed earlier.
• The Reason-Phrase is a human-readable version of the status code.

A typical status line for a successful response might look like so:

HTTP/1.1 200 OK

The response headers are also fairly limited, and the full set is given below:

 response-header = Accept-Ranges                 | Age                 | ETag                               | Location                           | Proxy-Authenticate                 | Retry-After                        | Server                             | Vary                               | WWW-Authenticate

• Age is the time in seconds since the message was generated on the server.
• ETag is the MD5 hash of the entity and used to check for modifications.
• Location is used when sending a redirection and contains the new URL.
• Server identifies the server generating the message.

It’s been a lot of theory upto this point, so I won’t blame you for drowsy eyes. In the next sections, we will get more practical and take a survey of the tools, frameworks and libraries.

Tools to View HTTP Traffic

There are a number of tools available to monitor HTTP communication. Here, we list some of the more popular tools.

Undoubtedly, the Chrome/Webkit inspector is a favorite amongst web developers:

There are also web debugging proxies, like Fiddler on Windows and Charles Proxy for OSX. My colleague, Rey Bango wrote an excellent article on this topic.

For the command line, we have utilities like curl, tcpdump and tshark for monitoring HTTP traffic.

---

Using HTTP in Web Frameworks and Libraries

Now that we have looked at the request/response messages, it’s time that we learn how libraries and frameworks expose it in the form of an API. We’ll use ExpressJS for Node, Ruby on Rails, and jQuery Ajax as our examples.

ExpressJS

If you are building web servers in NodeJS, chances are high that you’ve considered ExpressJS. ExpressJS was originally inspired by a Ruby Web framework, called Sinatra. As expected, the API is also equally influenced.

Because we are dealing with a server-side framework, there are two primary tasks when dealing with HTTP messages:

• Read URL fragments and request headers.
• Write response headers and body

Understanding HTTP is crucial for having a clean, simple and RESTful interface between two endpoints.

ExpressJS provides a simple API for doing just that. We won’t cover the details of the API. Instead, we will provide links to the detailed documentation on ExpressJS guides. The methods in the API are self-explanatory in most cases. A sampling of the request-related API is below:

• req.body: get the request body.
• req.query: get the query fragment of the URL.
• req.originalUrl
• req.host: reads the Host header field.
• req.accepts: reads the acceptable MIME-types on the client side.
• req.get OR req.header: read any header field passed as argument.

On the way out to the client, ExpressJS provides the following response API:

• res.status: set an explicit status code.
• res.set: set a specific response header.
• res.send: send HTML, JSON or an octet-stream.
• res.sendFile: transfer a file to the client.
• res.render: render an express view template.
• res.redirect: redirect to a different route. Express automatically adds the default redirection code of 302.

Ruby on Rails

The request and response messages are mostly the same, except for the first line and message headers.

In Rails, the ActionController and ActionDispatch modules provide the API for handling request and response messages.

ActionController provides a high level API to read the request URL, render output and redirect to a different end-point. An end-point (aka route) is handled as an action method. Most of the necessary context information inside an action-method is provided via the request, response and params objects.

• params: gives access to the URL parameters and POST data.
• request: contains information about the client, headers and URL.
• response: used to set headers and status codes.
• render: render views by expanding templates.
• redirect_to: redirect to a different action-method or URL.

ActionDispatch provides fine-grained access to the request/response messages, via the ActionDispatch::Request and ActionDispatch::Response classes. It exposes a set of query methods to check the type of request (get?(), post?(), head?(), local?()). Request headers can be directly accessed via the request.headers() method.

On the response side, it provides methods to set cookies(), location=() and status=(). If you feel adventurous, you can also set the body=() and bypass the Rails rendering system.

jQuery Ajax

Because jQuery is primarily a client-side library, its Ajax API provides the opposite of a server-side framework. In other words, it allows you to read response messages and modify request messages. jQuery exposes a simple API via jQuery.ajax(settings):

By passing a settings object with the beforeSend callback, we can modify the request headers. The callback receives the jqXHR (jQuery XMLHttpRequest) object that exposes a method, called setRequestHeader() to set headers.

$.ajax({    url: 'https://www.articles.com/latest',    type: 'GET',    beforeSend: function (jqXHR) {      jqXHR.setRequestHeader('Accepts-Language', 'en-US,en');    }  });

• The jqXHR object can also be used to read the response headers with the jqXHR.getResponseHeader().
• If you want to take specific actions for various status codes, you can use the statusCode callback:

$.ajax({  statusCode: {    404: function() {      alert("page not found");    }  }});

---

Summary

So that sums up our quick tour of the HTTP protocol.

We reviewed URL structure, verbs and status codes: the three pillars of HTTP communication.

The request and response messages are mostly the same, except for the first line and message headers. Finally, we reviewed how you can modify the request and response headers in web frameworks and libraries.

Understanding HTTP is crucial for having a clean, simple, and RESTful interface between two endpoints. On a larger scale, it also helps when designing your network infrastructure and providing a great experience to your end users.

In part two, we’ll review connection handling, authentication and caching! See you then.

---

References

• HTTP specification
• HTTP Definitive Guide