A Security-focused Introduction to HTTP, Part 2

This is the second part of a two part series on HTTP basics. In this second part, we cover several attributes of the HTTP protocol such as encoding, HTTP headers and authentication in more detail.

Query strings

The query string is defined using the question mark (?) character after the URL within an HTTP request. The query string defines a series of key-value parameters separated by the ampersand (&) character.

GET /search?query=example&lang=en_US HTTP/1.1
Host: www.example.com

Query string parameters are one of the primary mechanisms web applications use as user input. It’s therefore no surprise that most web application vulnerabilities arise from poorly handled user input within query string parameters.

URL encoding

URL encoding is a way to represent characters that can not (or should not) be present within URLs to be represented safely within a URL. This allows encoding and decoding of characters which would otherwise cause problems or conflicts. The following are some examples of URL encoded characters.

Character URL-encoded Character
@ %40
%22
< %3C
% %25

Cookies

Since HTTP is a stateless protocol, Cookies are a built-in mechanism to pass state data to the server. Typical examples included in Cookies would be state information such as session identifiers and user preferences.

Cookies are crucial to security since they are widely used to store session information. This means that if an attacker can steal a user’s Cookie (using attacks such as Cross-site Scripting for example), in many web applications, this alone provides the attacker with all they need to impersonate that user.

Cookies are set by the server using the Set-Cookie HTTP response header, the browser then stores the Cookie value and submits it with every request. This may also introduces vulnerabilities such as Cross-site Request Forgery. The Cookie value may contain several values delimited by a semicolon (;).

Additional security features around Cookies include HttpOnly, Expires and Secure flags.

Authentication

The HTTP protocol includes two types of built-in authentication mechanisms, Basic and Digest. While these two methods are built-in to HTTP they are by no means the only authentication methods that can leverage HTTP, including NTLM, IWA (integrated Windows Authentication, a.k.a Kerberos) and TLS client certificates. Additionally, form authentication, OAuth/OAuth2, SAML, JWT and a whole host of other types of authentication options re-use features within HTTP such as form data or headers to authenticate a client.

Basic Authentication

Basic authentication is a built-in HTTP authentication method. When a client sends an HTTP request to a server which requires Basic authentication, the server will respond with a 401 HTTP response and a WWW-Authenticate header containing the value basic. The client then submits a username and a password separated by a colon (:) and base64 encoded.

It’s important to note that Basic authentication sends credentials in the clear (without any form of encryption). This means that for Basic authentication alone is not secure and must be paired with the use of TLS.

Digest

Digest authentication is also built-in to HTTP and similarly to Basic authentication, it also returns a 401 HTTP response and a WWW-Authenticate header. In the case of digest, the WWW-Authenticate header will contain the value of digest together with a nonce (number only used once) and a realm (defines a URL path which may share the same credentials)

The HTTP client would then concatenate the supplied credentials together with the with the nonce and realm and produce an MD5 hash (1st hash). The HTTP client then concatenates the HTTP method, and the URI and generates an MD5 hash (2nd hash). The HTTP client then sends an Authorize header containing the realm, nonce, URI, and the response. The response is an MD5 sum of the two hashes combined.

While digest is a more secure alternative to Basic authentication, it is still highly advised for any authentication traffic to be transmitted over a TLS connection.

Form based

Form based authentication is by far the most popular kind of authentication. It’s also not standard, in the sense that any application developer can dictate how an HTTP client should authenticate to an application.

Typically the HTTP client would send a POST request to the server with the combination of a username and a password, after which, if successful, the server will respond with some kind of token — this token could be placed in a Set-Cookie HTTP header, which would set a Cookie in the browser (meaning that this value will henceforth be passed with each request to the server).

Such POST requests can be made by the browser by using a <form> HTML tag, or via JavaScript. The following is an example of an HTML form.

<form name="login" action="https://login.example.com" method="post">
<input name="username" type="text">
<input name="password" type="password">
<input value="Login" type="submit">
</form>

This would send data from input fields username and password (field names are arbitrary) in a POST request to https://login.example.com. The POST request would be as follows.

POST / HTTP/1.1
Host: login.example.com
username=myusername&password=mypassword

Headers

HTTP headers are a way for an HTTP client and server to pass additional information within requests and responses. HTTP headers consist of a case-insensitive key (may not contain spaces), followed by a colon (:) which is in turn followed by the header’s value (may contain spaces). The header is terminated by a CRLF (line-break).

It’s worth noting that while there are a number of standard HTTP headers, the HTTP protocol allows custom headers. Typically, custom headers start with an X- (for example X-Custom-Header), however, this is simply a widely adopted convention and not something enforced by the HTTP protocol.

The following are some examples of commonly seen HTTP headers. A complete list of standard HTTP headers may be found here.

Header Description Example Type
Connection Indicates to the sender/receiver to keep the TCP connection open, or close it. Connection: close Request/Response
Content-Type Indicates the MIME type of the request/response body Content-Type: text/html; charset=utf-8 Request/Response
Content-Encoding Indicates the type of encoding used for the request/response body Content-Encoding: gzip Request/Response
Content-Length Indicates the size in bytes of the request/response body Content-Length: 512 Request/Response
Cookie Contains Cookie values which were set using the Set-Cookie header Cookie: session=E8DF34AE331AE94F5549B10B7C7A4FA; color=blue Request
Host Contains the hostname of the URL being requested. This is a required header in HTTP/1.1 and is important since the same server may serve different sites based on this header. This header is also important to keep in mind when defending against host header attacks. Host: example.com Request
Referer Indicates to the server the page from which a link was clicked from. The header was originally misspelled in the original specification and has remained so. Referer: https://www.example.com/page_1 Request
Authorization Contains authentication method, together with credentials Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ= Request
User-Agent Contains a string to identify the client (browser or other tool) that is making the request User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36 Request
Accept Indicates to the server what content MIME type it will accept Accept: text/html Request
Accept-Encoding Indicates to the server what types of encoding it will accept Accept-Encoding: compress, gzip Request
Set-Cookie Sets a cookie in the browser that will be later submitted. Set-Cookie: session=E8DF34AE331AE94F5549B10B7C7A4FA; color=blue Response
Server Indicates the type and version of the server. This is information that could be useful for attackers. Server: nginx/1.13.9 Response

This sums-up some useful basics of the HTTP protocol that should get you up to speed with the terminology surrounding HTTP. Now that you’ve covered the basics, you should be able to understand the reports containing HTTP requests, responses and other HTTP attributes, which means the next time you generate a Developer Report, you should be sorted.


A Security-focused Introduction to HTTP, Part 2

Share this post

Leave a Reply

Your email address will not be published.