This is the second part of a two part series on HTTP basics. In this second part, we cover several attributes of the HTTP protocol such as encoding, HTTP headers and authentication in more detail.
The query string is defined using the question mark (?) character after the URL within an HTTP request. The query string defines a series of key-value parameters separated by the ampersand (&) character.
GET /search?query=example&lang=en_US HTTP/1.1 Host: www.example.com
Query string parameters are one of the primary mechanisms web applications use as user input. It’s therefore no surprise that most web application vulnerabilities arise from poorly handled user input within query string parameters.
URL encoding is a way to represent characters that can not (or should not) be present within URLs to be represented safely within a URL. This allows encoding and decoding of characters which would otherwise cause problems or conflicts. The following are some examples of URL encoded characters.
Since HTTP is a stateless protocol, Cookies are a built-in mechanism to pass state data to the server. Typical examples included in Cookies would be state information such as session identifiers and user preferences.
Cookies are crucial to security since they are widely used to store session information. This means that if an attacker can steal a user’s Cookie (using attacks such as Cross-site Scripting for example), in many web applications, this alone provides the attacker with all they need to impersonate that user.
Cookies are set by the server using the Set-Cookie HTTP response header, the browser then stores the Cookie value and submits it with every request. This may also introduces vulnerabilities such as Cross-site Request Forgery. The Cookie value may contain several values delimited by a semicolon (;).
Additional security features around Cookies include
The HTTP protocol includes two types of built-in authentication mechanisms, Basic and Digest. While these two methods are built-in to HTTP they are by no means the only authentication methods that can leverage HTTP, including NTLM, IWA (integrated Windows Authentication, a.k.a Kerberos) and TLS client certificates. Additionally, form authentication, OAuth/OAuth2, SAML, JWT and a whole host of other types of authentication options re-use features within HTTP such as form data or headers to authenticate a client.
Basic authentication is a built-in HTTP authentication method. When a client sends an HTTP request to a server which requires Basic authentication, the server will respond with a 401 HTTP response and a WWW-Authenticate header containing the value basic. The client then submits a username and a password separated by a colon (:) and base64 encoded.
It’s important to note that Basic authentication sends credentials in the clear (without any form of encryption). This means that for Basic authentication alone is not secure and must be paired with the use of TLS.
Digest authentication is also built-in to HTTP and similarly to Basic authentication, it also returns a 401 HTTP response and a
WWW-Authenticate header. In the case of digest, the
WWW-Authenticate header will contain the value of digest together with a nonce (number only used once) and a realm (defines a URL path which may share the same credentials)
The HTTP client would then concatenate the supplied credentials together with the with the nonce and realm and produce an MD5 hash (1st hash). The HTTP client then concatenates the HTTP method, and the URI and generates an MD5 hash (2nd hash). The HTTP client then sends an
Authorize header containing the realm, nonce, URI, and the response. The response is an MD5 sum of the two hashes combined.
While digest is a more secure alternative to Basic authentication, it is still highly advised for any authentication traffic to be transmitted over a TLS connection.
Form based authentication is by far the most popular kind of authentication. It’s also not standard, in the sense that any application developer can dictate how an HTTP client should authenticate to an application.
Typically the HTTP client would send a POST request to the server with the combination of a username and a password, after which, if successful, the server will respond with some kind of token — this token could be placed in a
Set-Cookie HTTP header, which would set a Cookie in the browser (meaning that this value will henceforth be passed with each request to the server).
<form name="login" action="https://login.example.com" method="post"> <input name="username" type="text"> <input name="password" type="password"> <input value="Login" type="submit"> </form>
This would send data from input fields
password (field names are arbitrary) in a POST request to https://login.example.com. The POST request would be as follows.
POST / HTTP/1.1 Host: login.example.com
HTTP headers are a way for an HTTP client and server to pass additional information within requests and responses. HTTP headers consist of a case-insensitive key (may not contain spaces), followed by a colon (:) which is in turn followed by the header’s value (may contain spaces). The header is terminated by a CRLF (line-break).
It’s worth noting that while there are a number of standard HTTP headers, the HTTP protocol allows custom headers. Typically, custom headers start with an X- (for example
X-Custom-Header), however, this is simply a widely adopted convention and not something enforced by the HTTP protocol.
The following are some examples of commonly seen HTTP headers. A complete list of standard HTTP headers may be found here.
|Connection||Indicates to the sender/receiver to keep the TCP connection open, or close it.||Connection: close||Request/Response|
|Content-Type||Indicates the MIME type of the request/response body||Content-Type: text/html; charset=utf-8||Request/Response|
|Content-Encoding||Indicates the type of encoding used for the request/response body||Content-Encoding: gzip||Request/Response|
|Content-Length||Indicates the size in bytes of the request/response body||Content-Length: 512||Request/Response|
|Cookie||Contains Cookie values which were set using the Set-Cookie header||Cookie: session=E8DF34AE331AE94F5549B10B7C7A4FA; color=blue||Request|
|Host||Contains the hostname of the URL being requested. This is a required header in HTTP/1.1 and is important since the same server may serve different sites based on this header. This header is also important to keep in mind when defending against host header attacks.||Host: example.com||Request|
|Referer||Indicates to the server the page from which a link was clicked from. The header was originally misspelled in the original specification and has remained so.||Referer: https://www.example.com/page_1||Request|
|Authorization||Contains authentication method, together with credentials||Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=||Request|
|User-Agent||Contains a string to identify the client (browser or other tool) that is making the request||User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36||Request|
|Accept||Indicates to the server what content MIME type it will accept||Accept: text/html||Request|
|Accept-Encoding||Indicates to the server what types of encoding it will accept||Accept-Encoding: compress, gzip||Request|
|Set-Cookie||Sets a cookie in the browser that will be later submitted.||Set-Cookie: session=E8DF34AE331AE94F5549B10B7C7A4FA; color=blue||Response|
|Server||Indicates the type and version of the server. This is information that could be useful for attackers.||Server: nginx/1.13.9||Response|
This sums-up some useful basics of the HTTP protocol that should get you up to speed with the terminology surrounding HTTP. Now that you’ve covered the basics, you should be able to understand the reports containing HTTP requests, responses and other HTTP attributes, which means the next time you generate a Developer Report, you should be sorted.