If you’re new to hacking, it will help to have a basic understanding of how the internet works and what happens under the hood when you enter a URL into a browser’s address bar. Although navigating to a website might seem simple, it involves many hidden processes, such as preparing an HTTP request, identifying the domain to send the request to, translating the domain to an IP address, sending the
request, rendering a response, and so on. In this chapter, you’ll learn basic concepts and terminology,
such as vulnerabilities, bug bounties, clients, servers, IP addresses, and HTTP. You’ll get a general understanding of how performing unintended actions and providing unexpected input or access to private information can result in vulnerabilities. Then, we’ll see what happens when you enter a URL in your browser’s address bar, including what HTTP requests and responses look like and the various HTTP action verbs. We’ll end the chapter with an understanding of what it means to say HTTP is stateless
VULNERABILITIES AND BUG BOUNTIES
A vulnerability is a weakness in an application that allows a malicious person to perform some unpermitted action or gain access to information they shouldn’t otherwise be allowed to
access. As you learn and test applications, keep in mind that vulnerabilities can result from attackers performing intended and unintended actions. For example, changing the ID of a record identifier to access information you shouldn’t have access to is an example of an unintended action. Suppose a website allowed you to create a profile with your name, email, birthday, and address. It would keep your
information private and share it only with your friends. But if the website allowed anyone to add you as a friend without your permission, this would be a vulnerability. Even though the site kept your information private from non-friends, by allowing anyone to add you as a friend, anyone could access your information. As you test a site, always consider how someone could abuse existing functionality.
A bug bounty is a reward a website or company gives to anyone who ethically discovers a vulnerability and reports it to that website or company. Rewards are often monetary and range from tens of dollars to tens of thousands of dollars. Other examples of bounties include cryptocurrencies, air miles, reward points, service credits, and so on. When a company offers bug bounties, it creates a program,
a term that we’ll use in this book to denote the rules and framework established by companies for people who want to test the company for vulnerabilities. Note that this is different from companies that operate a vulnerability disclosure program (VDP). Bug bounties offer some monetary reward, whereas a VDP does not offer payment (though a company may award swag). A VDP is just a way for ethical hackers to
report vulnerabilities to a company for that company to fix. Although not all reports included in this book were rewarded, they’re all examples from hackers participating in bug bounty programs.
CLIENT AND SERVER
Your browser relies on the internet, which is a network of computers that send messages to each other. We call these messages packets. Packets include the data you’re sending and information about where that data is coming from and where it’s going. Every computer on the internet has an address for
sending packets to it. But some computers only accept certain types of packets, and others only allow packets from a restricted list of other computers. It’s then up to the receiving computer to determine what to do with the packets and how to respond. For the purposes of this book, we’ll focus only on the
data included in the packets (the HTTP messages), not the packets themselves. I’ll refer to these computers as either clients or servers. The computer initiating requests is typically referred to as the
client regardless of whether the request is initiated by a browser, command line, or so on. Servers refer to the websites and web applications receiving the requests. If the concept is applicable to either clients or servers, I refer to computers in general. Because the internet can include any number of computers
talking to each other, we need guidelines for how computers should communicate over the internet. This takes the form of Request for Comment (RFC) documents, which define standards for how computers should behave. For example, the Hypertext Transfer Protocol (HTTP) defines how your internet browser communicates with a remote server using Internet Protocol (IP). In this scenario, both the client and server must agree to implement the same standards so they can understand the packets each is sending and receiving.
WHAT HAPPENS WHEN YOU VISIT A WEBSITE
Because we’ll focus on HTTP messages in this book, this section provides you with a high-level overview of the process that occurs when you enter a URL in your browser’s address bar.
Step 1: Extracting the Domain Name
Once you enter http://www.google.com/, your browser determines the domain name from the URL. A domain name identifies which website you’re trying to visit and must adhere to specific rules as defined by RFCs. For example, a domain name can only contain alphanumeric characters and underscores. An exception is internationalized domain names, which are beyond the scope of this book. To learn more, refer to RFC 3490, which defines their usage. In this case, the domain is www.google.com. The domain serves as one way to find the server’s address.
Step 2: Resolving an IP Address
After determining the domain name, your browser uses IP to look up the IP address associated with the domain. This process is referred to as resolving the IP address, and every domain on the internet must resolve to an IP address to work. Two types of IP addresses exist: Internet Protocol version 4 (IPv4) and Internet Protocol version 6 (IPv6). IPv4 addresses are structured as four numbers connected by periods, and each number falls in a range from 0 to 255. IPv6 is the newest version of the Internet Protocol. It was designed to address the problem of available IPv4 addresses running out. IPv6 addresses are made up of eight groups of four hexadecimal digits separated by colons, but methods exist to shorten IPv6
addresses. For example, 8.8.8.8 is an IPv4 address, and 2001:4860:4860::8888 is a shortened IPv6 address.
To look up an IP address using just the domain name, your computer sends a request to Domain Name System (DNS) servers, which consist of specialized servers on the internet that have a registry of all domains and their matching IP addresses. The preceding IPv4 and IPv6 addresses are Google DNS servers. In this example, the DNS server you connect to would match www.google.com to the IPv4 address 216.58.201.228 and send that back to your computer. To learn more about a site’s IP address, you can use the command dig A site.com from your terminal and replace site.com with the site you’re looking
up.
Step 3: Establishing a TCP Connection
Next, the computer attempts to establish a Transmission Control Protocol (TCP) connection with the IP address on port 80 because you visited a site using http://. The details of TCP aren’t important other than to note that it’s another protocol that defines how computers communicate with each other. TCP provides two-way communication so that message recipients can verify the information they receive and nothing
is lost in transmission. The server you’re sending a request to might be running multiple services (think of a service as a computer program), so it uses ports to identify specific processes to receive requests. You can think of ports as a server’s doors to the internet. Without ports, services would have to compete for the information being sent to the same place. This means that we need another standard to define how services cooperate with each other and ensure that the data for one service isn’t stolen by another. For example, port 80 is the standard port for sending and receiving unencrypted HTTP requests. Another
common port is 443, which is used for encrypted HTTPS requests. Although port 80 is standard for HTTP and 443 is standard for HTTPS, TCP communication can happen on any port, depending on how an administrator configures an application. You can establish your own TCP connection to a website
on port 80 by opening your terminal and running nc 80. This line uses the Netcat utility nc command to
create a network connection for reading and writing messages.
Step 4: Sending an HTTP Request
Continuing with http://www.google.com/ as an example, if the connection in step 3 is successful, your browser should prepare and send an HTTP request, as shown in Listing 1-1:
➊ GET / HTTP/1.1
➋ Host: www.google.com
➌ Connection: keep-alive
➍ Accept: application/html, /
➎ User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36
The browser makes a GET request to the / path ➊, which is the website’s root. A website’s content is organized into paths, just like the folders and files on your computer. As you get deeper into each folder, the path you take is denoted by recording each folder’s name followed by a /. When you visit the first page of a website, you access the root path, which is just a /. The browser also indicates it’s using the HTTP version 1.1 protocol. A GET request just retrieves information. We’ll learn more about it later.
The host header ➋ holds an additional piece of information that is sent as part of the request. HTTP 1.1 needs it to identify where a server at the given IP address should send the request because IP addresses can host multiple domains. A connection header ➌ indicates the request to keep the connection with the
server open to avoid the overhead of constantly opening and closing connections. You can see the expected response format at ➍. In this case, we’re expecting application/html but will accept any format,
as indicated by the wildcard (/). There are hundreds of possible content types, but for our purposes, you’ll see application/html, application/json, application/octet-stream, and text/plain most often. Finally, the User-Agent ➎ denotes the software responsible for sending the request.
Step 5: Server Response
In response to our request, the server should respond with something that looks like Listing 1-2:
HTTP message body is the text associated with a request or response ➌. In this case, we’ve removed the content and replaced it with –snip– because of how big the response body from Google is. This text in a response is usually the HTML for a web page but could be JSON for an application programming interface, file contents for a file download, and so on. The Content-Type header ➋ informs the browsers of the body’s media type. The media type determines how a browser will render body contents. But browsers don’t always use the value returned from an application; instead, browsers perform MIME sniffing, reading the first bit of the body contents to determine the media type for themselves. Applications can disable this browser behavior by including the header XContent-Type-Options: nosniff, which is not included in the preceding example. Other response codes starting with 3 indicate a redirection, which instructs your browser to make an additional request. For example, if Google theoretically needed to permanently redirect you from one URL to another, it could use a 301 response. In contrast, a 302 is a temporary redirect. When a 3xx response is received, your browser should make a new HTTP request to the URL defined in a Location header, as follows.
HTTP/1.1 301 Found
Location: https://www.google.com/
Responses starting with a 4 typically indicate a user error, such as response 403 when a request doesn’t include proper identification to authorize access to content despite providing a valid HTTP request. Responses starting with a 5 identify some type of server error, such as 503, which indicates a server is
unavailable to handle the sent request.
Step 6: Rendering the Response
Because the server sent a 200 response with the content type text/html, our browser will begin rendering the contents it received. The response’s body tells the browser what should be presented to the user.
For our example, this would include HTML for the page structure; Cascading Style Sheets (CSS) for the styles and layout; and JavaScript to add additional dynamic functionality and media, such as images or videos. It’s possible for the server to return other content, such as XML, but we’ll stick to the basics for this example. Chapter 11 discusses XML in more detail. Because it’s possible for web pages to reference external files such as CSS, JavaScript, and media, the browser might make additional HTTP requests for all a web page’s required files. While the browser is requesting those additional files, it continues parsing the response and presenting the body to you as a web page. In this case, it will render Google’s home page,
www.google.com. Note that JavaScript is a scripting language supported by every major browser. JavaScript allows web pages to have dynamic functionality, including the ability to update content
on a web page without reloading the page, check whether your password is strong enough (on some websites), and so on. Like other programming languages, JavaScript has built-in functions and can store values in variables and run code in response to events on a web page. It also has access to various
browser application programming interfaces (APIs). These APIs enable JavaScript to interact with other systems, the most important of which may be the document object model (DOM). The DOM allows JavaScript to access and manipulate a web page’s HTML and CSS. This is significant because if an
attacker can execute their own JavaScript on a site, they’ll have access to the DOM and can perform actions on the site on behalf of the targeted user. Chapter 7 explores this concept further.
HTTP REQUESTS The agreement between client and server on how to handle HTTP messages includes defining request methods. A request method indicates the purpose of the client’s request and what
the client expects as a successful result. For example, in Listing 1-1, we sent a GET request to http://www.google.com/ implying we expect only the contents of http://www.google.com/ to be returned and no other actions to be performed. Because the internet is designed as an interface between remote computers, request methods were developed and implemented to distinguish between the actions being
invoked. The HTTP standard defines the following request methods: GET, HEAD, POST, PUT, DELETE, TRACE, CONNECT, and OPTIONS (PATCH was also proposed but not commonly implemented in the HTTP RFC). At the time of this writing, browsers will only send GET and POST requests using HTML. Any PUT,
PATCH, or DELETE request is the result of JavaScript’s invoking the HTTP request. This will have implications later in the book when we consider vulnerability examples in applications expecting these method types. The next section provides a brief overview of request methods you’ll find in this book.
Request Methods The GET method retrieves whatever information is identified by the request Uniform Resource Identifier (URI). The term URI is commonly used synonymously with Uniform Resource Locator (URL). Technically, a URL is a type of URI that defines a resource and includes a way to locate that resource
by way of its network location.
For example, http://www.google.com//file.txt and //file.txt are valid URIs. But only
http://www.google.com//file.txt is a valid URL because it identifies how to locate the resource via the domain http://www.google.com. Despite the nuance, we’ll use URL throughout the book when referencing any resource identifiers. While there is no way to enforce this requirement, GET requests shouldn’t alter data; they should just retrieve data from a server and return it in the HTTP message body. For example, on a social media site, a GET request should return your profile name but not update your profile. This behavior is critical for the cross-site request forgery (CSRF) vulnerabilities discussed in Chapter 4. Visiting any URL or website link (unless invoked by JavaScript) causes your browser to send a GET request to the intended server. This behavior is crucial to the open redirect vulnerabilities discussed in Chapter 2. The HEAD method is identical to the GET method except the server must not return a message body in the response. The POST method invokes some function on the receiving server, as determined by the server. In other words, typically there will be some type of backend action performed, such as creating a comment, registering a user, deleting an account, and so on. The action performed by the server in response to a POST can vary. Sometimes, the server may take no action at all.
For example, a POST request could cause an error to occur while a request is being processed, and a record wouldn’t be saved on the server. The PUT method invokes some function that refers to an
already existing record on the remote website or application. For example, it might be used when updating an account, a blog post, or so on that already exists. Again, the action performed can vary and might result in the server taking no action at all. The DELETE method requests that the remote server delete a remote resource identified with a URI. The TRACE method is another uncommon method; it is used to reflect the request message back to the requester. It allows the requester to see what is being received by the server and to use that information for testing and collecting diagnostic
information. The CONNECT method is reserved for use with a proxy, a server that forwards requests to other servers. This method starts two-way communications with a requested resource. For example, the CONNECT method can access websites that use HTTPS via a proxy. The OPTIONS method requests information from a server about the communication options available. For example, by calling for OPTIONS, you can find out whether the server accepts GET, POST, PUT, DELETE, and OPTIONS calls. This method won’t indicate whether a server accepts HEAD or TRACE calls. Browsers automatically send this type of request for specific content types, such as application/json. This method, referred to as
a preflight OPTIONS call, is discussed more in depth in Chapter 4 because it serves as a CSRF vulnerability protection. HTTP Is Stateless HTTP requests are stateless, which means that every request sent to a server is treated as a brand-new request. The server knows nothing about its previous communication with your
browser when receiving a request. This is problematic for most sites because the sites want to remember who you are. Otherwise, you’d have to reenter your username and password for every HTTP request sent. This also means that all the data required to process an HTTP request must be reloaded with every request a client sends to a server. To clarify this confusing concept, consider this example: if you and I had a stateless conversation, before every sentence spoken, I’d have to start with “I’m Peter Yaworski; we were
just discussing hacking.” You’d then have to reload all the information about what we were discussing about hacking. Think of what Adam Sandler does for Drew Barrymore every morning in 50 First Dates (if you haven’t seen the movie, you should).