Browser Developer Tools

Learn how to inspect the HTML doc and get familiar with browser developer tools.

We'll cover the following

In this lesson, we will explore the browser tools that enable us to navigate and comprehend the structure of an HTML document. The previous lesson introduced using an HTML document for this purpose. We will delve deeper into the tools available in a web browser to facilitate this process.

We will use Google Chrome for demonstration purposes, but similar functions are available in other browsers, such as Mozilla Firefox, Apple Safari, and Microsoft Edge.

Introduction to developer tools

The browser developer tools are built-in tools that provide insights into the HTML, CSS, and JavaScript code and allow us to debug our web code in real time.

There are several methods for accessing the browser developer tools. We can easily access the DOM or CSS of any webpage by right-clicking the desired element and selecting Inspect. Or, for even faster access, try using keyboard shortcuts like "Command+Option+C" (Mac), "Control+Shift+C/I" (Windows, Linux, Chrome OS), or "F12" (Internet Explorer, Edge).

Press + to interact
Inspecting the title in chrome
Inspecting the title in chrome

The inspector tool

With the inspector tool, we can easily examine the HTML structure of the entire page by selecting an element and clicking "Inspect." This will take us to the relevant HTML tag. If we move our mouse pointer over something written in the code on a web page, it will show us where it is on the actual page.

Press + to interact
Hover over an element
Hover over an element

We can even change any text by running JavaScript code, and the browser will display our changes.

Applications

  • Viewing and editing the DOM and CSS

  • Inspecting and modifying HTML pages

  • Analyzing animations

  • Locating unused CSS

Note: When we inspect a page's source, the displayed HTML may not be the original HTML code. It may have been altered after undergoing browser processing and executing JavaScript code. For instance, the addition of <tbody> elements to tables is a known behavior of Firefox.

Working with the original HTML document is crucial in web scraping, so it's important to seek out an unmodified HTML source.

The network tool

This tool provides us with all the browser requests. We can find the headers, cookies, and response objects, which we will discuss in more detail later.

Let's explore the network panel by using the website Quotes to Scrape, which allows us to scrape its content (quotes) as a use case.

Observe the network tab by doing the following:

  1. In Chrome, visit the website URL. 

  2. Then, right-click and select “Inspect” to open the developer tools.

  3. Select the "Network" tab and choose "Fetch/XHR."

  4. Refresh the page and then scroll down the page.

  5. Select any "quotes?page=<Page_num>" on the left panel "Name," and the headers will be displayed on the right panel "Header."

Now, we can see the underlying requests made by the browser to retrieve this dynamic content.

Press + to interact
Exploring the Network panel
Exploring the Network panel
Press + to interact
The URL obtained from the Headers tab
The URL obtained from the Headers tab

Applications

  • Analyzing individual resource properties such as HTTP headers, content, and size

  • Checking the list of network requests

  • Viewing a recording of network traffic

  • Creating performance analysis of network activity

  • Inspecting web sockets and server-sent events

  • Throttling the network speed for testing purposes

Conclusion

We have covered the two essential browser developer tools. Other tools, such as the "Console" and "Debugger," can be useful for debugging web pages, but they are unrelated to our purpose of web scraping. Therefore, we will concentrate on tools that make the scraping process easier.

Get hands-on with 1300+ tech skills courses.