Static vs. Dynamic Websites
Develop the ability to differentiate between static and dynamic websites.
We'll cover the following
Websites can be static or dynamic, and our scraping script must be adjusted accordingly to handle the dynamic content. Let’s explore the clear difference between static and dynamic websites.
Static websites
A static website comprises pages that remain the same for all users without needing interaction. Each page on a static website is stored as a single HTML file, so when we access the page, all the content will be displayed in the HTML response. Static websites are simple to scrape. We can access all of the information by just exploring the HTML document.
Books to Scrape is the best example of a static website where we will find all the information available in the developer tools without scrolling or clicking.
Dynamic websites
A dynamic website relies significantly on user interactions to provide all the information. Behind the scenes, when the user interacts with the web page, the browser executes a JavaScript code to fetch data from an API or an internal database. Without this data, the HTML document obtained through web scraping would only contain empty tags.
The website OpenTable is the perfect example of a dynamic website.
In the Inspector tool, the first row of restaurants is a <div> tag with other elements inside it. However, the following <div> tags are empty, and the data for the other restaurants will only appear when scrolling.
Modifications to our navigation script are necessary to scrape dynamic websites effectively, which we will discuss later.
Note: Some dynamic sites are particularly designed to resist scraping, thus more work may be necessary to retrieve the relevant information.
Get hands-on with 1300+ tech skills courses.