Introduction to Scrapy

Scrapy vs. Beautiful Soup

Here are some common differences between Scrapy and Beautiful Soup:

Advantages vs. disadvantages

Scrapy brings many benefits, surpassing straightforward tools like Beautiful Soup or Selenium. This becomes especially evident when tackling intricate websites, juggling multiple requests, and managing data pipelines. While Scrapy offers numerous advantages for web scraping, it's also essential to be aware of some potential disadvantages.

Here are some advantages and disadvantages of using Scrapy:

Installation

We can install the Scrapy library in any Python environment by running the command pip install Scrapy.

Usage

Scrapy's structure differs slightly from the other libraries we've explored so far. When creating a project with Scrapy, we initiate the process by executing the command scrapy startproject project_name. For instance, if we opt for the project name scraper and execute this command, it will generate the following project structure.

Run the command scrapy startproject scraper in the terminal below, and navigate to the scraper directory using the ls scraper to see the same files generated.

• scrapy.cfg: This file serves as the configuration hub for our Scrapy project. It contains settings about the project structure, including global configurations such as the location of settings.py and the project default name.

• scraper/: The main project directory.

  • items.py: Defines the data structure (items) we want to scrape and extract from websites. These items could represent information such as titles, descriptions, prices, and more.

  • middlewares.py: Implements custom middleware components to attach to each request and response during the scraping process.

  • pipelines.py: Set up data processing pipelines to handle scraped items (e.g., clean, validate, and store the scraped data, such as saving it to a database or exporting it to a file).

  • settings.py: The central hub for configuring various settings for our project. It allows us to customize Scrapy’s behavior, set download delays, manage user agents, configure pipelines, etc.

  • spiders/: This directory is where we will create and store our scraping scripts, which define how to scrape specific websites. We will have a script for multiple websites for each one, and all will have the same Scrapy configuration and settings.

Example

Let's dive deeper into the concepts with a practical example. We will scrape the Books to Scrape site using Scrapy. To accomplish this, we'll create a file named books.py and place it in the designated spiders directory within our project.

The structure of the scraping code remains consistent with what we've previously explored using XPath. However, the key distinction lies in how the script is organized and presented.

• Line 3: In Scrapy, every spider is defined as a Python class. This class is required to inherit from the scrapy.Spider class, a crucial step that enables Scrapy to recognize and interact with our spider effectively.

• Line 4: We assign our spider a unique identity through the name identifier. Each spider must possess a distinct name. Scrapy recognizes which spider to run when we give this name in the running command.

• Line 6–8: The start_requests() is one of Scrapy functions that we should implement in our spider.

  • It orchestrates the initiation of our crawling process by returning an iterable of scrapy.Request objects. These objects essentially encapsulate the URLs from which our spider will commence its crawl.

  • Scrapy is asynchronous by default; as a result, every function call within Scrapy must be designed to be iterable. This enables Scrapy to seamlessly control the progression from one URL to the next without blocking or waiting issues.

  • scrapy.Request() conducts an HTTP request to the specified URL. The result is then channeled to the callback function, as determined by the callback attribute.

• Line 10: Here we define the callback function that we called in start_requests, parse().

  • Scrapy will call this function by default if we didn't specify any callback function in the Request() function.

  • The response object that this function receives is simply the HTTP response we used to get with requests library, and then we can search within this response normally using XPaths or CSS Selectors as before.

• Line 26: We will yield from this function as Scrapy deals with it as a generator function. Each result will be yielded to the terminal since we didn't define any output files.

• Expected output: We'll notice abundant text displayed in the terminal alongside the extracted content when executing the code. Scrapy has a built-in logging module set to DEBUG mode as a default configuration. This is the reason why the extensive debug information is being showcased. However, this data deluge is surprisingly beneficial as it allows us to track and comprehend the behavior of the scraping process effectively.

Our focus centers on specific segments of this log, exemplified below:

This log snippet provides valuable insights into Scrapy's current settings, including the default middleware and item pipelines. Since we haven't employed any item pipelines, they remain empty.

Subsequently, we'll witness the extracted items being presented in dictionary format within the terminal. These items encapsulate all the fields we gathered during the scraping process. Finally, as the culmination of the scraping process, Scrapy will display the ultimate statistics, encompassing various key metrics of the operation:

Scrapy offers a comprehensive overview of vital statistics concerning the scraping endeavor in this phase. These statistics encompass a multitude of factors, including:

• The volume of data requested and received in bytes.

• The number of requests initiated.

• Breakdown of request methods, like GET.

• The total bytes of responses received.

• Response count with specific HTTP statuses, such as 200.

• The elapsed time of the entire process is in seconds.

• The reason for finishing the operation (usually "finished").

• The precise date and time when the operation concluded.

• The number of items successfully scraped.

• Counts of various log types, such as DEBUG, INFO, and WARNING.

• Memory usage details at startup and maximum memory usage during operation.

• The count of received responses.

• The count of items dequeued and enqueued in the scheduler.