Introduction to XML and XPath

XML vs. HTML

Here are some differences between XML and HTML:

Navigating XML DOM

Navigating XML documents follows a process similar to using CSS selectors and Beautiful Soup.

The key difference lies in passing the appropriate XML parser to the Beautiful Soup object before initiating navigation.

Note: The HTML declaration <?xml version="1.0" encoding="UTF-8"?> in the above page.xml file is also called a prologue. It is optional but it must comes first in the document and it doesn't have a closing tag. The prologue defines the XML version and the encoding method.

Although CSS selectors are commonly associated with styling HTML documents, they can also be utilized with XML. However, it's important to note that the class attribute, frequently used in HTML for applying styles, may not be as prevalent in XML.

Introduction to XPath

XPath (XML Path Language) provides a powerful and flexible means of navigating and querying XML documents. With XPath, we can traverse the hierarchical structure of XML documents, accessing elements based on their element names, attributes, relationships, or positions within the document. XPath is not limited to XML alone; it can also be employed with Beautiful Soup and Selenium as an alternative to CSS Selectors, our primary focus in this learning module.

XPath syntax

XPath functions similarly to the expressions we use with computer file systems. By following a sequence of steps or a path, we can precisely select the desired node or nodes.

The main structure of XPath is as follows:

Let's explore each part in detail by applying using this example.

Axes (Selectors)

XPath employs path expressions for node selection within an XML document, achieved by traversing a sequence of steps or paths. The following is a compilation of some of the most valuable path expressions for this purpose:

Steps (Predicates)

Predicates are functions we use to find a tag with a specific value or order. We can divide them into four different types:

• Order predicates

• Attribute predicates

• Sibling predicates

• Functional predicates

Order predicates

Attribute predicates

Sibling predicates

Functional Predicates

There are about 37 functions we can utilize with XPath.

Advanced predicates

An important advantage of using XPath predicates is the ability to chain and combine multiple predicates to achieve the desired result. With XPath, a predicate can return another element, which allows us to apply additional predicates to narrow down our selection. This flexibility enables us to construct complex and precise queries to obtain our target data.

Suppose we want to select smartphones with a price greater than $800 and a rating equal to or higher than 4.5 from this HTML sample.

XPath solution:

• //div: This selects all <div> elements in the HTML document, regardless of their position or hierarchy.

• [span[@class='price' and number(substring-after(text(), '$')) > 800]: This is the first predicate that filters the selected <div> elements based on the price condition.

  • span[@class='price']: This selects the <span> element with the class attribute "price" that is a direct child of the current <div> element.

  • number(substring-after(text(), '$')) > 800: This condition checks if the numeric value of the price (obtained by removing the '$' symbol) is greater than 800. It does this by using the substring-after() function to extract the text after the '$' symbol and then converting it to a number using the number() function.

• and: This is the logical operator that combines the two predicates.

• span[@class='rating' and number(text()) >= 4.5]: This is the second predicate that filters the <div> elements further based on the rating condition.

  • span[@class='rating']: This selects the <span> element with the class attribute "rating" that is a direct child of the current <div> element.

  • number(text()) >= 4.5: This condition checks if the rating value, obtained as text, is equal to or higher than 4.5. It uses the number() function to convert the rating text to a number for comparison.

XPath vs. CSS selectors

CSS selectors primarily rely on the syntax and patterns defined by CSS specifications. They are commonly used to select elements based on their attributes, classes, IDs, or hierarchical relationships within the document's structure. On the other hand, XPath (XML Path Language) provides a more extensive and flexible querying capability. XPath is not limited to HTML documents but can also be used with XML and other structured data formats. With XPath, we can navigate the DOM tree using a combination of element names, attributes, relationships, and predicates. This allows for highly specific and granular element selection, even when dealing with complex document structures.

Here are some common differences between XPath and CSS selectors:

Note: Depending on the specific requirements of a web scraping task, the choice between XPath and CSS selectors will depend on the complexity of the target document and the precision needed to extract the desired data.