Implicit vs. Explicit Wait
Learn about Selenium wait and the difference between explicit and implicit wait.
Having discussed the AJAX/JS loading concept and how Selenium can aid in retrieving the necessary data, it is crucial to remember that loading times can vary between the elements within the page. Additionally, other factors can influence the loading process, potentially causing Selenium to overlook data and return an empty Document Object Model (DOM). Selenium provides a solution for these issues.
Types of wait
In Selenium, the wait functionality refers to the ability to pause the execution of a script for a specific duration or until certain conditions are met. It assists in synchronizing the interactions between the scraping script and the target web application being scraped.
1. Implicit wait
This kind of wait directs the web driver to poll the DOM for a while when trying to find an element if it is not immediately found. For instance, a value of 5 indicates that the driver will poll the DOM for 5 seconds before commencing the search for elements.
Note: The implicit wait affects the entire lifespan of the WebDriver object, so it is recommended to set it once at the beginning of the script and adjust the duration as per your specific needs.
For instance, in our previous code for scraping movies, we defined an implicit wait after retrieving the element:
elem_2013 = driver.find_element(By.ID, "2013")elem_2013.click()driver.implicitly_wait(3)
However, we can change this and configure it at the beginning of the script, and the driver will wait for the duration each time we call find_element/s().
## defining a global wait for any element we look for.
driver.implicitly_wait(3)
driver.get("https://www.scrapethissite.com/pages/ajax-javascript/")
print("\nPage's title is: \n", driver.title)
elem_2013 = driver.find_element(By.ID, "2013")
elem_2013.click()
films = driver.find_elements(By.CSS_SELECTOR, "tr.film td.film-title")
print("\n2013 Movies: \n")
print([x.text for x in films])
driver.close()
2. Explicit wait
In contrast, explicit wait does not wait for a specific duration of time. Instead, it waits for a specific condition to be met before proceeding. This condition can be waiting for a particular element to load or waiting for a certain attribute to have a specific value. If the condition is not met within the specified time frame, the driver will throw a TimeoutException.
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
driver.get("https://www.scrapethissite.com/pages/ajax-javascript/")
print("\nPage's title is: \n", driver.title)
elem_2013 = driver.find_element(By.ID, "2013")
elem_2013.click()
try:
films = WebDriverWait(driver, 10).until(
EC.presence_of_all_elements_located((By.CSS_SELECTOR, "tr.film td.film-title"))
)
except TimeoutException as ex:
films = []
print("\n2013 Movies: \n")
print([x.text for x in films])
driver.close()
Lines 12–17: We define the waiting time condition inside try/except block. We first initiate an object from WebDriverWait with the driver instance and the waiting time we want. Then we use until() function and pass to it the condition, which, in this case, is waiting till all the table items are loaded.
The expected_conditions class provides other useful different conditions:
title_is:Check if an item with a specific title is loaded.title_contains:Check if an item with a title containing X is loaded.presence_of_element_located:Check the element’s presence.visibility_of_element_located:Check if the element is visible using thevisibleattribute.presence_of_all_elements_located:Check the presence of a group of elements.text_to_be_present_in_element:Check if a specific text appears in the element.element_to_be_clickable:Check if the element became clickable.element_to_be_selected:Check if the element can be selected.
How to choose between implicit or explicit wait
There are the following clear differences between implicit and explicit wait, which may help us to choose between them:
Implicit Wait | Explicit Wait |
It can be defined only a single time for all the test flow. | It can be defined per element. |
It applies to all elements on the page. | It applies only to specific elements. |
There is no need to explicitly specify | The use of |
It is most effective when elements will be located within the time frame specified in the implicit wait. | It is most effective when used when the elements on the page are taking a long time to load. |
It affects the execution speed since each action on elements will this amount of time if element not found yet. | It does not affect the execution speed since it is applicable to a particular element. |
The implementation is easy but can vary between different drivers which might be slower in one than another. | The implementation is complex and is the same for all the drivers but it is language-specific. |
Overall, explicit wait is preferred over implicit wait. It allows more control and better quality over the output by encapsulating each one in a try/except block to debug which element isn't loaded easily. One scenario where implicit wait can be useful is if we have multiple elements, so rather than coding all of these blocks, we can set a general time that is suitable for all of them.
3. Python time.sleep() wait
This is the least preferred option for waiting. When we use time.sleep(), the entire thread sleeps for a specific amount of time, even if all the elements are loaded. In contrast, implicit and explicit waits only stop blocking the thread when the element is loaded or the condition is met. However, in some cases, we may need to slow down thread execution. For example, when running multiple parallel Selenium scripts, the driver may crash because it can't keep up with the speed of thread execution.
Conclusion
At this point, we should be familiar with Selenium wait modules and how to use them efficiently. We now have the power to utilize Selenium to scrape any dynamic website.
Get hands-on with 1300+ tech skills courses.