part 4 - multiple pages / scrolling

I ran into errors,

Look at part 5 to see

1

This is how you scrape from multiple pages, and infinite scrolling pages,

Change url to recipe website and searching for chicken recipes


https://www.allrecipes.com/search?q=chicken

the for loop with range(2) will only get the first two pages in this example

the driver.execute_script("arguments[0].scrollIntoView({block: 'center'})", next_btn) and the one below it with

click()

allow the site to wait for elements to load, keep in mind, sites changes, this didnt' work, but part 5 shows a working script from chat gpt, with try and except

import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.options import Options

driver = webdriver.Chrome()
driver.get('https://www.allrecipes.com/search?q=chicken')

wait = WebDriverWait(driver,10)




accept_btn = wait.until(
    EC.element_to_be_clickable((By.ID, "onetrust-accept-btn-handler"))
    )
    accept_btn.click()

time.sleep(1)

for i in range(2):
    next_btn = wait.until(
    EC.presence_of_element_located((By.CSS_SELECTOR,"li.mntl-pagination__next a"))
    )
    driver.execute_script("arguments[0].scrollIntoView({block: 'center'})", next_btn)
    driver.execute_script("arguments[0].click()",next_btn)
time.sleep(2)
driver.quit()