04-06 add css selectors in parse()

1

Put the css selectors and values within the parse() method in BookspiderSpider

create the books var from response, then loop through, yield will create object, (whatever it is called in python) for each book

import scrapy


class BookspiderSpider(scrapy.Spider):
    name = "bookspider"
    allowed_domains = ["books.toscrape.com"]
    start_urls = ["https://books.toscrape.com"]

    def parse(self, response):
        books = response.css('article.product_pod')
        for book in books:
            yield{
                'name': book.css('h3 a::text').get(),
                'price': book.css('.product_price .price_color::text').get(),
                'url': book.css('h3 a').attrib['href']
            }

exit scrapy, run spide in terminal, make sure you are in the directory of bookscraper with bookspider in it.

cd bookscraper

then run,

scrapy crawl bookspider

This will display the result in the terminal

2

get the next page button url link from in shell, then add the ability to click it with the condition

scrapy shell
fetch('https://books.toscrape.com/')

he does it different here with ::attr(href)

response.css('li.next a::attr(href)').get()