parse()
Put the css selectors and values within the parse() method in
BookspiderSpider
create the books var from response, then loop through, yield will create object, (whatever it is called in python) for each book
import scrapy
class BookspiderSpider(scrapy.Spider):
name = "bookspider"
allowed_domains = ["books.toscrape.com"]
start_urls = ["https://books.toscrape.com"]
def parse(self, response):
books = response.css('article.product_pod')
for book in books:
yield{
'name': book.css('h3 a::text').get(),
'price': book.css('.product_price .price_color::text').get(),
'url': book.css('h3 a').attrib['href']
}
exit scrapy, run spide in terminal, make sure you are in the directory of bookscraper
with bookspider in it.
cd bookscraper
then run,
scrapy crawl bookspider
This will display the result in the terminal
get the next page button url link from in shell, then add the ability to click it with the condition
scrapy shell
fetch('https://books.toscrape.com/')
he does it different here with ::attr(href)
response.css('li.next a::attr(href)').get()