pythonscrapy

Scrapy script not scraping items


I'm not sure why my script isn't scraping any items, it is the same script that I'm using for another website. Maybe the classes I'm using are wrong.

import scrapy
import os
from scrapy.crawler import CrawlerProcess
from datetime import datetime

date = datetime.now().strftime("%d_%m_%Y")

class stiendaSpider(scrapy.Spider):
    name = 'stienda'
    start_urls = ['https://stienda.uy/tv']

    def parse(self, response):
        for products in response.css('.grp778'):
            price = products.css('.precioSantander::text').get()
            name = products.css('#catalogoProductos .tit::text').get()
            if price and name:
                yield {'name': name.strip(),
                       'price': price.strip()}

os.chdir('C:\\Users\\cabre\\Desktop\\scraping\\stienda\\data\\raw')
process = CrawlerProcess(
#    settings={"FEEDS": {"items.csv": {"format": "csv"}}}
     settings={"FEEDS": {f"stienda_{date}.csv": {"format": "csv"}}}
)
process.crawl(stiendaSpider)
process.start()

Why is it not working?


Solution

  • I was able to get the name field, but the price attribute is rendered empty and filled in later from an ajax call. That is why it's not being extracted.

    import scrapy
    import os
    from scrapy.crawler import CrawlerProcess
    from datetime import datetime
    
    date = datetime.now().strftime("%d_%m_%Y")
    
    class stiendaSpider(scrapy.Spider):
        name = 'stienda'
        start_urls = ['https://stienda.uy/tv']
    
        def parse(self, response):
            for products in response.xpath('//div[@data-disp="1"]'):
                name = products.css('.tit::text').get()
                if name:
                    yield {'name': name.strip()}
    

    You can see it if you look at the page source... all of the elements with the class 'precioSantander' are empty.

    enter image description here