pythonweb-scrapingscrapy

How to scrape item with no reference or name attribute?


Hi I am really new at scrapy scrape I tried the basic code but this is one kinda unique and how I tried different approach here. How can I get the number of like, love and informative here https://teslamotorsclub.com/tmc/threads/tesla-tsla-the-investment-world-the-2019-investors-roundtable.139047/

here's my code

<ul class="dark_postrating_outputlist">
<li>
<i class="fa fa-info-circle"></i> Informative x <strong>1</strong>
</li>
<li>
<i class="fa fa-thumbs-o-up"></i> Like x <strong>1</strong>
</li>
</ul>

I wanted to get the specific item inside I have tried this

response.css('ul.dark_postrating_outputlist i.fa.fa-thumbs-o-up strong::text').extract_first()

But its not working, any idea please? thank you


Solution

  • Try the following to get the required content:

    import scrapy
    
    class TeslamotorsclubSpider(scrapy.Spider):
        name = "teslamotorsclub"
        start_urls = ["https://teslamotorsclub.com/tmc/threads/tesla-tsla-the-investment-world-the-2019-investors-roundtable.139047/"]
    
        def parse(self, response):
            for item in response.css("[id^='fc-post-']"):
                author = item.css(".author::text").get()
                like = item.css(".fa-thumbs-o-up + strong::text").get()
                love = item.css(".fa-heart-o + strong::text").get()
                informative = item.css(".fa-info-circle + strong::text").get()
                yield {"author":author,"like":like,"love":love,"informative":informative}
    

    Partial output:

    {'author': 'Unpilot', 'like': '1', 'love': '4', 'informative': '1'}
    {'author': 'UnknownSoldier', 'like': '7', 'love': '2', 'informative': '1'}
    {'author': 'SpaceCash', 'like': '2', 'love': '15', 'informative': '2'}
    {'author': 'gene', 'like': '45', 'love': '18', 'informative': '1'}
    {'author': 'engle', 'like': '31', 'love': '5', 'informative': '15'}
    {'author': 'Unpilot', 'like': '11', 'love': '3', 'informative': None}
    {'author': 'SebastianR', 'like': '3', 'love': None, 'informative': None}
    {'author': 'Buckminster', 'like': '1', 'love': '4', 'informative': None}