web-scrapingscrapyscrapy-shell

need help Web extracting <ul> and <li> using scrapy


Currently learning scrapy and with only a small knowledge of extracting a and tags but in need of help when it comes to extracting ul and li for example:

response.css('a.example')

response.css('div.example')

but what is the command for 'ul' and 'li"?


Solution

  • What you are missing here is an understanding of CSS selectors. For example,

    Once you learn this, you would know that selecting ul and li doesn't need a separate command. you just need to use the correct selector. Here is an example:

    <ul>
    <li> one</li>
    <li> two</li>
    </ul
    

    Now if you want to select the text inside the first li element, you would use

    response.css('li::text).get()
    

    If you want to extract both the element's text, you would use

    response.css('li::text).getall()
    

    I would suggest to go through sites like W3 Schools and understand CSS selectors