pythonajaxweb-scrapingscrapy

Scraping ajax pages using python


I've already seen this question about scraping ajax, but python isn't mentioned there. I considered using scrapy, i believe they have some docs on that subject, but as you can see the website is down. So i don't know what to do. I want to do the following:

I only have one url, example.com you go from page to page by clicking submit, the url doesn't change since they're using ajax to display the content. I want to scrape the content of each page, how to do it?

Lets say that i want to scrape only the numbers, is there anything other than scrapy that would do it? If not, would you give me a snippet on how to do it, just because their website is down so i can't reach the docs.


Solution

  • First of all, scrapy docs are available at https://scrapy.readthedocs.org/en/latest/.

    Speaking about handling ajax while web scraping. Basically, the idea is rather simple:

    Also see:

    Hope that helps.