pythonpython-requestspython-requests-html

Why render / requests-html doesn't scrape dynamic content?


Long story short : switched from Selenium to Requests(-html).

Works OK but not in every case.

Page : https://www.winamax.fr/paris-sportifs/sports/1/1/1

Upon load it charges dynamic content with english games (example : Sheffield United - West Ham).

But when I try to do this :

from requests_html import HTMLSession
session = HTMLSession()
r = session.get('https://www.winamax.fr/paris-sportifs/1/1/1')
r.html.render()
print(r.html.text) # I also tried print(r.html.html)

the games don't show in the output.

Why ? Thanks !


Solution

  • add timeout, it should work, sorry this must be a comment but I cannot comment..

    from requests_html import HTMLSession
    session = HTMLSession()
    r = session.get('https://www.winamax.fr/paris-sportifs/sports/1/1/1')
    r.html.render(timeout=20)
    print(r.html.html)
    session.close()