pythonpython-requests-html

r.html.render in requests-html can not work in python 3.8


I want use requests-html and Beautifulsoup to get html, but the function of'response.html.render()'didn't work.

from requests_html import HTMLSession
session = HTMLSession()
response = session.get('https://www.xxxxxxx')
response.html.render()

enter image description here


Solution

  • There's a bug in the requests-html library that messes up the r.html.render() method in Python 3.8. It gets stuck and won't finish loading the webpage.

    To fix it, either use an older version of requests-html (0.10.0 or earlier) or switch to the arender() method instead of render(). The arender() method is a bit trickier but will work in your case!

    from requests_html import AsyncHTMLSession
    
    async def main():
        async with AsyncHTMLSession() as session:
            r = await session.get('https://example.com')
            await r.html.arender()
            data = r.html.full_text
            print(data)
    
    if __name__ == '__main__':
        asyncio.run(main())