I want use requests-html and Beautifulsoup to get html, but the function of'response.html.render()'didn't work.
from requests_html import HTMLSession
session = HTMLSession()
response = session.get('https://www.xxxxxxx')
response.html.render()
There's a bug in the requests-html library that messes up the r.html.render() method in Python 3.8. It gets stuck and won't finish loading the webpage.
To fix it, either use an older version of requests-html (0.10.0 or earlier) or switch to the arender() method instead of render(). The arender() method is a bit trickier but will work in your case!
from requests_html import AsyncHTMLSession
async def main():
async with AsyncHTMLSession() as session:
r = await session.get('https://example.com')
await r.html.arender()
data = r.html.full_text
print(data)
if __name__ == '__main__':
asyncio.run(main())