A beginner in JS/Html, I'm trying to fetch the content of a page after clicking on a link, through Pyppeteer0.2.5 (Python3.6.9/Chromium 87.0.4280.66) using the following code:
import asyncio, pyppeteer, time
from pyppeteer import launch
async def main():
browser = await launch(executablePath='/usr/bin/chromium-browser', headless=False)
page = await browser.newPage()
await page.goto('https://en.wikipedia.org')
time.sleep(10)
cont = await page.content()
print(cont)
asyncio.get_event_loop().run_until_complete(main())
time.sleep(10)
, the main Wikipedia page content is printed as expected, after 10 seconds.time.sleep(10)
, the new page opens as expected, but after the 10 seconds have elapsed I get a Python error (see below). I've had the same happen with different webpages and also when clicking through Pyppeteer rather than manually. Is there a reason why page.content()
fails after clicking on a link? Traceback (most recent call last):
File "/home/martin/.local/lib/python3.6/site-packages/pyppeteer/execution_context.py", line 105, in evaluateHandle
'userGesture': True,
pyppeteer.errors.NetworkError: Protocol error (Runtime.callFunctionOn): Cannot find context with specified id
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "To_delete/pythonProject/testtt.py", line 11, in <module>
asyncio.get_event_loop().run_until_complete(main())
File "/usr/lib/python3.6/asyncio/base_events.py", line 484, in run_until_complete
return future.result()
File "To_delete/pythonProject/testtt.py", line 9, in main
cont = await page.content()
File "/home/martin/.local/lib/python3.6/site-packages/pyppeteer/page.py", line 803, in content
return await frame.content()
File "/home/martin/.local/lib/python3.6/site-packages/pyppeteer/frame_manager.py", line 393, in content
'''.strip())
File "/home/martin/.local/lib/python3.6/site-packages/pyppeteer/frame_manager.py", line 309, in evaluate
pageFunction, *args, force_expr=force_expr)
File "/home/martin/.local/lib/python3.6/site-packages/pyppeteer/execution_context.py", line 54, in evaluate
pageFunction, *args, force_expr=force_expr)
File "/home/martin/.local/lib/python3.6/site-packages/pyppeteer/execution_context.py", line 108, in evaluateHandle
_rewriteError(e)
File "/home/martin/.local/lib/python3.6/site-packages/pyppeteer/execution_context.py", line 237, in _rewriteError
raise type(error)(msg)
pyppeteer.errors.NetworkError: Execution context was destroyed, most likely because of a navigation.
It turns out I needed to add await page.waitForNavigation()
just above await page.content()
to make it work.