pythonpython-3.6chromiumpyppeteer

Pyppeteer - error when running page.content() after clicking on a link


A beginner in JS/Html, I'm trying to fetch the content of a page after clicking on a link, through Pyppeteer0.2.5 (Python3.6.9/Chromium 87.0.4280.66) using the following code:

import asyncio, pyppeteer, time
from pyppeteer import launch

async def main():
    browser = await launch(executablePath='/usr/bin/chromium-browser', headless=False)
    page = await browser.newPage()
    await page.goto('https://en.wikipedia.org')
    time.sleep(10)
    cont = await page.content()
    print(cont)
asyncio.get_event_loop().run_until_complete(main())
  1. If I don't click or do anything during time.sleep(10), the main Wikipedia page content is printed as expected, after 10 seconds.
  2. If I (manually) click on any link during time.sleep(10), the new page opens as expected, but after the 10 seconds have elapsed I get a Python error (see below). I've had the same happen with different webpages and also when clicking through Pyppeteer rather than manually. Is there a reason why page.content() fails after clicking on a link?
    Traceback (most recent call last):
      File "/home/martin/.local/lib/python3.6/site-packages/pyppeteer/execution_context.py", line 105, in evaluateHandle
        'userGesture': True,
    pyppeteer.errors.NetworkError: Protocol error (Runtime.callFunctionOn): Cannot find context with specified id
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "To_delete/pythonProject/testtt.py", line 11, in <module>
        asyncio.get_event_loop().run_until_complete(main())
      File "/usr/lib/python3.6/asyncio/base_events.py", line 484, in run_until_complete
        return future.result()
      File "To_delete/pythonProject/testtt.py", line 9, in main
        cont = await page.content()
      File "/home/martin/.local/lib/python3.6/site-packages/pyppeteer/page.py", line 803, in content
        return await frame.content()
      File "/home/martin/.local/lib/python3.6/site-packages/pyppeteer/frame_manager.py", line 393, in content
        '''.strip())
      File "/home/martin/.local/lib/python3.6/site-packages/pyppeteer/frame_manager.py", line 309, in evaluate
        pageFunction, *args, force_expr=force_expr)
      File "/home/martin/.local/lib/python3.6/site-packages/pyppeteer/execution_context.py", line 54, in evaluate
        pageFunction, *args, force_expr=force_expr)
      File "/home/martin/.local/lib/python3.6/site-packages/pyppeteer/execution_context.py", line 108, in evaluateHandle
        _rewriteError(e)
      File "/home/martin/.local/lib/python3.6/site-packages/pyppeteer/execution_context.py", line 237, in _rewriteError
        raise type(error)(msg)
    pyppeteer.errors.NetworkError: Execution context was destroyed, most likely because of a navigation.

Solution

  • It turns out I needed to add await page.waitForNavigation() just above await page.content() to make it work.