pythonweb-scrapingplaywrightplaywright-python

Playwright Python is not getting the response


When I wait for a response from an API call that is made on a page, PlayWright never gets the response. I've seen this in several attempts to wait for a response on different pages but here is some example code that narrows it down to an easy to replicate situation I hope.

The code times out

from playwright.sync_api import Page
from playwright.sync_api import sync_playwright

import json


with sync_playwright() as p:

    browser = p.chromium.launch(headless=False)
    page = browser.new_page()

    # Goto playwright homepage and press search box:
    page.goto("https://playwright.dev/")
    page.get_by_role('button', name='Search').click()

    # Catching response associated with filling searchfield:
    with page.expect_response("**.algolia.net/**") as response:

        # Fill a letter in searchbox to trigger the post-request:
        page.get_by_placeholder('Search docs').fill('A')

        # Printing the value of the response as a python json object:
        print(response.value.json())

        # Printing the value of the response as raw json:
        print(json.dumps(response.value.json()))

Here is the error message that happens when the response times out

Exception has occurred: InvalidStateError
invalid state
  File "C:\Users\ronwa\Documents\playwright-test\pwright-search.py", line 23, in <module>
    print(response.value.json())
          ^^^^^^^^^^^^^^
playwright._impl._errors.TimeoutError: Timeout 30000ms exceeded while waiting for event "response"
=========================== logs ===========================
waiting for response **.algolia.net/**
============================================================

During handling of the above exception, another exception occurred:

  File "C:\Users\ronwa\Documents\playwright-test\pwright-search.py", line 7, in <module>
    with sync_playwright() as p:
asyncio.exceptions.InvalidStateError: invalid state

Solution

  • I don't think you can elide the entire protocol prefix with **. Try adding slashes:

    from playwright.sync_api import sync_playwright
    
    
    with sync_playwright() as p:
        browser = p.chromium.launch()
        page = browser.new_page()
        page.goto("https://playwright.dev/")
        page.get_by_role("button", name="Search").click()
    
        with page.expect_response("*//*.algolia.net/**") as response:
            page.get_by_placeholder("Search docs").fill("A")
    
        print(response.value.json())