pythonfreezemechanicalsoup

Mechanicalsoup freezes with some websites

I use MechanicalSoup for a custom search engine. When I try to open some websites, MechanicalSoup freezes, up to few hours, and gives an exception only after this delay. Sometimes it works, but it is always very long with some websites ex:

import mechanicalsoup
browser = mechanicalsoup.StatefulBrowser()
browser.open('https://www.amd.com')
browser.open('https://www.airfrance.fr')

How can I use mechanicalsoup and not have this freeze, or how can I set a short time out (10s max for example), in order to avoid waiting hours.

Solution

  • The MechanicalSoup API is a thin wrapper around the Requests library, which accepts a timeout argument in the relevant function calls (see https://requests.readthedocs.io/en/master/user/quickstart/#timeouts).

    Therefore, you can set a 10 second timeout in MechanicalSoup by adding timeout=10, e.g.:

    import mechanicalsoup
browser = mechanicalsoup.StatefulBrowser()
browser.open('https://www.amd.com', timeout=10)
browser.open('https://www.airfrance.fr', timeout=10)