pythonweb-scrapingwebpostpython-requests

Changing data(payload) but keeping the same session in python requests post


I am trying to make ONE session and keeps it alive for the whole iteration of payload.

The problem is if I make a session then iterate the payload, the session would also be iterated so I wouldn't get any benefit of using a session.

This is my code:

while i > 1 and i < 1570:
    i += 1
    payload = {
        'number1': i,
    }
    
    r = requests.post(url, data=payload)
    soup = BeautifulSoup(r.content, "lxml")

Solution

  • Getting your session and sending your request are two separated steps, so you can totally have one session for your script and send several requests with it:

    with requests.Session() as s:
        while i > 1 and i < 1570:
            i += 1
            payload = {
                'number1': i,
            }
    
            r = s.post(url, data=payload)
            soup = BeautifulSoup(r.content, "lxml")