Python stem+requests: Not switching circut/changing IP address when using a session

Cnosider the following script tortest.py, which repeatedly switches the TOR circuit and checks if the IP address has changed:

#!/usr/bin/env python2.7
# -*- coding: utf-8 -*-

import json
import time

# pip install requests[socks]
import requests

# pip install stem
from stem import Signal
from stem.control import Controller

http = requests.session()
proxies = {'http': 'socks5://127.0.0.1:9050', 'https': 'socks5://127.0.0.1:9050'}


def get_new_ip():
    with Controller.from_port(port=9051) as controller:
        controller.authenticate(password="xxx")
        controller.signal(Signal.NEWNYM)
        time.sleep(controller.get_newnym_wait())


for _ in xrange(5):
    get_new_ip()
    try:
        ip1 = json.loads(requests.get("https://httpbin.org/ip", timeout=(5, 5), proxies=proxies).content)["origin"]
        ip2 = json.loads(http.get("https://httpbin.org/ip", timeout=(5, 5), proxies=proxies).content)["origin"]
        print "Without session: {}".format(ip1)
        print "With session:    {}".format(ip2)
        print
    except Exception as e:
        pass

Running this script, I get the following output:

$ python2 tortest.py
Without session: 137.74.169.241
With session:    137.74.169.241

Without session: 145.249.104.203
With session:    137.74.169.241

Without session: 5.189.170.221
With session:    137.74.169.241

Without session: 192.36.27.6
With session:    137.74.169.241

Without session: 199.249.224.43
With session:    137.74.169.241

As we can see, the IP address changes every time in the sessionless case. However, in the case where we use a session object, the IP address only changes for the very first request and all subsequent requests show the same IP address indicating that the circuit did not change.

Why does this happen and how can it be resolved? How can I use request sessions without losing the ability to switch circuits?

Solution

This is because of...

Keep-Alive

Excellent news — thanks to urllib3, keep-alive is 100% automatic within a session! Any requests that you make within a session will automatically reuse the appropriate connection!

Note that connections are only released back to the pool for reuse once all body data has been read; be sure to either set stream to False or read the content property of the Response object.

When you use a session, it sends a Connection: keep-alive header and will hang onto the connection which causes it to use the previously established circuit in Tor.

As mentioned in the docs, you can set stream=False to get around it:

http.get("https://httpbin.org/ip", timeout=(5, 5), proxies=proxies, stream=False)