pythonperformancehttphttpspython-requests

Python - Entire PC network gets slower during repeated request


First of all, I am new to the network (HTTP communication) and Python.

I am currently using requests and threading module to periodically send or receive data with a specific site. The target site is 'https://api.coinone.co.kr' but I think it does not matter here.

By the example code below, I let Python fetch data every 1 second. At first, it works pretty well. Each request takes about 0.07 s in my computer.

import requests
import time
import threading

url0 = 'https://api.coinone.co.kr/ticker/'


class Fetch:
    def __init__(self):
        self.thread = threading.Thread(target=self.fcn)
        self.t0 = time.perf_counter()
        self.period = 1
        self.req0 = None

    def fcn(self):
        while True:
            # headers = None
            headers = {'Connection': 'close'}

            # requests
            t0 = time.perf_counter()
            req0 = requests.get(url0, headers=headers, params={'currency': 'all'})
            resp0 = req0.json()
            self.req0 = req0
            reqTimeInt0 = time.perf_counter() - t0

            # prints
            print('elapsed time: %0.3f' % t0)
            # print(req0.request.headers)
            print(resp0['result'])
            print('requests time interval: %0.3f' % reqTimeInt0)
            print('')

            # timer
            self.t0 += self.period
            now = time.perf_counter()
            sleepInterval = self.t0 - now
            if sleepInterval > 0:
                time.sleep(sleepInterval)
            else:
                while self.t0 - now <= 0:
                    self.t0 += self.period


f1 = Fetch()
f1.thread.start()

But as time passes, the time needed for each 'http get' increases. After about 3 hours, one request takes 0.80 s where it is 10 times larger than it took in the initial state. Furthermore, not only does Python request get slower, but also the entire PC network gets slower (including internet browsing) without any increase in CPU, RAM resources, and network usage. Closing the console does not get back the network speed to normal and I have to reboot the PC. Anyway, after rebooting, the network is completely recovered and the internet works fine.

It seems like some burdens in the network connection are accumulated at each Python request. So I tried adding 'Connection: close' to the header, but it didn't work. Will 'requests.Session()' fix the problem?

I don't even know what to do to figure out the problem. I have to make the repeated requests for at least several days without breaking the connection.

Thank you.


Solution

  • Use a session, as it won't open new network connections, just use one, to make all the requests.

    There is the preferred modifications:

    class Fetch:
        def __init__(self):
            self.session = requests.Session
            self.thread = threading.Thread(target=self.fcn)
            self.t0 = time.perf_counter()
            self.period = 1
            self.req0 = None
    
        def fcn(self):
            while True:
                # headers = None
                headers = {'Connection': 'close'}
    
                # requests
                t0 = time.perf_counter()
                req0 = self.session.get(url0, headers=headers, params={'currency': 'all'})
                resp0 = req0.json()
                self.req0 = req0
    
                ... other codes goes there ...