So, basically I have a python program and I need to intercept 137 DDE & RTD signals that are feeding excel files. I'm doing this by using a socket with the following code:
import socket
import win32api
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('127.0.0.1', 12002))
s.sendall(str.encode('COT$S|' + stock + '#'))
data = s.recv(32768)
But, when I run this 137 times, it takes 27 seconds to finish, but I need something between 5s and 10s. So, I've tried to use multiprocessing, but, it starts taking infinity amounts of ram of my computer, until it crashes. Is there anything that I can do to optimize this code? Thanks!
My suggestion is to use multiprocessing.Pool
. Having had some time on hand, I did some mock test, and this works. Tune to fit your needs.
As it is, it will use a pool of 5 workers to cycle through each item in the stocks
list, so memory consumption should be in check, yet execution should be quite fast. You can adjust the pool size, of course.
import socket
import win32api
from multiprocessing import Pool
stocks = ['AA', 'BB', 'CC', 'DD', 'EE'] # just a sample list
def network_op(stock):
print('Running ' + stock)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('127.0.0.1', 12002))
s.sendall(str.encode('COT$S|' + stock + '#'))
d = s.recv(32768)
return d
if __name__ == '__main__':
with Pool(5) as p:
print(p.map(network_op, stocks))