pythonsocketsmultiprocessingpython-2.xmultiprocessing-manager

Python 2.7 multiprocessing - cant pass socket object via Manager dictionary


I have a simple program that searches for open ports in a local network and stores the connected sockets in a dictionary along with their local address. Now, I am using a Manager shared dictionary to store these entries but it only accepts simple objects and not socket instances. Here is the code:

from multiprocessing import Process, Manager
import socket

manager = Manager()
# Store connected sockets
sockets = manager.dict()


def ping_addr(addr=None, port=None, timeout=None):
    """
    Create a socket and try to establish a connection to a specific address. If a connection is established, append
    the socket to the sockets dictionary.
    :param addr: The address.
    :param port: The port number.
    :param timeout: How many seconds to wait until its decided that the connection has been refused.
    :return: None
    """
    global sockets
    # Setup the client socket
    csocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    csocket.settimeout(timeout)
    # Try to connect
    try:
        csocket.connect((addr, port))
        print 'connected to {}:{}'.format(addr, port)
        # This works
        sockets.update({addr: 0})
        # This doesnt work
        sockets.update({addr: csocket})
    except socket.error:
        pass


for i in range(256):
    proc = Process(target=ping_addr, kwargs={'addr': '192.168.1.{}'.format(i), 'port': 14540, 'timeout': 0.5})
    proc.start()

The error I get is:

Process Process-4:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/alex/PycharmProjects/proj/test.py", line 29, in ping_addr
    sockets.update({addr: csocket})
  File "<string>", line 2, in update
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 758, in _callmethod
    conn.send((self._id, methodname, args, kwds))
TypeError: expected string or Unicode object, NoneType found

I did a short research about proxies but couldn't find a way to implement them into my code, so I am asking for help. How do I make socket instances compatible with a Manager dictionary?


Solution

  • passing socket from parent to child process is a very common practice. starting with python3.4, you could share a socket object directly via multiprocessing.Manager, but in python2.7, you have to wire it by yourself(see full gist here):

    import socket
    import copy_reg
    from multiprocessing.reduction import rebuild_socket, reduce_socket
    
    copy_reg.pickle(socket.socket, reduce_socket, rebuild_socket)