pythonmultiprocessingpipedup2

python: multiprocessing.Pipe and redirecting stdout


I am using multiprocessing package to spawn a second process from which I would like to redirect stdout and stderr into the first process. I am using multiprocessing.Pipe object:

dup2(output_pipe.fileno(), 1)

Where output_pipe is an instance of multiprocessing.Pipe. However, when I try to read on the other end, it just hangs. I tried reading using Pipe.recv_bytes with a limit, but that raises an OSError. Is this possible at all or should I just switch to some lower level pipe functions?


Solution

  • After experimenting in Python 2.7 I got this working example. With os.dup2 pipe's file descriptor is copied to standard output file descriptor, and each print function ends up writing to a pipe.

    import os
    import multiprocessing
    
    
    def tester_method(w):
        os.dup2(w.fileno(), 1)
    
        for i in range(3):
            print 'This is a message!'
    
    
    if __name__ == '__main__':
        r, w = multiprocessing.Pipe()
    
        reader = os.fdopen(r.fileno(), 'r')
    
        process = multiprocessing.Process(None, tester_method, 'TESTER', (w,))
        process.start()
    
        for i in range(3):
            print 'From pipe: %s' % reader.readline()
    
        reader.close()
        process.join()
    

    Output:

    From pipe: This is a message!
    
    From pipe: This is a message!
    
    From pipe: This is a message!