pythoniosubprocessnonblocking

A non-blocking read on a subprocess.PIPE in Python


I'm using the subprocess module to start a subprocess and connect to its output stream (standard output). I want to be able to execute non-blocking reads on its standard output. Is there a way to make .readline non-blocking or to check if there is data on the stream before I invoke .readline? I'd like this to be portable or at least work under Windows and Linux.

Here is how I do it for now (it's blocking on the .readline if no data is available):

p = subprocess.Popen('myprogram.exe', stdout = subprocess.PIPE)
output_str = p.stdout.readline()

Solution

  • fcntl, select, asyncproc won't help in this case.

    A reliable way to read a stream without blocking regardless of operating system is to use Queue.get_nowait():

    import sys
    from subprocess import PIPE, Popen
    from threading  import Thread
    
    try:
        from queue import Queue, Empty
    except ImportError:
        from Queue import Queue, Empty  # python 2.x
    
    ON_POSIX = 'posix' in sys.builtin_module_names
    
    def enqueue_output(out, queue):
        for line in iter(out.readline, b''):
            queue.put(line)
        out.close()
    
    p = Popen(['myprogram.exe'], stdout=PIPE, bufsize=1, close_fds=ON_POSIX)
    q = Queue()
    t = Thread(target=enqueue_output, args=(p.stdout, q))
    t.daemon = True # thread dies with the program
    t.start()
    
    # ... do other things here
    
    # read line without blocking
    try:  line = q.get_nowait() # or q.get(timeout=.1)
    except Empty:
        print('no output yet')
    else: # got line
        # ... do something with line