pythonsubprocessstdoutstderr

Python read from subprocess stdout and stderr separately while preserving order


I have a python subprocess that I'm trying to read output and error streams from. Currently I have it working, but I'm only able to read from stderr after I've finished reading from stdout. Here's what it looks like:

process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout_iterator = iter(process.stdout.readline, b"")
stderr_iterator = iter(process.stderr.readline, b"")

for line in stdout_iterator:
    # Do stuff with line
    print line

for line in stderr_iterator:
    # Do stuff with line
    print line

As you can see, the stderr for loop can't start until the stdout loop completes. How can I modify this to be able to read from both in the correct order the lines come in?

To clarify: I still need to be able to tell whether a line came from stdout or stderr because they will be treated differently in my code.


Solution

  • Here's a solution based on selectors, but one that preserves order, and streams variable-length characters (even single chars).

    The trick is to use read1(), instead of read().

    import selectors
    import subprocess
    import sys
    
    p = subprocess.Popen(
        ["python", "random_out.py"], stdout=subprocess.PIPE, stderr=subprocess.PIPE
    )
    
    sel = selectors.DefaultSelector()
    sel.register(p.stdout, selectors.EVENT_READ)
    sel.register(p.stderr, selectors.EVENT_READ)
    
    while True:
        for key, _ in sel.select():
            data = key.fileobj.read1().decode()
            if not data:
                exit()
            if key.fileobj is p.stdout:
                print(data, end="")
            else:
                print(data, end="", file=sys.stderr)
    

    If you want a test program, use this.

    import sys
    from time import sleep
    
    
    for i in range(10):
        print(f" x{i} ", file=sys.stderr, end="")
        sleep(0.1)
        print(f" y{i} ", end="")
        sleep(0.1)