pythonpython-3.xcurio

How do I manage a python subprocess with an infinite loop


I want to read input from a subprocess in python, interact with it, see the results of that interaction, and then kill the process. I have the following parent and child processes.

child.py

from random import randint

x = 1
while x < 9:
    x = randint(0, 10)
    print("weeeee got", x)


name = input("what's your name?\n")
print("hello", name)

x = 0
while True:
    x += 1
    print("bad", x)
parent.py

import curio
from curio import subprocess


async def main():
    p = subprocess.Popen(
        ["./out"],
        stdout=subprocess.PIPE,
        stdin=subprocess.PIPE,
    )

    async for line in p.stdout:
        line = line.decode("ascii")
        print(p.pid, "got:", line, end="")
        if "what" in line and "name" in line:
            out, _ = await p.communicate(input=b"yaboi\n")
            print(p.pid, "got:", out.decode("ascii"), end="")
            # stuff to properly kill the process ...
            return


if __name__ == "__main__:
    curio.run(main)

If I run this, the parent hangs on:

            out, _ = await p.communicate(input=b"yaboi\n")

Removing the following section from the child fixes the issue:

x = 1
while x < 9:
    x = randint(0, 10)
    print("weeeee got", x)

Why doesn't it work with the infinite loop? How do I fix this?

Edit 1: putting flush=True in the print statement does not fix the issue.

Edit 2: The processes are in the following states:

4219 13.2  1.5 493968 487936 pts/0   S+   14:08   0:01 python parent.py
4223  100  0.0  13764  9600 pts/0    R+   14:08   0:14 python child.py

It looks like the python code is hanging on the communicate call.


Solution

  • I suspect there's something weird going on with bidirectional process communication in python's curio. I haven't tested using vanilla subprocess, but managed to get something working in bash.

    #!/bin/bash
    
    set -e
    
    inPipe="/tmp/in-$RANDOM"
    outPipe="/tmp/out-$RANDOM"
    
    cleanup() {
      set +e
    
      rm -f "$inPipe"
      rm -f "$outPipe"
    }
    
    trap cleanup EXIT SIGINT SIGTERM
    mkfifo "$inPipe" "$outPipe"
    
    a() {
      X=6
      while [ $X -lt 9 ]; do
        X=$((RANDOM % 11))
        echo yaay $X
      done
    
      local name
      echo "what's your name"
      read name <$inPipe
    
      echo hello "$name"
    
      while true; do
        echo 1
      done
    } >>$outPipe
    
    a&
    
    echo bg proc started
    
    while read -r line; do
      echo got "$line"
      if (exec 1>/dev/null; rg what <<< "$line" && rg name <<< "$line"); then
        echo finally got "$line"
        echo "don juan" >> $inPipe
        echo waiting on response
        read _ name
        echo "got name \"$name\" back"
        echo "done"
    
        exit 0
      fi
    done <$outPipe
    

    The above communicates with a background process using two named pipes. This was surprisingly finicky to write, and deadlock occurred if care wasn't taken with shell redirects. based on this experience I strongly recommend individuals attempting to do IPC in scripts to use shell as opposed to python (as it's faster to prototype and debug).

    The bash hacker's wiki is a great resource for people wanting to learn bash.