pythonmultiprocessingsubprocessfile-descriptor

Combine multiprocessing with subprocess stdin


I am trying to populate input for my fzf sript from multiple parallel processes.

fzf = subprocess.Popen(
    [
        "fzf",
    ],
    stdin=subprocess.PIPE,
    text=True,
)

so I want to be able write to directly to this PIPE from other proccesses.

As multiprocessing does not support pickling IO objects I used file descriptor id. Like this:

python_process = multiprocessing.Process(
    target=generate_input,
    args=(fzf.stdin.fileno(),),
)

Inside my process I did write like this:

os.dup(write_fd) # I thought it could help to resolve error with fd.
os.write(write_fd, "Any string".encode("utf-8"))

This was leading to the OSError: [Errno 9] Bad file descriptor. I dont understand why I am gettings this, because all file descriptors are inherited from parent process. To ensure this, I tried to call this code inside my child process.

_fd_types = (
    ("REG", stat.S_ISREG),
    ("FIFO", stat.S_ISFIFO),
    ("DIR", stat.S_ISDIR),
    ("CHR", stat.S_ISCHR),
    ("BLK", stat.S_ISBLK),
    ("LNK", stat.S_ISLNK),
    ("SOCK", stat.S_ISSOCK),
)

for fd in range(100):
    try:
        s = os.fstat(fd)
    except:
        continue
    for fd_type, func in _fd_types:
        if func(s.st_mode):
            break
    else:
        fd_type = str(s.st_mode)
    print(f"fd: {fd}, type: {fd_type}")

This snippet printed that my file descriptor was available via os.fstat function.

I googled a lot a have not found something similar. Or at least some projects that utilize python in this way. I understand that I can approach this in other ways.
For example:

    child = subprocess.Popen(
        [
            "my_python_program --child",
        ],
        stdout=subprocess.PIPE,
    )
    fzf = subprocess.Popen(
        [
            "fzf",
        ],
        stdin=child.stdout,
    )

MacM1, Python3.12

But I really want to learn how to it in a more "basic" way. I will be very grateful for any info related, thank you.


Solution

  • Still learning, but I found solution that satisfies me at least.

    1. Create fifo descriptor in parent
    if not os.path.exists(fifo_path):
            os.mkfifo(fifo_path)
    
    1. Launch process that will write to this fifo
        python_process = multiprocessing.Process(
            target=child_func,
        )
        python_process.start()
        
       
        write_fd = open(fifo_path, "w")
        write_fd.close()
    
    1. Open fifo and pass to subprocess
    
          fifo = os.open(fifo_path, os.O_RDONLY)
          fzf = subprocess.Popen(
               [
                   "fzf",
               ],
               stdin=fifo,
               text=True,
           )