I have the following code which executes a process and calls a callback function when the process is done

import os
import subprocess
import tempfile


def callback(future):
    print(future.temp_file_name)
    try:
        with open(future.temp_file_name, 'r') as f:
            print(f"Output from {future.temp_file_name}:")
            print(f.read())
    except Exception as e:
        print(f"Error in callback: {e}")


def execute(args):
    from concurrent.futures import ProcessPoolExecutor as Pool

    args[0] = os.path.expanduser(args[0])
    with tempfile.NamedTemporaryFile(delete=False) as temp_file:
        temp_file_name = temp_file.name

        print('temp_file_name', temp_file_name)

        pool = Pool()
        with open(temp_file_name, 'w') as output_file:
            process = subprocess.Popen(args, stdout=output_file, stderr=output_file)
            future = pool.submit(process.wait)
            future.temp_file_name = temp_file_name
            future.add_done_callback(callback)

            print("Running task")


if __name__ == '__main__':

    execute(['~/testSpawn.sh', '1'])
    execute(['~/testSpawn.sh', '2'])
    execute(['~/testSpawn.sh', '3'])

But if I try to print out the result in the callback with

def callback(future):
    print(future.temp_file_name, future.result())

I get the following error

TypeError: cannot pickle '_thread.lock' object
exception calling callback for <Future at 0x77d48bc55a90 state=finished raised TypeError>

How can I get the result in a callback? The whole point is that I want to be notified when the process is complete

Update

I modified my callback function to have the following:

 if future.exception() is not None:
            print('exception', future.exception())
        else:
            print('result', future.result())

This just means it's not as ugly and it prints out exception cannot pickle '_thread.lock' object But it tells me that the error is not in printing out future.results(), but that there is just an inherent error in the callback function

subprocess.Popen objects can't be passed across process boundaries.

Use a ThreadPoolExecutor, not a ProcessPoolExecutor.