Python: Non-modal window with subprocesses

I need to implement the following UI: - There is a window with a label "running experiment 1/X" and a button - When the window is loaded, some experiments are started. The experiments are run by os.system of subprocess.Popen, they are just pre-compiled C++ programs - The experiments should run strictly one after another and not simultaneously (hence I can't use subprocess.Popen) - The window should be active while the experiments are running and the user can press the button - When the button is pressed, the experiments stop (we can just wait until the current experiment ends and stop) and the window closes - When all experiments are over, the window should close itself

First I tried running experiments in threading.Thread, but it still blocked the window. So I switched to multiprocessing.Process:

class StoppableProcess(Process):
    def __init__(self, name, alg, proj, parent):
        Process.__init__(self)
        self.stop = False
        self.name = name
        self.alg = alg
        self.proj = proj
        self.parent = parent 

    def stop(self):
        self.stop = True

    def stopped(self):
        return self.stop

    def run(self):
        count = len([k for k in self.proj.values()])
        i = 1
        for p in self.proj.values():
            self.parent.label.setText("Running experiment " + str(i) + " / " + str(count))
            os.system("some command here")
            i += 1
            if self.stopped():
                break  
        self.parent.hide() 



class Runner(QDialog):
    def __init__(self, parent):
        QDialog.__init__(self, parent)
        self.layout = QVBoxLayout()
        self.label = QLabel("Running experiment 0 / 0")
        self.setWindowTitle("Running experiments")
        button = QPushButton("Break experiments")
        self.layout.addWidget(self.label)
        self.layout.addWidget(button)
        self.setLayout(self.layout)
        QObject.connect(button, SIGNAL("clicked()"), self.Break)

    def Run(self, name, alg, proj):
        self.thread = StoppableProcess(name, alg, proj, self)
        self.thread.start()
        self.show()
        self.thread.join()

    def Break(self):
        self.thread.stop()
        self.hide()

However, this doesn't work at all, apparently because the Runner object should be pickled to be passed to a subprocess, but pickling fails. I was thinking about avoiding passing the parent argument and using Qt signals instead, but maybe there's a better solution?

First of all, you can indeed use subprocess.Popen to start background processes and wait for their completion. See the documentation, specifically, the poll() method. Run the UI event loop until the process has exited.

Second, it is usually a good idea to avoid threads in Python. The multiprocessing module is mostly useful when you want to parallelize tasks written in Python. IMO, I think it is easier to use the subprocess module if you are just launching external child processes.

The following pseudocode illustrates the idea:

experiments = [...]
process = None

def start_next_experiment():
    if not experiments:
        print "Done!"
    else:
        experiment = experiments.pop()
        process = subprocess.Popen(experiment)

def on_start_clicked():
    start_next_experiment()

def on_stop_clicked():
     # Clear the queue
    experiments = []

    # optional: Kill the process
    if process:
        process.terminate()

def on_idle():
    if process:
        # use e.g. a PyQT timer to run this method periodically
        process.poll()
        if process.returncode is not None:
            process = None
            start_next_experiment()