pythonparallel-processingcondor

Is it possible to retrieve the output of python scripts in HTCondor?

I would like to run some python scripts that return some dictionaries on a cluster managed by HTCondor. I run the same on another cluster running MPI/Slurm and mpi4py and I was able to collect the output without saving files and transfer them to the main node. I cannot understand how to do it in HTCondor (even using python wrappers).

Is it possible to retrieve the output of python scripts in HTCondor? Do i need to use the schedd.retrieve("ClusterId == %d" % cluster) command? if so an example will be really appreciated.

Thanks!

Solution

  • You may specify a file to which you want to save your stdout in your submit file (see also Condor output file updating).

    You can watch the output using htcondor_tail (not sure what the equivalent is in the python api).

    Slightly off-topic: If you run python you can also use e.g. Dask to do your remote python stuff (see also Dask with HTCondor scheduler)