I have submitted a Slurm job to a server, and while I can view the outputs on the terminal, there is no output being generated in the slurm.out
file. Surprisingly, there are no errors reported, and the job appears to be running smoothly. However, my objective is to be able to monitor the job's output in the slurm.out
file.
Is there a recommended approach or solution to ensure that the output is properly captured and recorded in the slurm.out
file for monitoring purposes?
Tested with a hello world
I had the same issue with python scripts.
Some tips
#!/bin/bash
#SBATCH --output=python.%j.out
#SBATCH --error=python.%j.err
#SBATCH -N 1
#SBATCH --ntasks-per-node=1
export PYTHONUNBUFFERED=TRUE
srun python yourscript.py
Print in python
print ("Hello World ", flush=True
or
print ("Hello World ",file=sys.stdout)
Do not forget in python script import sys