[SOLVED] Using printf with MPI leads to non-deterministic output

Using printf with MPI leads to non-deterministic output

The following code has non-deterministic behaviour on my machine (already with using only two processes).

#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char *argv[]) {
  int rank, size;
  MPI_Init(&argc, &argv);
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
  MPI_Comm_size(MPI_COMM_WORLD, &size);

  if (rank == size - 1) {
    printf("I am the last rank\n");
    MPI_Barrier(MPI_COMM_WORLD);
  } else {
    MPI_Barrier(MPI_COMM_WORLD);
    printf("I am rank %d\n", rank);
  }
  MPI_Finalize();
  return 0;
}

Sometimes the output from the last rank appears first on the terminal but sometimes it does appear later, even though a barrier is used.

I assume the reason for this is that printf does internal buffering and that MPI respectively mpirun/mpiexec and printf do not really cooperate with each other. Is there a more valid source to read up on this topic?

Solution

Output redirection is strongly platform-dependent and is only tangentially mentioned in the MPI standard. There is absolutely no guarantee as to what order output from different ranks will be displayed in. There isn't even a guarantee that you can get that output - some implementations only redirect the output from rank 0, some redirect no output, and both cases are compliant with the MPI standard.

If you want strongly ordered output, you should only do output in a single rank.