Displaying MPI Debug Information

The I_MPI_DEBUG environment variable provides a convenient way to get detailed information about an MPI application at runtime. You can set the variable value from 0 (the default value) to 1000. The higher the value, the more debug information you get. For example:

$ mpirun  -genv I_MPI_DEBUG=2 -n 2 ./testc 
-genv I_MPI_DEBUG=2 -n 2 testc[1] MPI startup(): Internal info: pinning initialization was done[0] MPI startup(): Internal info: pinning initialization was done...

By default, each printed line contains the MPI rank number and the message. You can also print additional information in front of each message, like process ID, time, host name and other information, or exclude some information printed by default. You can do this in two ways:

• Add the '+' sign in front of the debug level number. In this case, each line is prefixed by the string <rank>#<pid>@<hostname>. For example:

  $ mpirun -genv I_MPI_DEBUG=+2 -n 2 ./testc
  [0#3520@clusternode1] MPI startup(): Multi-threaded optimized library
  ...

  To exclude any information printed in front of the message, add the '-' sign in a similar manner.

• Add the appropriate flag after the debug level number to include or exclude some information. For example, to include time but exclude the rank number:

  $ mpirun -genv I_MPI_DEBUG=2,time,norank -n 2 ./testc
  11:59:59 MPI startup(): Multi-threaded optimized library
  ...

  For the list of all available flags, see the description of I_MPI_DEBUG in the Developer Reference.

To redirect the debug information output from stdout to stderr or a text file, use the I_MPI_DEBUG_OUTPUT environment variable:

$ mpirun -genv I_MPI_DEBUG=2 -genv I_MPI_DEBUG_OUTPUT=/tmp/debug_output.txt -n 2 ./testc

Note that the output file name should not be longer than 256 symbols.