MVAPICH for Infiniband

Compiler your program

MVAPICH for infiniband is installed at /usr/local/ibgd/mpi/gcc/mvapich-0.9.5-mlx1.0.4. Below are step-by-step guide to use MVAPICH for Infiniand

To compile your code, either type the full path to your desired MPI compiler for your language. Below are an example.

[user@tera mvapich]$ /usr/local/ibgd/mpi/gcc/mvapich-0.9.5-mlx1.0.4/bin/mpicc -o cpi cpi.c

The character in orange is source code of MPI application and the charactor in green is excutable program. If you want, you can download cpi.c from here.

Run your program

To run your compiled program, you need to submit your program to SGE (Sun Grid Engine) batch scheduling system. Below are step-by-step instruction

• Create job submission script for SGE scheduler with your desired editor, such as vi or nano

  [user@tera mvapich]$ vi mvapich_test.job

• Below are an example content of job script. You’ll need to replace MVAPICH_APP with the full path to your program. Note that, $NSLOTS and $TMPDIR will be replaced automatically by SGE.

  #!/bin/bash
  #$ -cwd
  #$ -S /bin/bash
  export MVAPICH_HOME=/usr/local/ibgd/mpi/gcc/mvapich-0.9.5-mlx1.0.4
  export MVAPICH_APP=/home/user/mvapich/cpi
  $MVAPICH_HOME/bin/mpirun -np $NSLOTS -machinefile $TMPDIR/machines \
      $MVAPICH_APP

• Now submit the script to SGE. Since this application is the parallel application, user need to indicate parallel environment (highlighted in green) and a number of processors (in orange) with option -pe. Note that, you can specify queue name other than default queue by using -q option. You can find more instruction on SGE here.

  [user@tera mvapich]$ qsub -pe mpich 4 -q highperf.q mvapich_test.job

• To view your job status

  [user@tera mvapich]$ qstat

• The output and error from MPI application are created in directory where user submitted jobs.