Parallel jobs : Using automatic job submission with mpisub

An automated job submission script is available for casual users who don't want to be bothered writing a parallel mpi job script. This will accept a binary argument and write and submit the script for you. Additionally the standard output from from a job submitted this way contains additional information, for instance whether the job ran 1 or 2 processes per node, the start and stop time of the job, which nodes it ran on and the total elapsed time taken.

To use this you first need to compile your mpi binary in the usual way. Use mpisub to submit an mpi parallel job on the Myrinet nodes.

To invoke mpisub use the following:

  mpisub nx1 ./myexec.exe ["arguments"]
  mpisub nx2 ./myexec.exe ["arguments"]

This will submit an mpi parallel job using n compute nodes with either 1 or 2 cpus per node. For example:

  mpisub 1x2 /home/nrcb/benchmarks/DLPOLY.parallel/execute/DLPOLY.X
  mpisub 2x2 /home/nrcb/benchmarks/DLPOLY.parallel/execute/DLPOLY.X
  mpisub 4x1 /home/nrcb/benchmarks/DLPOLY.parallel/execute/DLPOLY.X

would submit a 2,4 and a 4 process DLPOLY.X job running. The 2x2 job uses 2 compute nodes (2 processes per node) and the 4x1 job uses 4 compute nodes (4 processes per node). The job script that mpisub writes is placed in the current directory and named the same as the executable appended with .sh . If you don't specify a full path to the executable, mpisub assumes that it is in the current directory.

NOTE: Any arguments to the executable must be placed inside double quotes. e.g.

  mpisub 2x1 ./PMB-MPI1 "-npmin 2 Pingpong Sendrecv"

You can set the QSUB_OPTIONS variable - this is a list of options to the qsub command. See man qsub for details. If you invoke mpisub without arguments it will print out a usage guide. Additionally you may set the SCORE_OPTIONS variable with a list of scrun options. See the html man pages for scrun in file:/opt/score/doc/html/en/man/index.html on magnum or at the pccluster.org web site for a list of scrun options. For instance you may want to turn on mpi_zerocopy, which you can do using:

  setenv SCORE_OPTIONS mpi_zerocopy=on

Then invoke mpisub.