submitting SCore MPI jobs on multi-processor nodes

In the previous section it was assumed that each compute node had a single processor. If your compute nodes have 2 or more processors then it is much more efficient in the use of resources to use all the processors on each node used. For example an 8 cpu SCore MPI job requires only 4 nodes if your nodes are dual processor, and only 2 nodes if your nodes are quad processor. By default the scrun SCore command will use the maximum number of cpu's per node. However the qsub SGE command only deals with compute nodes. In this case you have to introduce an appropriate scaling between scrun nodes (= cpus) and qsub nodes (= compute nodes).

For example suppose we want to submit a 16 cpu SCore MPI job running the mpitest code on dual processor compute nodes. Our job script called score.sh may look something like:

#!/bin/bash
#$ -masterq ehtpx-cluster.q -cwd -V
scout -wait -F $HOME/.score/ndfile.$JOB_ID -e /tmp/scrun.$JOB_ID \
 -nodes=$((NSLOTS-1))x2 /users/nrcb/mpi/mpitest

By default, scrun will use only 8 compute nodes since it knows that each machine is dual processor. Therefore we submit this job like thus:

[nick@ehtpx-cluster]$ qsub -pe score 9 score.sh

This submits a job requesting 8 compute nodes, plus an extra slot for the spawning process. If you are not familiar with shell scripting, your administrator can probably write a script which will do the correct thing on your system.

You can specify most options to scrun except checkpoint in the job script after the -nodes option - see the SCore multi-user section. For example to submit a parallel SGE job with zerocopy enabled your job script would look like this:

#!/bin/bash
#$ -masterq ehtpx-cluster.q -cwd -V
scout -wait -F $HOME/.score/ndfile.$JOB_ID -e /tmp/scrun.$JOB_ID \
 -nodes=$((NSLOTS-1))x2,mpi_zerocopy=on /users/nrcb/mpi/mpitest

See the SCore man page online document in file:/opt/score/doc/html/en/man/index.html for SCore commands and options.