allocating groups of nodes to each scheduler

By default all nodes are allocated to the multi-user system. The nodes are available for use once the multi-user daemons are started, at which point no nodes are useable under PBS or SGE.

To allocate some of the nodes to multi-user mode you need to edit the file /opt/score/etc/scorehosts.db.

In here you will find a full list of all compute nodes together with there resources - e.g ethernet, smp, myrinet networks. For example suppose we have the 4 compute nodes vm0,..,vm3, then the appropriate section in scorehosts.db might look like:

##
##
#include "/opt/score//etc/ndconf/0"
#include "/opt/score//etc/ndconf/1"
#include "/opt/score//etc/ndconf/2"
#include "/opt/score//etc/ndconf/3"
##
#define MSGBSERV        msgbserv=(vmserve.streamline.com:8764)

vm0.streamline.com HOST_0 network=ethernet group=_scoreall_,VM smp=1 MSGBSERV
vm1.streamline.com HOST_1 network=ethernet group=_scoreall_,VM smp=1 MSGBSERV
vm2.streamline.com HOST_2 network=ethernet group=_scoreall_,VM smp=1 MSGBSERV
vm3.streamline.com HOST_3 network=ethernet group=_scoreall_,VM smp=1 MSGBSERV

This file is a set of C preprocessor directives. The entry group= specifies which group each node belongs to. A node can belong to any number of groups, with group names being separated by commas. The group name can be any arbitrary acsii string. In the above example all the nodes belong to the _scoreall_ and VM groups. The best way to partition the cluster is to add an extra group name to those hosts that you want to run under multi-user mode. In the above example we could add the group name G0 to hosts vm0 and vm1:

##
##
#include "/opt/score//etc/ndconf/0"
#include "/opt/score//etc/ndconf/1"
#include "/opt/score//etc/ndconf/2"
#include "/opt/score//etc/ndconf/3"
##
#define MSGBSERV        msgbserv=(vmserve.streamline.com:8764)

vm0.streamline.com HOST_0 network=ethernet group=_scoreall_,VM,G0 smp=1 MSGBSERV
vm1.streamline.com HOST_1 network=ethernet group=_scoreall_,VM,G0 smp=1 MSGBSERV
vm2.streamline.com HOST_2 network=ethernet group=_scoreall_,VM smp=1 MSGBSERV
vm3.streamline.com HOST_3 network=ethernet group=_scoreall_,VM smp=1 MSGBSERV

Next you need to modify the group name in the sc_watch bootup script /etc/rc.d/init.d/sc_watch. In this file change the sc_watch -g option to the new group name. E.g in our example:

sc_watch  -g VM

becomes

sc_watch  -g G0

Next edit the PBS configuration file /var/scored/pbs/server_priv/nodes and remove the multi-user nodes. So in the example this would now look like:

#
# `nodes' file for pbs_server, generated by scbd2pbs.
#
vm2.streamline.com       VM score  chem
vm3.streamline.com       VM score  chem

i.e the group multi-user G0 nodes and the PBS nodes correspond to all your compute nodes with no overlap.
Similarly if you have SGE installed you must disable all the execution queus corresponding to SCore multi-user nodes.

Finally you must restart some daemons:

[root@vmserve mpi]# /etc/rc.d/init.d/score stop
[root@vmserve mpi]# /etc/rc.d/init.d/pbs_server restart
[root@vmserve mpi]# /etc/rc.d/init.d/score start

This will bring up both schedulers. By default the multi-user scored daemon and the sc_console daemon associated with the multi-user mode both run on the last host - in this example vm1, so when invoking sc_console you should use this one - e.g here it would be:

[root@vmserve mpi]# sc_console vm1
SCore-D Console:

See the online manual page for sc_console. If you change the groups like this then you will need to specify the scored host on the scrun line for multi-user mode. e.g here it would be:

[nrcb@vmserve mpi]$ scrun -nodes=2,scored=vm1 ./mpitest

You can set this up as a default option by changing the SCORE_OPTIONS environment variable.

Another way to achieve the same result without having to restart the PBS server is to stop the pbs_mom service on each node that is going to run in multi-user mode. The score service still needs to be stopped and started.