Automatic Selection of Distribution and Threading Options for Jaguar Jobs

While it is possible to set options for the different forms of parallel execution, it is usually best to allow the Jaguar startup script to determine the mix of job distribution and threads from the command line by using the -PARALLEL nproc option. This option specifies the total number of processors to use for the entire job, including all subjobs, and is the recommended way of running jobs over multiple processors. The number of subjobs and threads is set automatically. Generally, distribution of subjobs over available processors is done first, as this is the most efficient. For example,

jaguar run filename -PARALLEL 6 -HOST queue

requests 6 processors, and the job, which has a single input file, would request 6 threads, whereas

jaguar run file1 file2 file3 file4 -PARALLEL 4 -HOST queue

would run one subjob with one thread for each of the four files, started simultaneously. One of the subjobs would run on the same processor as the driver (which isn't explicitly specified here, but is present whenever you provide multiple input files).

This mechanism can also be used with Python batch scripts, e.g.

jaguar run script.py filename -PARALLEL 12 -HOST queue

The subjobs are distributed as well, and the number of subjobs to run concurrently is determined along with the number of threads. Each subjob is run with -PARALLEL nsubproc where nsubproc is the number of processors specified for the job divided by the number of subjobs. In some instances, the subjob can request more processors for threaded execution than there are on a node, and the subjob then fails. You can avoid this situation by specifying the maximum number of processors per node with the -procs_per_node option. You can also set this option for each host in the hosts file—see The processors and processors_per_node Settings in the Hosts File for information.