Automatic Selection of Distribution and Threading Options for Jaguar Jobs

The recommended way of running jobs over multiple processors is to use the -PARALLEL N option. This option specifies the total number of processors (CPUs) to use simultaneously for the entire job, including all subjobs. The Jaguar backend automatically decides how many subjobs to launch simultaneously and how many OpenMP threads to assign to each. Generally, distribution of subjobs over available processors is done first, as this is the most efficient. For example,

jaguar run filename -PARALLEL 6 -HOST queue

requests 6 processors, and a single-input-file job would use up to 6 OpenMP threads, whereas

jaguar run file1 file2 file3 file4 -PARALLEL 4 -HOST queue

would run one subjob with one thread for each of the four files, started simultaneously. One of the subjobs would run on the same processor as the driver (which isn't explicitly specified here, but is present whenever you provide multiple input files).

This mechanism can also be used with Python batch scripts, e.g.

jaguar run script.py filename -PARALLEL 12 -HOST queue

The subjobs are distributed as well, and the number of subjobs to run concurrently is determined along with the number of threads. Each subjob is run with -PARALLEL nsubproc where nsubproc is the number of processors specified for the job divided by the number of subjobs.

Default thread cap

Jaguar does not typically scale efficiently beyond about 8 OpenMP threads. Therefore, the algorithm for distributing CPUs via -PARALLEL does not assign more than 8 threads to individual Jaguar subjobs by default. You can change this cap with the -max_threads option.

SmartDistribution

When a workflow is launched via a 1-CPU driver process, the Jobcontrol feature SmartDistribution allows the idle driver CPU to be reused by a 1-CPU subjob. For example, if a workflow has 3 subjobs and each is assigned 1 CPU, all 3 subjobs can run simultaneously using only 3 CPUs total (instead of requiring a fourth CPU for the driver). SmartDistribution is used automatically when applicable.

Optimized CPU assignments

When running a batch of multi-threaded Jaguar jobs, CPU cores are assigned optimally by default based on jaguar_timer estimates. Longer-running jobs are assigned more CPU cores, while shorter-running jobs are assigned fewer. The subjob submission order is also optimized, with longer jobs submitted first. This optimization is also applied to Jaguar workflows: if a batch of multi-threaded jobs is launched during a workflow, the subjob order and CPU core assignments are also optimized, provided that jaguar_timer supports all jobs in the batch.

Avoiding too many processors per node

In some instances, a subjob can request more processors for threaded execution than there are on a node, and the subjob then fails. You can avoid this situation by specifying the maximum number of processors per node with the -procs_per_node option. The default value is taken from the schrodinger.hosts file; if undefined, it falls back to 8. You can also set the processors-per-node value for each host in the hosts file—see The processors and processors_per_node Settings in the Hosts File for information.

Recommended uses

To minimize runtime for a single job, use -PARALLEL N where N is large (but appropriate for the hardware). The backend automatically sets up the job for maximum acceleration, including OpenMP parallelism where possible. However, if the job is submitted to a busy queue system, requesting many CPUs per subjob may increase the time spent waiting for resources.

To minimize total runtime for a large suite of jobs, it is usually more efficient to use -HOST queue without -PARALLEL, so that the queue is filled with as many 1-CPU subjobs as possible. This avoids the queue-wait overhead and the sublinear scaling of OpenMP, and is likely to result in the shortest wallclock time for the full suite.