Running a Multithreaded Jaguar Job with OpenMP

In this method, parts of the code are executed in parallel, using threads. See http://www.openmp.org for details. Use of threads takes only one license no matter how many threads are used, but it requires extra memory, and all threads spawned from a single process must run on the same node. OpenMP is useful in speeding up expensive calculations for single jobs. Use of OpenMP is supported on all platforms.

OpenMP is resource-intensive, and you can overload a computer by requesting too many OpenMP threads. A parallel job using 2 OpenMP threads will use almost twice as much memory as a single-processor job, at least for those portions of the job that can be run in parallel.

Also keep in mind that OpenMP is not 100% efficient, nor is the scaling linear. Thus a 4-processor job does not run 4 times faster than a 1-processor job, and a 32-processor job does not run twice as fast as a 16-processor job. The efficiency depends on the computer hardware, on the kind of calculation you are doing, on the level of theory, and on what portion of the program code has been enabled to use OpenMP.

For a simple job with a single input file and no subjobs, the command syntax to use OpenMP threads is:

jaguar run input-file -PARALLEL nproc

where nproc is the number of CPUs to use, which for a single-input job corresponds to the number of OpenMP threads. In practice you should not specify more threads than there are processor cores on the host, or you risk overloading the host.

For jobs with multiple input files or workflows, -PARALLEL nproc specifies the total number of CPUs to use simultaneously for the entire job. The Jaguar backend automatically allocates these CPUs among subjobs and OpenMP threads—see Automatic Selection of Distribution and Threading Options for Jaguar Jobs.

Because a team of OpenMP threads must run on the same node, you should be careful when using OpenMP on a cluster, so that you do not request more threads than there are cores on any one compute node. You can avoid this situation by specifying the maximum number of processors (cores) per node with the -procs_per_node option, or setting the maximum for each host in the hosts file—see The processors and processors_per_node Settings in the Hosts File for information.

When OpenMP is being used, the output file indicates the number of threads:

  Using up to   4 threads per process

If the job has subjobs (including the case of multiple input files), you can restrict the number of simultaneous subjobs by adding -HOST:njobs to the command, e.g.

jaguar run [script.py | script.bat] input-files -PARALLEL nproc -HOST hostname:njobs

In this case, only njobs subjobs are submitted for execution by the master job at any one time. The -PARALLEL option still specifies the total number of CPUs to use, and the backend distributes them among the njobs simultaneous subjobs. You can set the maximum number of OpenMP threads per subjob using the -max_threads option (default: 8). If you omit the -HOST option, the -PARALLEL option is interpreted differently—see Automatic Selection of Distribution and Threading Options for Jaguar Jobs.

One-node execution mode

When launching workflows on a queue system, the default behavior is to launch a 1-CPU driver to the host, which orchestrates the workflow and resubmits subjobs to the queue as they are generated. An alternative mode requests all CPUs upfront on one node at the start of the job. The subjobs then reuse this pool of CPUs for the duration of the workflow, without resubmitting anything to the queue. You can request this mode with the -use_one_node option:

jaguar run script.py input-files -PARALLEL N -HOST queue -use_one_node

In one-node mode, no CPU is consumed by a separate driver process, so all N CPUs are available for subjobs. If the workflow has multiple stages, the subjobs generated in subsequent stages also reuse the same pool of CPUs. However, requesting many CPUs upfront on one node may take longer to schedule on a busy queue.

Some scripts default to one-node execution, including csrch.py, gsm.py, and xtb.py. You can override this default and force the standard multi-node behavior with the -use_multiple_nodes option. The -use_one_node and -use_multiple_nodes options are mutually exclusive.

Controlling thread and processor limits

The -max_threads option limits the number of OpenMP threads used by any individual Jaguar Fortran backend subjob. The default is 8, which reflects the known parallel efficiency of the Jaguar code. The -procs_per_node option is more general: it limits the number of CPUs per node for all subjob types, including workflow subjobs launched in the one-node model. Its default value is taken from the schrodinger.hosts file; if not defined there, it falls back to 8. The -procs_per_node limit always takes precedence: the number of OpenMP threads per subjob never exceeds the processors-per-node value.