Configure License Checking

Schrödinger jobs require licenses to run, usually per CPU or GPU used. Schrödinger job throughput on clusters therefore might be limited by license availability rather than compute resource availability. Queuing systems manage the usage of limited compute resources, but by default they do not account for limited license availability.

Configuring License Checking allows queueing systems to track Schrödinger licenses as consumable resources, reducing job failures due to unavailable licenses. License Checking also adds a runtime license check when a batch job starts up on a compute node; if the required licenses are not available, the job is automatically requeued. This mechanism prevents job failures even if the queuing system attempts to start a job but there aren’t sufficient licenses available.

All queueing systems require the following steps to configure license checking:

  • Schrödinger license resources must be defined in the queuing system configuration

  • A load sensor must be configured so the queuing system can account for external use of the licenses

  • Job Server must be configured so jobs it submits request the appropriate license resources from the queuing system

Follow the instructions below for your queueing system.

NOTE: Administrator privileges are required for configuring queuing systems, Job Server, and system services.
  • Slurm
  • Grid Engine
  • PBS Pro
  • LSF

Slurm

The Slurm queueing system offers two different methods of license checking, "remote" (recommended) and “local” (See Slurm Licenses Guide for more information).

  • Slurm Remote
  • Slurm Local

Remote license checking can handle multiple clusters and license checkouts from systems outside a Slurm cluster, (e.g., laptops and workstations).

If a Slurm Accounting Database is already configured, skip to Register the Compute Clusters with the Slurm Accounting Database.

Requirements

See Supported Queuing Systems for specific versions of the queueing systems that are supported.

 

The load sensor must be run on a machine that has access to:

  1. the Schrodinger lictool executable

  2. the license server(s)

  3. the Slurm ‘scontrol’ and ‘sacctmgr’ executables(whose location must be in PATH)

NOTE: Jobs launched directly from a cluster node (e.g., via an interactive session or direct login) will consume licenses without the queuing system being notified of that resource usage. We therefore recommend running all Schrödinger jobs as Slurm batch jobs via Job Server so they will request the appropriate license resources from Slurm.

 

The configuration process consists of the following steps:

Install, Configure, and Register the Slurm Accounting Database Required only if not already done. If a Slurm Accounting Database is already configured, skip to Configure Slurm for License Checking.

  1. Install the MariaDB SQL server
  2. Start the MariaDB SQL server
  3. Create the Slurm Accounting Database
  4. Configure the Slurm Control Daemon
  5. Configure the Slurm Database Daemon
  6. Register the Compute Clusters with the Slurm Accounting Database

 

Set up Slurm for License Checking

  1. Configure Slurm for License Checking
  2. Customize the Load Sensor Script
  3. Create Load Center System Service
  4. Turn On License Checking in Job Server
  5. Test the Configuration

 

 

Install, Configure, and Register the Slurm Accounting Database

NOTE: If a Slurm Accounting Database is already configured, skip to Configure License Checking.

Requirements

For more information about the Slurm accounting database, see the Slurm Administration Guide.

NOTE: A single Slurm accounting database can manage multiple Slurm clusters. Therefore, the steps below only have to be performed once.
  • the Slurm database daemon slurmdbd and the SQL database server are running on the same instance
  • firewalls are configured to allow traffic between the license server, the Slurm accounting database, the license sensor, and the compute cluster (both the master node and the compute nodes)

This section describes how to install configure the Slurm accounting database for license checking. For more information about the Slurm accounting database, see https://slurm.schedmd.com/accounting.html.

The configuration described below has the following requirements: 

  • the Slurm database daemon slurmdbd and the SQL database server are running on the same instance;
  • the user account slurm already exists in the operating system and can be used to run the Slurm daemons; and
  • firewalls are configured to allow traffic between the license server, the Slurm accounting database, the license sensor, and the compute cluster (both the master node and the compute nodes).

 

The process consists of the following steps:

  1. Install the MariaDB SQL server
  2. Start the MariaDB SQL server
  3. Create the Slurm Accounting Database
  4. Configure the Slurm Control Daemon
  5. Configure the Slurm Database Daemon
  6. Register the Compute Clusters with the Slurm Accounting Database

 

 

Install the MariaDB SQL server

We recommend MariaDB as SQL database. Prebuilt packages are available in the repositories for all supported Linux distributions and can be installed using the distribution's package manager. If you want to build Slurm from source, you also need to install the development packages for MariaDB. For more information regarding building Slurm from source, see the Slurm Admin Quick Start.

Install the following dependencies with your distribution’s package manager:

  • RedHat Enterprise Linux, Rocky Linux, or Ubuntu: mariadb-server

Start the MariaDB SQL server

  1. Create a configuration file (/etc/my.cnf.d/slurm.cnf) with the following content:

    [mysqld]
innodb_buffer_pool_size=1024M
innodb_log_file_size=64M
innodb_lock_wait_timeout=900

  2. Start the MariaDB server and enable its startup when the system boots:

    systemctl enable mariadb
systemctl start mariadb

  3. Confirm that mariadb is running:

    systemctl status mariadb

 

Create the Slurm Accounting Database

Execute the following command to create an SQL database:

mysql -u root << EOF
CREATE DATABASE slurm_acct_db;
GRANT ALL ON slurm_acct_db.* to 'slurm'@'localhost' IDENTIFIED BY 'SQL_PASSWORD' WITH GRANT OPTION;
FLUSH PRIVILEGES;
EOF

This creates the database slurm_acct_db and grants non-privileged access to the user slurm, which is the user account on the operating system that operates all Slurm processes. Replace SQL_PASSWORD with a password you want to use for connecting to this database. The single quotes around the password are required by SQL and are not part of the password.

 

Configure the Slurm Control Daemon

Configure the Slurm control daemon to use the Slurm accounting database. Add the following lines to the Slurm configuration file slurm.conf:

AccountingStorageType=accounting_storage/slurmdbd
AccountingStorageHost=SLURMDBD_HOST

 Replace SLURMDBD_HOST with the hostname of the instance where the Slurm database daemon slurmdbd is running.

 

Configure the Slurm Database Daemon

  1. Create a configuration file for the Slurm database daemon (slurmdbd.conf) with at least the following entries:

    DbdHost=SLURMDBD_HOST
LogFile=/var/log/slurmdbd.log
SlurmUser=slurm
StorageHost=SLURMDBD_HOST
StorageLoc=slurm_acct_db
StorageUser=slurm
StoragePass=SQL_PASSWORD
StorageType=accounting_storage/mysql

    Also here, replace SLURMDBD_HOST with the hostname of the instance where the Slurm database daemon slurmdbd is running, and replace SQL_PASSWORD with the password you selected when setting up the Slurm accounting database (as described in the first step of Create the Slurm Accounting Database). See Slurm documentation at for more information.

  2. Enable and start the Slurm database daemon and restart the Slurm control daemon:

    systemctl enable slurmdbd
systemctl start slurmdbd
systemctl restart slurmctld

  3. Verify that the Slurm control and database daemons are running:

    systemctl status slurmctld
systemctl status slurmdbd

 

Register the Compute Clusters with the Slurm Accounting Database

  1. Register one or more Slurm clusters with the accounting database using a unique name for each cluster:

    sacctmgr -i add cluster CLUSTER_NAME

    Replace CLUSTER_NAME with the name of the Slurm cluster. You can determine the cluster name from the ClusterName setting in the Slurm configuration file slurm.conf.

  2. Add the user slurm as an administrative user:

    sacctmgr -i add account users
sacctmgr -i add user slurm Account=users AdminLevel=Admin

    This user account will be used to update the license count in the accounting database, whenever a license check-out or check-in is registered.

  3. Create the directory /home/slurm/.schrodinger. This directory is required by the license server software to search for license files.

 

 

Configure Slurm for License Checking

Confirm that the license can query and modify the Slurm accounting database:

sudo -u slurm $SCHRODINGER/run python3 slurm-license-sensor

Add the following settings to the Slurm configuration file slurm.conf, if not already present:

JobRequeue=1
RequeueExit=99
SchedulerParameters=allow_zero_lic

The Requeue settings will cause Slurm to automatically requeue job if the startup license check indicates the expected licenses aren’t actually available. The ‘allow_zero_lic’ setting prevents Slurm from rejecting submissions that request more license resources than what Slurm considers the current “max” for that resource; this is useful if there is significant usage of a license type outside the cluster, which can cause the “total” available for cluster jobs to drop temporarily to a very small number.

If slurm.conf was modified, restart the Slurm controller, and confirm it restarted successfully:

systemctl restart slurmctld
systemctl status slurmctld

 

Customize the Load Sensor Script

The Slurm license sensor script regularly queries the license servers and updates number of licenses available to Slurm in the accounting database. It automatically updates the Slurm accounting database with changes to the licenses, for example, when new licenses are added. Each Slurm accounting database requires one Slurm license sensor.

There should be license resources with names <feature_type>@schrodinger_<cluster_name>, with “Total”, “Used”, and “Free” values.

  1. Copy $SCHRODINGER/utilities/slurm-license-sensor to a location independent of the Schrödinger installation. This is so your customizations won't be lost if you later remove that installation.

    • CLUSTER_DNS_NAME_REGEX: a regular expression that matches the DNS names of all nodes managed by Slurm, but does not match the names of any machines not managed by Slurm. This regex is used to identify internal (i.e., cluster) license usage versus external (e.g., laptop, workstation) license usage from the license server status report. External license usage will cause a decrease in the total number of license resources available for Slurm jobs.

    • SLURM_CLUSTERS: a comma-separated list of all clusters managed by Slurm that report to the same Slurm accounting database (the cluster names are those from Register the Compute Clusters with the Slurm Accounting Database above).

    • SLURM_INSTALLDIR: the installation directory of Slurm. This directory should contain a ‘bin’ subdirectory with the ‘scontrol’ and ‘sacctmgr’ executables.

    • [Optional ] CLUSTER_LICENSE_LIMITS: a dictionary of upper limits on the number of license resources of specific license types allowed for Slurm jobs. This can be used to prevent 100% usage of a license type by the cluster, so there are some licenses available for use on user laptops/workstations.

  2. Confirm that the load sensor is retrieving license usage information from the license server(s) and identifying the current internal vs. external usage correctly:

    $SCHRODINGER/run python3 slurm-license-sensor --dry-run

    To see what machines currently have the licenses checked out, run:

    $SCHRODINGER/run lictool status

The names of the licenses stored in the Slurm accounting database are the same as found in the license file of the Schrödinger Suite. For example, if there is one license server hosting 100 licenses of feature X to two Slurm clusters, alice and bob, modify the SLURM_CLUSTERS setting: 

SLURM_CLUSTERS = ["alice", "bob"]

The Slurm license sensor adds the resource information to the Slurm accounting database as follows:

sacctmgr -i add resource name=X count=100 server=schrodinger_alice cluster=alice type=license percentallowed=100
sacctmgr -i add resource name=X count=100 server=schrodinger_bob cluster=bob type=license percentallowed=100

 

Create Load Center System Service

Confirm that the slurm user can query the license server(s) and modify the Slurm accounting database:

sudo -u slurm $SCHRODINGER/run python3 <sensor_dir>/slurm-license-sensor

To keep the information in the Slurm accounting database up to date, the license sensor should run in regular, short intervals. We recommend executing the license sensor every 30 seconds. An easy way to implement this is by using a systemd timer.

  1. Create a file /etc/systemd/system/slurm-license-sensor.service:

    [Unit]
Description=Schrodinger license sensor for Slurm
After=slurmctld.service slurmdbd.service

[Service]
Type=simple
User=slurm
ExecStart=SCHRODINGER/run python3 <sensor_dir>/slurm-license-sensor
					
[Install]
WantedBy=multiuser.target

    Replace SCHRODINGER with the absolute path of the Schrödinger Suite installation.

  2. Create a file /etc/systemd/system/slurm-license-sensor.timer

    [Unit]
Description=Schrodinger license sensor for Slurm

[Timer]
OnCalendar=*:*:0/30
AccuracySec=1s

[Install]
WantedBy=timers.target

    The line OnCalendar=*:*:0/30 specifies that the license sensor script should run every 30 seconds.

  3. Enable and activate the timer:

    systemctl enable slurm-license-sensor.timer
systemctl start slurm-license-sensor.timer
Note: while the customized load sensor script is no longer located in a specific Schrödinger installation, it must be run via a Schrödinger installation, and this is set in the system service. Therefore, the system service must be updated if that Schrödinger installation later will be remove

 

Turn On License Checking in Job Server

To turn on license checking, edit the Job Server configuration file <jobserver_dir>/config/jobserver.yml.

In the job_server section, change the following option to set up license checking:

  • license_checking: set to true if you have configured the queueing system and the Schrödinger Suite for license checking.

  • remote_license_server: The name of the Slurm cluster which is connected to this Job Server installation, with a "schrodinger_" prefix. For example, 

    remote_license_server: schrodinger_CLUSTER_NAME

    The name of the cluster ( CLUSTER_NAME) (the cluster names are those from Register the Compute Clusters with the Slurm Accounting Database above) must match the name specified in the SLURM_CLUSTERS list in the Slurm license sensor script.

After making changes to the configuration file, you must restart the Job Server:

sudo systemctl stop jobserverd
sudo systemctl start jobserverd

 

Test the Configuration

To confirm that the license checking is working correctly, perform the following tests.

First, check the current status of the Slurm MMLIBS license resource:

scontrol show lic | grep -A1 mmlibs

Note the “Total” number of license resources available to the cluster.

 

Test License Checking with external license usage

  1. On an machine not managed by Slurm but with access to the same license server, execute:

    $SCHRODINGER/run lictest -d -l MMLIBS:4 -p 100

    This checks out 4 MMLIBS licenses for 100 seconds.

  2. At the same time, for Slurm jobs, execute:

    scontrol show lic | grep -A1 mmlibs

  3. Confirm that the total number of licenses available to Slurm is reduced by four. If no other Schrödinger jobs are running, the previous command returns:

    Total=<N-4> Used=0 Free=<N-4> Remote=yes

    where N is the total number of available MMLIBS licenses on your license server. If a license is checked out outside of Slurm, the number of licenses available for Slurm jobs is reduced. The “Total” and “Free” counters are reduced by the same amount. The “Used” counter remains zero, as no license is checked out from a Slurm instance.

It may need up to 30 seconds until the number of available licenses is updated since the license sensor runs every 30 seconds.

 

Test License Checking from instances managed by Slurm

  1. Submit a test job to the Slurm cluster:

    $SCHRODINGER/testapp -HOST CLUSTER_HOST_ENTRY -t 300 -l MMLIBS:4 -j testapp.mmlibs_4

    This submits a test job with the name testapp_mmlibs_4 to the schrodinger.hosts entry SLURM_QUEUE. The job checks out 4 MMLIBS licenses and runs for 300 seconds.

  2. While the job is running, check the Slurm license resource usage:

    scontrol show lic | grep -A1 mmlibs

    This returns:

    Total=<N> Used=4 Free=<N-4> Remote=yes

    where N is the total number of available MMLIBS licenses on your license server. If a license is checked out from an instance that is part of a Slurm cluster, the “Used” counter increases and the “Free” counter decreases by the same amount, but the “Total” number of license resources available to the cluster does not change.

 

Local license checking is only applicable when the license is used by jobs running on a single Slurm cluster. It is not recommended except in specific circumstances (e.g., there are only a few Schrodinger job types run on the cluster, and those job types are never run anywhere else). If there will be license usage outside the cluster competing with the cluster’s use of licenses, the Configure License Checking License Checking configuration is strongly recommended.

 

The process consists of the following steps:

 

1. Configure Slurm Local for license checking

  1. A resource attribute representing each license must be defined in the queuing system configuration. The utility licutil can be used to parse your license file or query the license server and generate the configuration text that needs to be added to the queuing system configuration files.

    To print the license resource list, run:

    $SCHRODINGER/utilities/licutil -slurmconf

  2. Add the licenses to the Slurm configuration file (e.g. /etc/slurm/slurm.conf).

  3. Make sure the following settings are also included in slurm.conf:

    JobRequeue=1
RequeueExit=99

  4. Restart the Slurm control daemon:

    systemctl restart slurmctld

  5. Confirm that the license information is available in Slurm:

    scontrol show lic

There should be license resources with names <feature_type>, with “Total”, “Used”, and “Free” values.

NOTE: If your licenses change (e.g. new license types are added or license counts change), the above steps must be repeated.

 

2. Turn On License Checking in Job Server

To turn on license checking, edit the Job Server configuration file <jobserver_dir>/config/jobserver.yml.

In the job_server section, change the following option to set up license checking:

  • license_checking: set to true if you have configured the queueing system and the Schrödinger Suite for license checking.

  • remote_license_server: The name of the Slurm cluster which is connected to this Job Server installation, with a "schrodinger_" prefix. For example, 

    remote_license_server: schrodinger_CLUSTER_NAME

    The name of the cluster ( CLUSTER_NAME) (the cluster names are those from Register the Compute Clusters with the Slurm Accounting Database above) must match the name specified in the SLURM_CLUSTERS list in the Slurm license sensor script.

After making changes to the configuration file, you must restart the Job Server:

sudo systemctl stop jobserverd
sudo systemctl start jobserverd

3. Test the Configuration

To confirm that the license checking is working correctly, perform the following tests.

First, check the current status of the Slurm MMLIBS license resource:

scontrol show lic | grep -A1 mmlibs

Note the “Total” number of license resources available to the cluster.

 

Test License Checking with external license usage

  1. On an machine not managed by Slurm but with access to the same license server, execute:

    $SCHRODINGER/run lictest -d -l MMLIBS:4 -p 100

    This checks out 4 MMLIBS licenses for 100 seconds.

  2. At the same time, for Slurm jobs, execute:

    scontrol show lic | grep -A1 mmlibs

  3. Confirm that the total number of licenses available to Slurm is reduced by four. If no other Schrödinger jobs are running, the previous command returns:

    Total=<N-4> Used=0 Free=<N-4> Remote=no

    where N is the total number of available MMLIBS licenses on your license server. If a license is checked out outside of Slurm, the number of licenses available for Slurm jobs is reduced. The “Total” and “Free” counters are reduced by the same amount. The “Used” counter remains zero, as no license is checked out from a Slurm instance.

It may need up to 30 seconds until the number of available licenses is updated since the license sensor runs every 30 seconds.

 

Test License Checking from instances managed by Slurm

  1. Submit a test job to the Slurm cluster:

    $SCHRODINGER/testapp -HOST CLUSTER_HOST_ENTRY -t 300 -l MMLIBS:4 -j testapp.mmlibs_4

    This submits a test job with the name testapp_mmlibs_4 to the schrodinger.hosts entry SLURM_QUEUE. The job checks out 4 MMLIBS licenses and runs for 300 seconds.

  2. While the job is running, check the Slurm license resource usage:

    scontrol show lic | grep -A1 mmlibs

    This returns:

    Total=<N> Used=4 Free=<N-4> Remote=no

    where N is the total number of available MMLIBS licenses on your license server. If a license is checked out from an instance that is part of a Slurm cluster, the “Used” counter increases and the “Free” counter decreases by the same amount, but the “Total” number of license resources available to the cluster does not change.

Grid Engine

Requirements

See Supported Queuing Systems for specific versions of the queueing systems that are supported.

 

The load sensor must be run on a machine that has access to:

  1. the Schrodinger lictool executable

  2. the license server(s)

  3. the Grid Engine ‘qconf’ executable (whose location must be in PATH)

NOTE: Jobs launched directly from a cluster node (e.g., via an interactive session or direct login) will consume licenses without the queuing system being notified of that resource usage. We therefore recommend running all Schrödinger jobs as Grid Engine batch jobs via Job Server so they will request the appropriate license resources from Grid Engine.

 

The process consists of the following steps:

 

Configuring the Load Sensor Script

Copy both $SCHRODINGER/utilities/sge_license_sensor and $SCHRODINGER/internal/bin/lictool to a location independent of the Schrodinger installation (<sensor_dir>), so your customizations won’t be lost if you later remove that installation.

Make the load sensor script executable:

chmod +x <sensor_dir>/sge_license_sensor

Edit the <sensor_dir>/sge_license_sensor script, setting LICTOOL to <sensor_dir>/lictool.

 

Other configuration options

  • CLUSTER_LICENSE_LIMITS: (optional) This is a dictionary of upper limits on the number of license resources of specific license types allowed for jobs. This can be used to prevent 100% usage of a license type by the cluster, so there are some licenses available for use on user laptops/workstations.

Test that the load sensor script is able to query your license servers

Test that the load sensor script is able to query your license server and convert that information into load values by running:

<sensor_dir>/sge_license_sensor --dry-run --skip-complex-setup

 

Adding license attributes

A resource attribute representing each license must be defined in the queuing system configuration. To define the license attributes, run:

 <sensor_dir>/sge_license_sensor --setup-complexes

This also will confirm that the script is able to run ‘qconf’. To confirm that the license attributes have been added, run:

 qconf -sc

Once the load sensor is installed, it will set up the complex each time it starts up. However, if the Schrödinger licenses issued for your server change (e.g., new license types have been added) while the load sensor is already running, you should run the command above manually to update the configuration.

 

Installing the Load Sensor Script

To install the load sensor script, add a load_sensor attribute to the host configuration of one of the hosts managed by Grid Engine. The load sensor is executed periodically on that host and feeds information about license availability back into the queueing system. You can get a list of the execution hosts using the command:

qconf -sel

To add the load sensor to the host configuration for the chosen host:

1. Enter the following command on one of the execution hosts, as listed in the output from qconf -sel: 
qconf -mconf hostname
2. In the resulting edit window, add the line 
load_sensor    <sensor_dir>/sge_license_sensor

where <sensor_dir> is the directory where you stored the edited copy of sge_license_sensor.

3. Save the modified file and close the editor.
4. Verify that your changes were accepted by displaying the host configuration with the command: 
qconf -sconf hostname

Grid Engine starts the load sensor script automatically after you have made the load_sensor setting, and also restarts it automatically if you modify or update the script sge_license_sensor. The command

qhost -F -h hostname

displays all of the resource attributes for hostname If the load sensor is running and working properly, you should see your licenses in this list.

For example,

   gl:IMPACT_MAIN=20.000000
   gl:MMLIBS=20.000000

The default load sensor report time interval is 40 seconds. If you observe delay, consider decreasing the interval with the following command:

qconf -mconf load_report_time 00:00:10

This command sets the time to 10 seconds. See man sge_conf for more details about load_report_time.

If the load sensor has been configured to enable internal tracking of license resources by the Grid Engine, run the following command to confirm that the license resources have been defined both as 'complex_values' (for internal tracking) and as 'load_values' (from the load sensor).

qconf -se global

Grid Engine should be now configured to handle Schrödinger licenses properly.

 

Turning On License Checking in Job Server

To turn on license checking, edit the Job Server configuration file <jobserver_dir>/config/jobserver.yml.

In the job_server section, change the following option to set up license checking:

  • license_checking: set to true if you have configured the queueing system and the Schrödinger Suite for license checking.

After making changes to the configuration file, you must restart the Job Server:

sudo systemctl stop jobserverd
sudo systemctl start jobserverd

PBS Pro

Requirements

See Supported Queuing Systems for specific versions of the queueing systems that are supported.

 

The load sensor must be run on a machine that has access to:

  1. the Schrodinger lictool executable

  2. the license server(s)

  3. the PBS Pro configuration files

NOTE: Jobs launched directly from a cluster node (e.g., via an interactive session or direct login) will consume licenses without the queuing system being notified of that resource usage. We therefore recommend running all Schrödinger jobs as PBS Pro batch jobs via Job Server so they will request the appropriate license resources from PBS Pro.

 

The process consists of the following steps:

 

Configuring the Load Sensor Script

Copy both $SCHRODINGER/utilities/pbs_lic_sensor and $SCHRODINGER/internal/bin/lictool to a location independent of the Schrodinger installation (<sensor_dir>), so your customizations won’t be lost if you later remove that installation.

Make the load sensor script executable:

chmod +x <sensor_dir>/pbs_lic_sensor

Edit the <sensor_dir>/pbs_lic_sensor script, setting LICTOOL to <sensor_dir>/lictool.

 

Other configuration options

  • CLUSTER_LICENSE_LIMITS: (optional) This is a dictionary of upper limits on the number of license resources of specific license types allowed for jobs. This can be used to prevent 100% usage of a license type by the cluster, so there are some licenses available for use on user laptops/workstations.

Test that the load sensor script is able to query your license servers

Test that the load sensor script is able to query your license server and convert that information into load values by running:

<sensor_dir>/pbs_lic_sensor --dry-run

 

Adding License Attributes and the Load Sensor Execution to PBS Pro

A resource attribute representing each license must be defined in the queuing system configuration. The pbs_lic_sensor script can be used to parse your license file or query the license server and generate the configuration text that needs to be added to the queuing system configuration files. To define the license attributes, run:

<sensor_dir>/pbs_lic_sensor --print-resources

 

The output from pbs_lic_sensor should consist of lines like the following:

Copy the text between the dashed lines below into your
<PBS_HOME>/server_priv/resourcedef file.
----------------------------------------------------------------------
IMPACT_MAIN  type=long
MMLIBS  type=long
----------------------------------------------------------------------

Add the text between the dashed lines below to the value of
the 'resources' parameter in your <PBS_HOME>/sched_priv/sched_config file.
E.g., if the 'resources' line in the file was
resources: "ncpus, mem, arch"
and the text between the lines was
feature1, feature3, feature5
then the new 'resources' line should be
resources: "ncpus, mem, arch, feature1, feature3, feature5"
----------------------------------------------------------------------
IMPACT_MAIN, MMLIBS
----------------------------------------------------------------------

Copy the text between the dashed lines below into your
<PBS_HOME>/sched_priv/sched_config file.
----------------------------------------------------------------------
server_dyn_res: "IMPACT_MAIN !<sensor_dir>/pbs_lic_sensor IMPACT_MAIN"
server_dyn_res: "MMLIBS !<sensor_dir>/pbs_lic_sensor MMLIBS"
----------------------------------------------------------------------

 

In the first and third parts of this output, there should be one line for each license type available from your server. To add the new license attributes, follow the instructions in the output. The files that you need to modify are $PBS_HOME/server_priv/resourcedef and $PBS_HOME/sched_priv/sched_config, where $PBS_HOME is the PBS home directory, often something like /var/spool/PBS on the head node of your cluster.

The first two configuration steps add the license attributes to PBS Pro. The third step instructs PBS Pro to execute the configured load sensor script to retrieve the current license resource availability.

For the changes to take effect, you must restart PBS Pro. The command for doing this is typically the following:

sudo systemctl restart pbs

Refer to your PBS Pro Administrator Guide for more information about restarting PBS Pro.

 

Turning On License Checking in Job Server

To turn on license checking, edit the Job Server configuration file <jobserver_dir>/config/jobserver.yml.

In the job_server section, change the following option to set up license checking:

  • license_checking: set to true if you have configured the queueing system and the Schrödinger Suite for license checking.

After making changes to the configuration file, you must restart the Job Server:

sudo systemctl stop jobserverd
sudo systemctl start jobserverd

LSF

Requirements

See Supported Queuing Systems for specific versions of the queueing systems that are supported.

 

The load sensor must be run on a machine that has access to:

  1. the Schrodinger lictool executable

  2. the license server(s)

  3. the LSF ‘lsid’ executable (whose location must be in PATH)

NOTE: Jobs launched directly from a cluster node (e.g., via an interactive session or direct login) will consume licenses without the queuing system being notified of that resource usage. We therefore recommend running all Schrödinger jobs as LSF batch jobs via Job Server so they will request the appropriate license resources from LSF.

 

The process consists of the following steps:

 

Configuring the Load Sensor Script

Copy both $SCHRODINGER/utilities/elim.schrodinger and $SCHRODINGER/internal/bin/lictool to a location independent of the Schrodinger installation (<sensor_dir>), so your customizations won’t be lost if you later remove that installation.

Note: LSF will not execute the load sensor script from <sensor_dir> because the customized script later will be copied to the LSF configuration (see below). That script needs to run lictool, though, and for stability it is best to have a copy separate from any Schrodinger installation. So <sensor_dir> will be the location of lictool and your "in progress" version of the load sensor script whose final version will get copied to the LSF installation.

Make the load sensor script executable:

chmod +x <sensor_dir>/elim.schrodinger

Edit the <sensor_dir>/elim.schrodinger script, setting LICTOOL to <sensor_dir>/lictool.

Also set CLUSTER_DNS_NAME_REGEX to a regular expression that matches the DNS names of all nodes managed by LSF, but does not match the names of any machines not managed by LSF. This regex is used to identify internal (i.e., cluster) license usage versus external (e.g., laptop, workstation) license usage from the license server status report. External license usage will cause a decrease in the total number of license resources available for LSF jobs.

 

Other configuration options

  • CLUSTER_LICENSE_LIMITS: (optional) This is a dictionary of upper limits on the number of license resources of specific license types allowed for jobs. This can be used to prevent 100% usage of a license type by the cluster, so there are some licenses available for use on user laptops/workstations.

  • POLLING_INTERVAL (default is 30 seconds)

Test that the load sensor script is able to query your license servers

Test that the load sensor script is able to query your license server and convert that information into load values by running:

<sensor_dir>/elim.schrodinger --dry-run

The ‘--dry-run’ option will cause the script to poll the license server just once, rather than every POLLING_INTERVAL.

If CLUSTER_DNS_NAME_REGEX has been set properly, the reported license availability should reflect usage outside the cluster, but not any usage on cluster nodes (because LSF tracks that internally).

 

Adding license attributes to LSF

A resource attribute representing each license must be defined in the queuing system configuration. The utility lictool is used to parse your license file or query the license server and generate the configuration text that needs to be added to the queuing system configuration files. To define the license attributes, run:

<sensor_dir>/elim.schrodinger --print-resources

 

The output from elim.schrodinger should consist of lines like the following:

  Copy the text between the dashed lines below into the "Resource" section of
  your lsf.shared file. (Don’t, however, copy the RESOURCENAME line below if
  there’s already such a line there.)
  ----------------------------------------------------------------------
  RESOURCENAME             TYPE      INTERVAL  INCREASING  DESCRIPTION
  IMPACT_MAIN              Numeric         10           N  (IMPACT_MAIN license)
  MMLIBS                   Numeric         10           N  (MMLIBS license)
  ----------------------------------------------------------------------
   
  Copy the text between the dashed lines below into the "Parameters" section
  of your lsf.cluster.<your_cluster_name> file:
  ----------------------------------------------------------------------
  LSF_ELIM_DEBUG=y
  ----------------------------------------------------------------------
   
  Copy the text between the dashed lines below into the "ResourceMap" section
  of your lsf.cluster.<your_cluster_name> file.  (Don’t, however, copy the
  RESOURCENAME line below if there’s already such a line there.)
  ----------------------------------------------------------------------
  RESOURCENAME             LOCATION
  IMPACT_MAIN              [all]
  MMLIBS                   [all]
  ----------------------------------------------------------------------

In the first and third parts of this output, there should be one line for each license type available from your server. To add the new license attributes, follow the instructions in the output. The files you need to modify, lsf.shared and lsf.cluster.<your_cluster_name>, are located in the $LSF_CONFDIR directory, which is usually something like /usr/lsf/conf. If this directory is not cross-mounted between all of the hosts to which LSF can submit jobs, you will have to make these changes in the configuration files on each host.

To verify that the changes were made correctly, restart LSF with the commands

  lsadmin reconfig
  badmin mbdrestart

and then list the resources LSF recognizes using the command

  lsload -l

The output should include a column for each of the new license resources, for example,

  HOST_NAME               status  <...>   mem  IMPACT_MAIN MMLIBS
  host1.mycluster.com         ok  <...>  365M            -      -
  host2.mycluster.com         ok  <...>  362M            -      -

 

Installing the Load Sensor Script

1. Copy the edited script into the $LSF_SERVERDIR directory on the LSF master host: 
cp <sensor_dir>/elim.schrodinger $LSF_SERVERDIR

The $LSF_SERVERDIR directory is typically named something like /usr/lsf/6.2/linux2.6-glibc2.3-x86/etc/ and also contains the program melim, which manages ELIMs.

2. Restart LSF with the following commands: 
lsadmin reconfig
badmin mbdrestart
3. Check that the new ELIM is active, using the command 
lsload -l

The output should now contain the number of available tokens for each license; for example,

  HOST_NAME               status  <...>   mem  IMPACT_MAIN MMLIBS
  host1.mycluster.com        ok   <...>  365M           10     15
  host2.mycluster.com        ok   <...>  365M           10     15

LSF should now be configured to handle Schrödinger license requirements.

 

Turning On License Checking in Job Server

To turn on license checking, edit the Job Server configuration file <jobserver_dir>/config/jobserver.yml.

In the job_server section, change the following option to set up license checking:

  • license_checking: set to true if you have configured the queueing system and the Schrödinger Suite for license checking.

After making changes to the configuration file, you must restart the Job Server:

sudo systemctl stop jobserverd
sudo systemctl start jobserverd