Configuring Remote License Checking for Slurm

A license sensor script queries the license server at regular intervals and updates the information in the Slurm accounting database. Slurm schedules a job only if both the computational as well as license resources are available.

In order to use remote license checking, you need to use Slurm version 18.08.2 or newer, and a configured Slurm accounting database. If you have already configured the Slurm accounting database, you can skip to Step 3: Configuring the Slurm License Sensor. If you need to set up the Slurm accounting database, refer to Step 1: Installing the Slurm Accounting Database.

Caveats and Limitations

License checking reduces job failures due to unavailable licenses. However, it is subject to the following limitations:

If you have multiple license servers, the load sensor sums the license availability over all the servers. However, an individual license checkout cannot span servers. For example, if a job requires two tokens of a given feature and there is only one token available on server A and only one token available on server B, the license checkout will fail.
The license sensor script runs in regular intervals. Situations may arise when a job may be started even though the required license is not available, as the competing license checkout is not yet registered in the Slurm accounting database.
Suite licenses are not properly accounted for. If license features A and B are both components of a library (a line starting with PACKAGE in the license file), the checkout of feature A does not update the number of available licenses for feature B.
The license sensor cannot discriminate between license checkouts from queued jobs and jobs launched directly from the same node. We recommend running all Schrödinger jobs under the control of Slurm.

1. Installing the Slurm Accounting Database

This section describes how to install configure the Slurm accounting database for license checking. For more information about the Slurm accounting database, see https://slurm.schedmd.com/accounting.html.

The configuration described below has the following requirements:

the Slurm database daemon slurmdbd and the SQL database server are running on the same instance;
the user account slurm already exists in the operating system and can be used to run the Slurm daemons; and
firewalls are configured to allow traffic between the license server, the Slurm accounting database, the license sensor, and the compute cluster (both the master node and the compute nodes).

A single Slurm accounting database can manage multiple Slurm clusters. Therefore, the steps below only have to be performed once.

Installing The MariaDB SQL Server

We recommend MariaDB as SQL database. Prebuilt packages are available in the repositories for all supported Linux distributions and can be installed using the distribution's package manager. If you want to build Slurm from source, you also need to install the development packages for MariaDB. For more information regarding building Slurm from source, see: https://slurm.schedmd.com/quickstart_admin.html

Install the following dependencies with your distribution’s package manager:

RedHat Enterprise Linux, CentOS, or Ubuntu: mariadb-server
SuSE Linux Enterprise Server: mariadb and mariadb-tools

Starting The MariaDB SQL Server

Create a configuration file (/etc/my.cnf.d/slurm.cnf) with the following content:

[mysqld]
innodb_buffer_pool_size=1024M
innodb_log_file_size=64M
innodb_lock_wait_timeout=900

Start the MariaDB server and enable its startup when the system boots:
```
systemctl enable mariadb
systemctl start mariadb
```
Confirm that mariadb is running:
```
systemctl status mariadb
```

Creating The Database

Execute the following command to create an SQL database:

mysql -u root << EOF
CREATE DATABASE slurm_acct_db;
GRANT ALL ON slurm_acct_db.* to 'slurm'@'localhost' IDENTIFIED BY 'SQL_PASSWORD' WITH GRANT OPTION;
FLUSH PRIVILEGES;
EOF

This creates the database slurm_acct_db and grants non-privileged access to the user slurm, which is the user account on the operating system that operates all Slurm processes. Replace SQL_PASSWORD with a password you want to use for connecting to this database. The single quotes around the password are required by SQL and are not part of the password.

2. Configuring Slurm for License Checking

Update both the Slurm and the Slurm accounting configuration files to use the Slurm accounting database.

Configuring the Slurm Control Daemon

Configure the Slurm control daemon to use the Slurm accounting database. Add the following lines to the Slurm configuration file slurm.conf:

AccountingStorageType=accounting_storage/slurmdbd
AccountingStorageHost=SLURMDBD_HOST

Replace SLURMDBD_HOST with the hostname of the instance where the Slurm database daemon slurmdbd is running.

Make sure the following settings are also included in slurm.conf:

JobRequeue=1
RequeueExit=99

In cases of multiple clusters or significant non-cluster usage of the licenses (e.g., workstation/laptops which share licenses with their cluster), it is possible for the "total" number of licenses available to drop to a very small number, or even 0. To prevent submission failures when a job requests more licenses than the "total" number, you can set the following in slurm.conf:

SchedulerParameters=allow_zero_lic

Configuring the Slurm Database Daemon

Create a configuration file for the Slurm database daemon (slurmdbd.conf) with at least the following entries:
```
DbdHost=SLURMDBD_HOST
LogFile=/var/log/slurmdbd.log
SlurmUser=slurm
StorageHost=SLURMDBD_HOST
StorageLoc=slurm_acct_db
StorageUser=slurm
StoragePass=SQL_PASSWORD
StorageType=accounting_storage/mysql
```
Replace SQL_PASSWORD with the password you selected when setting up the Slurm accounting database (as described in Step 1: Installing the Slurm Accounting Database). See also the Slurm documentation at https://slurm.schedmd.com/slurmdbd.conf.html for more information.
Enable and start the Slurm database daemon and restart the Slurm control daemon:
```
systemctl enable slurmdbd
systemctl start slurmdbd
systemctl restart slurmctld
```
Verify that the Slurm control and database daemons are running:
```
systemctl status slurmctld
systemctl status slurmdbd
```

4. Configuring the Slurm License Sensor

The Slurm license sensor script regularly queries the license servers and updates the number of available licenses in the Slurm accounting database. It automatically updates the Slurm accounting database with changes to the license, for example, when new licenses are added. Each Slurm accounting database requires one Slurm license sensor.

Edit the file $SCHRODINGER/utilities/slurm-license-sensor:
- CLUSTER_DNS_NAME_REGEX: a regular expression that matches the DNS names of all instances managed by Slurm, but does not match any instance not managed by Slurm. This is used to distinguish between license checkouts from within the Slurm cluster and external resources, for example, a laptop or a workstation.
- SLURM_CLUSTERS: a comma-separated list of all clusters managed by Slurm that report to the same Slurm accounting database
- SLURM_INSTALLDIR: the installation directory of Slurm
- SCHRODINGER_LICENSING_NAMESPACE = "SLM_":Set the prefix to add for license features from Schrodinger License Manager (SLM) servers, so those Slurm resources can be distinguished from FlexNet server licenses. This must match the prefix for SLM licenses set in the Job Server configuration (schrodinger_licensing_namespace in the jobserver.yml file). If you are no longer using FlexNet licenses, both settings can be an empty string.
Confirm that the license sensor can query the license server and passes this information to the Slurm accounting database:
```
$SCHRODINGER/utilities/slurm-license-sensor
```
Note: This command is best executed as the same user that updates the Slurm accounting database, for example the slurm user. This command requires that the slurm user account has access to the license server that hosts the licenses for the Schrödinger Suite.
Confirm that the license information is available in Slurm:
```
scontrol show lic
```

The names of the licenses stored in the Slurm accounting database are the same as found in the license file of the Schrödinger Suite. For example, if there is one license server hosting 100 licenses of feature X to two Slurm clusters, alice and bob, modify the SLURM_CLUSTERS setting:

SLURM_CLUSTERS = ["alice", "bob"]

The Slurm license sensor adds the resource information to the Slurm accounting database as follows:

sacctmgr -i add resource name=X count=100 server=schrodinger_alice cluster=alice servertype=flexlm percentallowed=100
sacctmgr -i add resource name=X count=100 server=schrodinger_bob cluster=bob servertype=flexlm percentallowed=100

Each Slurm cluster is listed in the Slurm accounting database in their own “server” namespace, using up to 100% of the available licenses. This avoids the need to assign fixed percentages of permitted license usage to each cluster. For example, if both clusters were to share the same server=schrodinger namespace with percentallowed=50, jobs submitted to cluster alice would only be able to use up to 50% of the licenses, even if there are no jobs running on cluster bob. See the Slurm Licenses Guide for more information.

To keep the information in the Slurm accounting database up to date, the license sensor should run in regular, short intervals. We recommend executing the license sensor every 30 seconds. An easy way to implement this is by using a systemd timer.

Creating a Systemd Timer for the Slurm License Sensor

Create a file /etc/systemd/system/slurm-license-sensor.service:
```
[Unit]
Description=Schrodinger license sensor for Slurm
After=slurmctld.service slurmdbd.service

[Service]
Type=simple
User=slurm
ExecStart=SCHRODINGER/run SCHRODINGER/utilities/slurm-license-sensor
					
[Install]
WantedBy=multiuser.target
```
Replace SCHRODINGER with the absolute path of the Schrödinger Suite installation. If users access the license server via environment variables LM_LICENSE_FILE or SCHROD_LICENSE_FILE, you can add a line:
```
Environment=SCHROD_LICENSE_FILE=PORT@LICENSE-SERVER
```
to the [Service] section of the service file.
Create a file /etc/systemd/system/slurm-license-sensor.timer
```
[Unit]
Description=Schrodinger license sensor for Slurm

[Timer]
OnCalendar=*:*:0/30
AccuracySec=1s

[Install]
WantedBy=timers.target
```
The line OnCalendar=*:*:0/30 specifies that the license sensor script should run every 30 seconds.

Change permissions to make the file executable:

chmod +x $SCHRODINGER/utilities/slurm-license-sensor

Enable and activate the timer:

systemctl enable slurm-license-sensor.timer
systemctl start slurm-license-sensor.timer