Advanced Configuration Options

  • Tasks in the configuration file
  • jsc admin Tasks

Tasks in the configuration file

To modify the configuration of Job Server, edit the Job Server configuration file <jobserver_dir>/config/jobserver.yml. You can check the configuration file for a complete list of configuration options.

After making changes to the configuration file, you must restart the server:

sudo systemctl stop jobserverd
sudo systemctl start jobserverd

The admin_users section contains a list of one or more user names that should be able to see and interact with jobs submitted by all users. Regular users can only see jobs they have submitted themselves. By default, this section will be templated for you as

admin_users: 	  
		  -  " "

If left unchanged, there will be no admin users. To add admin users, you should fill out the dashed list, one user per line, like so:

admin_users: 	  
		  -  "user1"
		  -  "user2"

For authenticating users via the LDAP “distinguished name” , two different patterns are supported. If all users’ distinguished names have the same pattern, use the bind_dn_template setting. If that is not possible, fill the lookup_user, lookup_password, base_dn, and search_filter settings.

  • addr: add the fully qualified domain name and port of the LDAP server separated by a colon, e.g., ldap-server.domain:636

  • tls (default: true): use the TLS protocol for communicating with the LDAP server. Setting this to false is not recommended as this transfers user credentials between Job Server and the LDAP server unencrypted over the network during the authentication challenge.

  • insecure_tls_skip_verify: set this to true if you want to use a TLS certificate that is not signed by a Certificate Authority. This skips the verification of the certificate chain.

  • bind_dn_template: create a string template that will form the user’s distinguished name when the template will be filled with the user name. For example:

    bind_dn_template: "uid={{.User}},ou=people,dc=example,dc=com"

    To query an LDAP server for the distinguished name, you can check:

    ldapsearch -H ldaps://<ldapserver> "(uid=$(whoami))" dn -x

    If the distinguished names have different patterns, set bind_dn_template to an empty string:

    bind_dn_template: " "

  • lookup_user: the username of the LDAP service account with permissions to search other users' distinguished names (required if bind_dn_template is not specified).

  • lookup_password: the password for the LDAP service account specified in lookup_user (required if bind_dn_template is not specified).

  • base_dn: the base LDAP distinguished name under which the user’s distinguished name is searched. For example:

    base_dn: "dc=example,dc=com"

    (required if bind_dn_template is not specified)

  • search_filter: a string template that when combined with base_dn uniquely identifies a user with the ldapsearch command. For example:

    search_filter: "(uid={{.User}})"

    or:

    search_filter: "(&(objectcategory=person)(samaccountname={{.User}}))"

    The parentheses are required. (search_filter is required if bind_dn_template is not specified).

In the job_server section, you can change the following options to set up license checking:

  • license_checking: set to true if you have configured the queueing system and the Schrödinger Suite for license checking.

  • remote_license_server: The name of the Slurm cluster which is connected to this Job Server installation, with a "schrodinger_" prefix. For example, 

    remote_license_server: schrodinger_SLURMClusterName

    The name of the cluster ( SLURMClusterName) must match the name specified in the SLURM_CLUSTERS list in the Slurm license sensor script: $SCHRODINGER/utilities/slurm-license-sensor. In order to set up license checking for Slurm, refer to Configuring Remote License Checking for Slurm

    This option is only relevant for the Slurm queueing system if remote license checking via a Slurm accounting database is used to track the license availability. See the Slurm Licenses Guide for more information.

In the job_server section, you can change the following option to change the Job Server API port:

  • port: If you want to change the Job Server API port, choose an available port accordingly. The default port is 8030/tcp.

In the job_server section, change the following option:

  • monitor_interval (default: 10): The interval in seconds between two successive pollings of the HPC queue for the status of queue jobs.

In the job_server section, change the following options:

  • Job files in the file store are removed automatically after they have been downloaded.  In the job_server section, there are several parameters controlling when jobs get removed from the Job Server database and when undownloaded job files get removed from the file store.

  • failed_job_cleanup_days (default: 7): The waiting period in days for automatic cleanup of “Failed” and “Canceled” jobs or canceled jobs.  This includes both removal of the job information from the Job Server database and the removal of any remaining job files in the file store.

  • done_job_cleanup_days (default: 14): The waiting period in days for automatic cleanup of “Completed” and “Stopped” jobs whose files already have already been downloaded and then removed from the file store.  Since the job files are no longer in the file store, only the job information needs to be removed from the Job Server database.

  • abandoned_job_cleanup_days (default: 28): The waiting period in days for automatic cleanup of “Completed” and “Stopped” jobs whose files have not yet been downloaded from the file store (i.e., they are “abandoned”).  This includes both removal of the job information from the Job Server database and the removal of any remaining job files in the file store.

Note: These cleanup criteria are applied only to the top-level job of a job hierarchy.  Subjobs, subsubjobs, etc. of a job hierarchy are cleaned up when the top-level job meets one of the above criteria.
The log_rotation section is a subsection of the job_server section. Use the use_log_rotation option to enable or disable log rotation, and the other fields to configure its behavior.
By default, log_rotation is enabled with the following settings:

  • use_log_rotation(default: true): By default, log files will be archived and compressed if they exceed the size specified by max_log_size and a new log file will be used. If set to false, one continuous log file will be used instead. Old log files are retained in the same directory with the name jobserverd-<timestamp>.log" where the <timestamp> is of the form
    2020-09-28T15-04-05.000.

  • max_log_size (default: 1024): Set the maximum size of a log file in MB. If the size of the log file exceeds the specified size, the log file will be archived and compressed and a new log file will be used.

  • max_age (default: 0): Set the maximum number of days an old log file is kept, based on the timestamp encoded in their filename. By default, old log files are not removed based on their age.

  • max_backups (default: 10) Specify the maximum number of old log files that are kept. If set to 0, the number of old log files to keep is not limited.

In the file_store section, you can change the following options to manage the File Store:

  • port (default: 8029/tcp): Specify the I/O port over which all files are transferred to and from the file store.

  • free_space_threshold: The minimum amount of free space on the disk containing the filestore (in bytes). If the actual free space falls below this threshold, filestore stops accepting uploads.

In the job_server section, change the following option:

  • schrodinger_hosts_hostname: Set to match the "host:" setting of the schrodinger.hosts file. Only the entries from the launch host's schrodinger.hosts file that match Job Server's entries will be available for job submission via this Job Server.

In the job_server section, change the following option:

  • use_local_auth (default: false): this field determines whether socket authentication is enabled for the Job Server. Normally set by jsc admin.

In the job_server section, set the following option:

  • installation_parent_directories: A list of directories in which Schrodinger installation subdirectories can be found.

 

The Job Server Host must have the same Schrödinger software version as the client machines that are submitting jobs. When a job is submitted to Job Server Host from a client machine, Job Server will search for a Schrodinger installation matching the release the client used to submit the job. Therefore, it is necessary to designate the locations of Schrödinger installations in the Job Server configuration file on the Job Server Host.

Multiple directories can be listed, and Job Server will search for Schrodinger installation subdirectories in those directories.

For example, if the following Schrödinger suites have been installed:

/nfs/applications/schrodinger2024-3
/nfs/applications/schrodinger2024-4
/shared/app/schrodinger/suite2025-1
/shared/app/schrodinger/suite2025-2
/shared/app/schrodinger/suite2025-3

then Job Server would be able to find all of them with the following configuration:

job_server:
 installation_parent_directories:
 - /nfs/applications
 - /shared/app/schrodinger

jsc admin Tasks

Job Server administrators can revoke user certificates on the Server Host, where the jobserverd process is running.

$SCHRODINGER/jsc admin revoke <user>

Updating and even deleting rows in a table creates new rows, leaving old rows written on the disk. These old rows waste space on the disk, and are usually reclaimed by the AUTOVACUUM daemon for use in future INSERT/UPDATE/DELETE operations. But, the total disk size of the table remains the same, and rows may be scattered across the total size.

Occasionally, periods of high INSERT/UPDATE/DELETE activity will cause more rows to be written than the AUTOVACUUM daemon can keep up with, and result in a lot of extra space (what we call bloat).

Large tables in the Job Database with bloat may cause significant slowdowns in standard jobserverd upgrade procedures. You can use the check-bloat command to run diagnostics on various tables in the Job Database. To reduce table bloat, we recommend running vacuum-fullⓘ command.

  1. Run the following command on either a running or stopped jobserverd instance: 

    $SCHRODINGER/jsc admin check-bloat -jobserver_dir <jobserver_dir> -table job

    Replace <jobserver_dir> with your own Job Server directory. Diagnostic measurements of the “job” table are displayed after running this command. The "job" table is commonly the one containing the most bloat.

    You can display measurements of other tables using the -table option. To see these and and for more information on the check-bloat command, display documentation using:

    $SCHRODINGER/jsc admin check-bloat -h

  2. Record the measurements displayed by the command in the previous step.

  3. Before running the vacuum-full command, stop the jobserverd service to reduce load on the server and speed up the operation:

    sudo systemctl stop jobserverd

  4. Run vacuum-fullⓘ on selected tables using:

    $SCHRODINGER/jsc admin vacuum-full -jobserver-dir <jobserver_dir> -table job

  5. Run the command in Step 1 again and compare the measurements. In most cases, there should be a decrease in the values, particularly the Table Size and Bloat Percent.

  6. Start the jobserverd service again, and ensure that the service started up correctly:

    sudo systemctl start jobserverd
sudo systemctl status jobserverd