Miscellaneous Modules

Various Pavilion support code modules.

LockFiles

Pavilion uses lock files to handle concurrency across multiple nodes and systems. It has to assume the file-system that these are written to has atomic, O_EXCL file creation.

class pavilion.lockfile.LockFile(lockfile_path: ~pathlib.Path, group: ~typing.Optional[str] = None, timeout: ~typing.Optional[float] = None, expires_after: int = 3, errfile: ~typing.TextIO = <_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>)

Bases: object

An NFS friendly way to create a lock file. Locks contain information on what host and user created the lock, and have a built in expiration date. To be used in a ‘with’ context.

In general, you should use a short expire period if possible and ‘.renew()’ the lock regularly while it’s in use for longer periods of time.

Variables

  • DEFAULT_EXPIRE (int) – Time till file is considered stale, in seconds. (3 seconds by default)

  • SLEEP_PERIOD (int) – How long to sleep between lock attempts. This shouldn’t be any less than 0.01 or so on a regular filesystem. 0.2 is pretty reasonable for an nfs filesystem and sporadically used locks.

  • LOCK_PERMS (int) – Default lock permissions

DEFAULT_EXPIRE = 3
LOCK_PERMS = 508
NOTIFY_TIMEOUT = 5
SLEEP_PERIOD = 0.2
lock()

Try to create and lock the lockfile.

read_lockfile()

Returns the components of the lockfile content, or None for each of these values if there was an error..

Returns

host, user, expiration (as a float), id

renew(rate_limit=False)

Renew a lockfile that’s been acquired by touching the file.

Parameters

rate_limit – Limit how frequently the lock can be renewed.

unlock()

Delete the lockfile, thereby releasing the lock.

Raises

RuntimeError – When we can’t delete our own lockfile for some reason.

class pavilion.lockfile.LockFilePoker(lockfile: LockFile)

Bases: object

This context creates a thread that regularly ‘pokes’ a lockfile to make sure it doesn’t expire.

Script Composition

The script composer makes it easy to build up a script with a prescribed environ in a programmatic way.

It also handles translating our module specifications into specific actions to add to the script.

class pavilion.scriptcomposer.ScriptComposer(header=None)

Bases: object

Manages the building of bash scripts for Pavilion.

command(command)

Add a line unadulterated to the script lines.

Parameters

command (str) – String representing the whole command to add.

comment(comment)

Function for adding a comment to the script.

Parameters

comment (str) – Text to be put in comment without the leading ‘# ‘.

env_change(env_dict)

Function to take the environment variable change requested by the user and add the appropriate line to the script.

Parameters

env_dict (dict) – A dictionary (preferably an OrderedDict) of environment keys and values to set. A value of None will unset the variable.

module_change(module, sys_vars, config_wrappers)

Take the module changes specified in the user config and add the appropriate lines to the script. This will parse the module name into various actions, find the appropriate module_wrapper plugin, and use that to get the lines to add to the script.

Parameters

  • module (str) – Name of a module or a list thereof in the format used in the user config.

  • sys_vars – The pavilion system variable dictionary.

  • config_wrappers – Moduler wrappers specified via config.

newline()

Function that just adds a newline to the script lines.

write(path: Path)

Function to write the script out to file.

Return bool result

Returns either True for successfully writing the file or False otherwise.

exception pavilion.scriptcomposer.ScriptComposerError

Bases: RuntimeError

Class level exception during script composition.

class pavilion.scriptcomposer.ScriptHeader(shebang='#!/bin/bash')

Bases: object

Class to serve as a struct for the script header.

get_lines()

Function to retrieve a list of lines for the script header.

reset()

Function to reset the values of the internal variables back to None.

property shebang

Function to return the value of the internal shell path variable.

Wget

pavilion.wget.ca_cert_path()

Try to get system ca certs, then fall back to certifi.

pavilion.wget.get(pav_cfg, url, dest)

Download the file at the given url and store it at dest. If a file already exists at dest it will be overwritten (assuming we have the permissions to do so). Proxies are handled automatically based on pav_cfg settings. This is done atomically; the download is saved to an intermediate location and then moved. :param pav_cfg: The pavilion configuration object. :param str url: The url for the file to download. :param Path dest: The path to where the file will be stored.

pavilion.wget.head(pav_cfg, url)

Get the header information for the given url. :param pav_cfg: The pavilion configuration object :param str url: The url we need information on. :returns: The http headers for the given url. :rtype dict:

pavilion.wget.missing_libs()

You should call this before using the wget module functions, to ensure all the dependencies are available. :returns: A list of one or more missing libraries. It won’t necessarily catch them all in one pass. An empty list is good.

pavilion.wget.update(pav_cfg, url, dest)

Check if the file needs to be re-downloaded, and do so if necessary. This will create a ‘{dest}.info’ file in the same directory that will be used to check if updates are necessary. :param pav_cfg: The pavilion configuration object. :param str url: The url for the file to download. :param Path dest: The path to where we want to store the file.