Miscellaneous Modules

Various Pavilion support code modules.

LockFiles

Pavilion uses lock files to handle concurrency across multiple nodes and systems. It has to assume the file-system that these are written to has atomic, O_EXCL file creation.

class pavilion.lockfile.LockFile(lockfile_path, group=None, timeout=None, expires_after=3)

Bases: object

An NFS friendly way to create a lock file. Locks contain information on what host and user created the lock, and have a built in expiration date. To be used in a ‘with’ context.

In general, you should use a short expire period if possible and ‘.renew()’ the lock regularly while it’s in use for longer periods of time.

Variables:
  • DEFAULT_EXPIRE (int) – Time till file is considered stale, in seconds. (3 seconds by default)
  • SLEEP_PERIOD (int) – How long to sleep between lock attempts. This shouldn’t be any less than 0.01 or so on a regular filesystem. 0.2 is pretty reasonable for an nfs filesystem and sporadically used locks.
  • LOCK_PERMS (int) – Default lock permissions
DEFAULT_EXPIRE = 3
LOCK_PERMS = 508
SLEEP_PERIOD = 0.2
lock()

Try to create and lock the lockfile.

read_lockfile()

Returns the components of the lockfile content, or None for each of these values if there was an error..

Returns:host, user, expiration (as a float), id
renew()

Renew a lockfile that’s been acquired by touching the file. This rate limits the renewal to not be overly stressful on the filesystem.

unlock()

Delete the lockfile, thereby releasing the lock.

Raises:RuntimeError – When we can’t delete our own lockfile for some reason.

Script Composition

The script composer makes it easy to build up a script with a prescribed environ in a programmatic way.

It also handles translating our module specifications into specific actions to add to the script.

class pavilion.scriptcomposer.ScriptComposer(header=None)

Bases: object

Manages the building of bash scripts for Pavilion.

command(command)

Add a line unadulterated to the script lines.

Parameters:command (str) – String representing the whole command to add.
comment(comment)

Function for adding a comment to the script.

Parameters:comment (str) – Text to be put in comment without the leading ‘# ‘.
env_change(env_dict)

Function to take the environment variable change requested by the user and add the appropriate line to the script.

Parameters:env_dict (dict) – A dictionary (preferably an OrderedDict) of environment keys and values to set. A value of None will unset the variable.
module_change(module, sys_vars)

Take the module changes specified in the user config and add the appropriate lines to the script. This will parse the module name into various actions, find the appropriate module_wrapper plugin, and use that to get the lines to add to the script.

Parameters:
  • module (str) – Name of a module or a list thereof in the format used in the user config.
  • sys_vars – The pavilion system variable dictionary.
newline()

Function that just adds a newline to the script lines.

static parse_module(mod_line)

Parse a module specification into it’s components. These can come in one of three formats:

  1. ‘mod-name[/version]’ - Load the given module name and version
  2. ‘-mod-name[/version]’ - Unload the given module/version.
  3. ‘old_name[/old_vers]->mod-name[/version]’ - Swap the given old module for the new one.
Parameters:mod_line (str) – String provided by the user in the config.
Return type:(str, (str, str), (str, str))
Returns:action, (name, vers), (old_name, old_vers)
write(path: pathlib.Path)

Function to write the script out to file.

Return bool result:
 Returns either True for successfully writing the file or False otherwise.
exception pavilion.scriptcomposer.ScriptComposerError

Bases: RuntimeError

Class level exception during script composition.

class pavilion.scriptcomposer.ScriptHeader(shebang='#!/bin/bash')

Bases: object

Class to serve as a struct for the script header.

get_lines()

Function to retrieve a list of lines for the script header.

reset()

Function to reset the values of the internal variables back to None.

shebang

Function to return the value of the internal shell path variable.

Wget

exception pavilion.wget.WGetError

Bases: RuntimeError

pavilion.wget.get(pav_cfg, url, dest)

Download the file at the given url and store it at dest. If a file already exists at dest it will be overwritten (assuming we have the permissions to do so). Proxies are handled automatically based on pav_cfg settings. This is done atomically; the download is saved to an intermediate location and then moved. :param pav_cfg: The pavilion configuration object. :param str url: The url for the file to download. :param Path dest: The path to where the file will be stored.

pavilion.wget.head(pav_cfg, url)

Get the header information for the given url. :param pav_cfg: The pavilion configuration object :param str url: The url we need information on. :returns: The http headers for the given url. :rtype dict:

pavilion.wget.missing_libs()

You should call this before using the wget module functions, to ensure all the dependencies are available. :returns: A list of one or more missing libraries. It won’t necessarily catch them all in one pass. An empty list is good.

pavilion.wget.update(pav_cfg, url, dest)

Check if the file needs to be re-downloaded, and do so if necessary. This will create a ‘{dest}.info’ file in the same directory that will be used to check if updates are necessary. :param pav_cfg: The pavilion configuration object. :param str url: The url for the file to download. :param Path dest: The path to where we want to store the file.