Miscellaneous Modules

Various Pavilion support code modules.

LockFiles

Pavilion uses lock files to handle concurrency across multiple nodes and systems. It has to assume the file-system that these are written to has atomic, O_EXCL file creation.

class pavilion.lockfile.LockFile(lockfile_path, group=None, timeout=None, expires_after=18000)

Bases: object

An NFS friendly way to create a lock file. Locks contain information on what host and user created the lock, and have a built in expiration date. To be used in a ‘with’ context.

Variables:
  • DEFAULT_EXPIRE (int) – Time till file is considered stale, in seconds. (5 minute default)
  • SLEEP_PERIOD (int) – How long to sleep between lock attempts. This shouldn’t be any less than 0.01 or so on a regular filesystem. 0.2 is pretty reasonable for an nfs filesystem and sporadically used locks.
  • LOCK_PERMS (int) – Default lock permissions
DEFAULT_EXPIRE = 18000
LOCK_PERMS = 508
SLEEP_PERIOD = 0.2
lock()

Try to create and lock the lockfile.

read_lockfile()

Returns the components of the lockfile content, or None for each of these values if there was an error..

Returns:host, user, expiration (as a float), id
unlock()

Delete the lockfile, thereby releasing the lock.

Raises:RuntimeError – When we can’t delete our own lockfile for some reason.

Script Composition

The script composer makes it easy to build up a script with a prescribed environ in a programmatic way.

It also handles translating our module specifications into specific actions to add to the script.

class pavilion.scriptcomposer.ScriptComposer(header=None, details=None)

Bases: object

Manages the building of bash scripts for Pavilion.

command(command)

Add a line unadulterated to the script lines.

Parameters:command (str) – String representing the whole command to add.
comment(comment)

Function for adding a comment to the script.

Parameters:comment (str) – Text to be put in comment without the leading ‘# ‘.
env_change(env_dict)

Function to take the environment variable change requested by the user and add the appropriate line to the script.

Parameters:env_dict (dict) – A dictionary (preferably an OrderedDict) of environment keys and values to set. A value of None will unset the variable.
module_change(module, sys_vars)

Take the module changes specified in the user config and add the appropriate lines to the script. This will parse the module name into various actions, find the appropriate module_wrapper plugin, and use that to get the lines to add to the script.

Parameters:
  • module (str) – Name of a module or a list thereof in the format used in the user config.
  • sys_vars (dict) – The pavilion system variable dictionary.
newline()

Function that just adds a newline to the script lines.

static parse_module(mod_line)

Parse a module specification into it’s components. These can come in one of three formats:

  1. ‘mod-name[/version]’ - Load the given module name and version
  2. ‘-mod-name[/version]’ - Unload the given module/version.
  3. ‘old_name[/old_vers]->mod-name[/version]’ - Swap the given old module for the new one.
Parameters:mod_line (str) – String provided by the user in the config.
Return type:(str, (str, str), (str, str))
Returns:action, (name, vers), (old_name, old_vers)
write()

Function to write the script out to file.

Return bool result:
 Returns either True for successfully writing the file or False otherwise.
exception pavilion.scriptcomposer.ScriptComposerError

Bases: RuntimeError

Class level exception during script composition.

class pavilion.scriptcomposer.ScriptDetails(path=None, group=None, perms=None)

Bases: object

Class to contain the final details of the script.

group
path
perms
class pavilion.scriptcomposer.ScriptHeader(shell_path=None, scheduler_headers=None)

Bases: object

Class to serve as a struct for the script header.

get_lines()

Function to retrieve a list of lines for the script header.

reset()

Function to reset the values of the internal variables back to None.

scheduler_headers

Function to return the list of scheduler header lines.

shell_path

Function to return the value of the internal shell path variable.

Wget

exception pavilion.wget.WGetError

Bases: RuntimeError

pavilion.wget.get(pav_cfg, url, dest)

Download the file at the given url and store it at dest. If a file already exists at dest it will be overwritten (assuming we have the permissions to do so). Proxies are handled automatically based on pav_cfg settings. This is done atomically; the download is saved to an intermediate location and then moved. :param pav_cfg: The pavilion configuration object. :param str url: The url for the file to download. :param Path dest: The path to where the file will be stored.

pavilion.wget.head(pav_cfg, url)

Get the header information for the given url. :param pav_cfg: The pavilion configuration object :param str url: The url we need information on. :returns: The http headers for the given url. :rtype dict:

pavilion.wget.missing_libs()

You should call this before using the wget module functions, to ensure all the dependencies are available. :returns: A list of one or more missing libraries. It won’t necessarily catch them all in one pass. An empty list is good.

pavilion.wget.update(pav_cfg, url, dest)

Check if the file needs to be re-downloaded, and do so if necessary. This will careate a ‘{dest}.info’ file in the same directory that will be used to check if updates are necessary. :param pav_cfg: The pavilion configuration object. :param str url: The url for the file to download. :param Path dest: The path to where we want to store the file.