Variable Components

Variable Inner Workings (test_config.variables)

This module contains functions and classes for building variable sets for string insertion.

There are three layers to every variable:

  • A list of variable values
  • A dictionary of sub-keys
  • The values of those sub keys. ie: [{key:value},…]

From the user perspective, however, all but the value itself is optional.

While variables are stored in this manner, these layers are automatically resolved in the trivial cases such as when there is only one element, or a single value instead of a set of key/value pairs.

There are expected to be multiple variable sets: plain variables (var), plugin provided via sys_vars (sys), core pavilion provided (pav), and scheduler provided (sched).

class pavilion.test_config.variables.DeferredVariable

The value for some variables may not be available until a test is actually running. Deferred variables act as a placeholder in such circumstances, and output an escape sequence when converted to a str.

__init__

Initialize self. See help(type(self)) for accurate signature.

get(index, sub_var)

Deferred variables should never have their value retrieved.

class pavilion.test_config.variables.VariableSetManager

Bases: object

This class manages the various sets of variables, provides complex key based lookups, manages conflict resolution, and so on. Anything that works with pavilion variables should do so through an instance of this class.

Usage:

var_man = VariableSetManager()
# pav_vars and sys_vars should be dictionary like objects
var_man.add_var_set('sys', sys_vars)
var_man.add_var_set('pav', pav_vars)

var_man['sys.sys_name']
var_man['sys_name']
VAR_SETS = ('var', 'sys', 'pav', 'sched')
add_var_set(name, value_dict)

Add a new variable set to this variable set manager. Variables in the set can then be retrieved by complex key.

Parameters:
  • name (str) – The name of the var set. Must be one of the reserved keys.
  • value_dict (Union(dict,collections.UserDict)) – A dictionary of values to populate the var set.
Returns:

None
Raises:

VariableError – On problems with the name or data.
as_dict()

Return the all variable sets as a single dictionary. This will be structured as the config data is expected to be received. (A dict of variables where the values are lists of either structs or string values).

Return type:dict
get_permutations(used_per_vars)

For every combination of permutation variables (that were used), return a new var_set manager that contains only a single value (possibly a complex one) for each permutation var, in every possible permutation.

Parameters:used_per_vars (set) – A list of permutation variable names that were used.
Returns:A list of permuted variable managers.
Return type:VariableSetManager
is_deferred(var_set, var, idx=None, sub_var=None)

Return whether the given variable is deferred. Fully specified variables (with idx and sub_var set) may be deferred specifically or in general at various levels.

Return type:bool
static key_as_dotted(key)

Turn a tuple based key reference back into a dotted string.

len(var_set, var)

Get the length of the given key.

Parameters:
  • var_set (str) – The var set to fetch from.
  • var (str) – The variable to fetch.
Return type:

int
Returns:

The number of items in the found ‘var_set.var’.
Raises:

KeyError – When the key has problems, or can’t be found.
classmethod load(path)

Load a saved variable set.

Parameters:path (pathlib.Path) – The variable file to load.
classmethod parse_key(key)

Parse the given complex key, and return a reasonable (var_set, var, index, sub_var) tuple.

Parameters:

key (Union(str,list,tuple)) – A 1-4 part key. These may either be given as a list/tuple of strings, or dot separated in a string. The components are var_set, var, index, and sub_var. Var is required, the rest are optional. Index is expected to be an integer, and var_set is expected to be a key category.
Raises:
  • KeyError – For bad keys.
  • TypeError – When the key isn’t a string.
Returns:

A (var_set, var, index, sub_var) tuple. Any component except var may be None.
resolve_key(key)

Resolve the given key using this known var sets. Unlike parse_key, the var_set returned will never be None, as the key must correspond to a found variable in a var_set. In case of conflicts, the var_set will be resolved in order.

Parameters:key (Union(str,list,tuple)) – A 1-4 part key. These may either be given as a list/tuple of strings, or dot separated in a string. The components are var_set, var, index, and sub_var. Var is required, the rest are optional. Index is expected to be an integer, and var_set is expected to be a key category.
Raises:KeyError – For bad keys, and when the var_set can’t be found.
Returns:A tuple of (var_set, var, index, sub_var), index and sub_var may be None.
resolve_references(parser)

Resolve all variable references that are within variable values defined in the ‘variables’ section of the test config.

Parameters:

parser (function) – The Parser function from string_parser. This must be passed to avoid import loops.
Raises:
  • TestConfigError – When reference loops are found.
  • KeyError – When an unknown variable is referenced.
save(path)

Save the variable set to the given stream as JSON.

Parameters:path (pathlib.Path) – The file path to write to.
set_deferred(var_set, var, idx=None, sub_var=None)

Set the given variable as deferred. Variables may be deferred as a whole, or as individual list or sub_var items.

Parameters:
  • var_set (str) – The var_set of the deferred var.
  • var (str) – The variable name.
  • None) idx (Union(int,) – The idx of the deferred var. If set to None, the variable is deferred for all indexes. Note that single valued variables have an index of zero.
  • None) sub_var (Union(str,) – The sub_variable name that is deferred.
undefer(new_vars, parser)

Get non-deferred values for all the deferred values in this variable set, leaving the non-deferred values intact.

Parameters:
  • new_vars (VariableSetManager) – A completely non-deferred variable set manager.
  • parser (function) – The parse function from string_parser.
class pavilion.test_config.variables.VariableSet(name, value_dict=None)

Bases: object

A set of of variables. Essentially a wrapper around a mapping of var names to VariableList objects.

get(var, index, sub_var)

Return the value of the var given the var name, index, and sub_var name.

set_value(var, index, sub_var, value)

Set the value at the given location to value.

class pavilion.test_config.variables.VariableList(values=None)

Bases: object

Wraps a list of SubVariable objects. Even variables with a single value end up as a list (of one).

get(index, sub_var)

Return the variable value at the given index and sub_var.

set_value(index, sub_var, value)

Set the value at the given location to value.

class pavilion.test_config.variables.SubVariable(value_pairs=None)

Bases: object

The final variable tier. Variables with no sub-var end up with a dict with a single None: value pair.

get(sub_var)

Gets the actual variable value.

set_value(sub_var, value)

Set the value at the given location to value.

exception pavilion.test_config.variables.VariableError(message, var_set=None, var=None, index=None, sub_var=None)

Bases: ValueError

This error should be thrown when processing variable data, and something goes wrong.

Variable String Parsing (test_config.string_parser)

The module contains the functions and classes needed for parsing strings and performing variable insertion in pavilion configurations.

While we don’t use a context free grammar for parsing, it is convenient for describing the format:

# pav strings can contain variable references, escapes, or sub strings.
PAV_STR  -> TEXT | TEXT VAR PAV_STR | TEXT ESC PAV_STR |
            TEXT SUB_STR PAV_STR
# Text is anything that doesn't start an escape, sub string, or variable
# reference. It may also be an empty string.
TEXT     -> [^\{[]*
# Variable references are a bracket surrounded variable name. These can
# consist of a var_set name, variable name, variable index, and sub-variable
# name. All but the variable name are optional.
VAR      -> {{VAR_KEY}} | {{VAR_KEY|DEFAULT}}
VAR_KEY  -> VAR_BASE | VAR_BASE.VAR_IDX | VAR_BASE.VAR_ID |
            VAR_BASE.VAR_IDX.VAR_ID
VAR_BASE -> VAR_ID | VAR_ID.VAR_ID
VAR_ID   -> [a-z][a-z0-9_-]+
VAR_IDX  -> [0-9]+
# The default value, if none was provided.
DEFAULT  -> [^{}]+

ESCAPE   -> \.
# A substring is a pav string surrounded by squared brackets. It's used to
# denote an area that can be copied as a unit when used with multi-valued
# variables.
SUB_STR  -> [~PAV_STR~] | [~PAV_STR~SEP]
# A separator to insert between each substring instance.
SEP      -> [^\]]+
exception pavilion.test_config.string_parser.ParseError(message, error_start, error_end)

Bases: pavilion.test_config.string_parser.ScanError

Error parsing the Pav string tokens

class pavilion.test_config.string_parser.PavString(tokens, is_substr=False)

Bases: pavilion.test_config.string_parser.Token

Provides a tokenized representation of a pavilion string, including summary information about variables used, etc. It is itself the root token of the parse tree.

get_substr_vars(var_man)

Get the variables we need to iterate over in all nested substrings.

Parameters:var_man (variables.VariableSetManager) –
resolve(var_man, _iter_vars=None)

Resolve all variables in this string to the values given in the variable managers.

Parameters:
  • var_man (variables.VariableSetManager) – Provides the variable values.
  • _iter_vars (dict) – Variables that are being iterated over in the resolution of a sub string.
Returns:

The string with all variables resolved, or None (if resolution is deferred).
Return type:

Union(str,None)
Raises:

ResolveError – Usually if deferred variables are not allowed
exception pavilion.test_config.string_parser.ResolveError

Bases: ValueError

Error resolving string variables.

exception pavilion.test_config.string_parser.ScanError(message, error_start, error_end)

Bases: ValueError

Error scanning and tokenizing a Pav string.

class pavilion.test_config.string_parser.SubStringEndToken(start, end, separator)

Bases: pavilion.test_config.string_parser.Token

The end of a sub string section.

resolve(var_man, **kwargs)

This should never be called.

class pavilion.test_config.string_parser.SubStringStartToken(start, end)

Bases: pavilion.test_config.string_parser.Token

The start of a sub string section.

resolve(var_man, **kwargs)

This should never be called.

class pavilion.test_config.string_parser.TextToken(text, start, end)

Bases: pavilion.test_config.string_parser.Token

A plaintext token.

resolve(var_man, **kwargs)

Resolve any variables in this token using the variable manager. :param var_man: A variable manager with the needed variables. :return: The resolved string.

class pavilion.test_config.string_parser.Token(start, end)

Bases: object

The base token class.

resolve(var_man, _iter_vars=None)

Resolve any variables in this token using the variable manager.

Parameters:
  • var_man (variables.VariableSetManager) – A variable manager with the needed variables.
  • _iter_vars (dict) – Variables that are being iterated over in the resolution of a sub string.
Returns:

The resolved string.
class pavilion.test_config.string_parser.VariableToken(var, start, end, default)

Bases: pavilion.test_config.string_parser.Token

A token representing a variable value.

resolve(var_man, iter_index=None)

Resolve any variables in this token using the variable manager.

Parameters:
  • var_man (variables.VariableSetManager) – The variable manager to use for resolution.
  • iter_index (int) – The index to force for this variable, when it’s being iterated over.
Returns:
pavilion.test_config.string_parser.parse(string)

Tokenize the given string and then parse it into a tree.

Return type:PavString
pavilion.test_config.string_parser.tokenize(string)

Tokenize the given string.

  • Plain text data will become TextTokens.
  • Variable references will become Variable Tokens.
  • Substrings will be marked as beginning and end SubString*Tokens.
  • Escaped characters become their own TextToken.
  • Detects Unmatched var brackets.

Variable Dict Class (var_dict)

This provides a dictionary class that can register functions to dynamically provide <function_name>:<return value> key:value pairs. The functions are lazily executed, and the results are cached.

class pavilion.var_dict.VarDict(name)

Bases: collections.UserDict

A dictionary for defining dynamic variables in Pavilion.

Usage: To add a variable, create a method and decorate it with either ‘@var_method’ or ‘@dfr_var_method()’. The method name will be the variable name, and the method will be called to resolve the variable value. Methods that start with ‘_’ are ignored.

get(key, default=None)

As per the dict class.

info(key)

Get an info dictionary about the given key.

items()

As per the dict class.

keys()

As per the dict class.

values()

As per the dict class.

pavilion.var_dict.normalize_value(value, level=0)

Normalize a value to one compatible with Pavilion variables. This means it must be a dict of strings, a list of strings, a list of dicts of strings, or just a string. Returns None on failure. :param value: The value to normalize. :param level: Controls what structures are allowed as this is called recursively.

pavilion.var_dict.var_method(func)

This decorator marks the given function as a scheduler variable. The function must take no arguments (other than self).

Pavilion Defined Variables (pav_vars)

This module provides the ‘pav’ variables, hardcoded into a VarDict object.

class pavilion.pav_vars.PavVars

Bases: pavilion.var_dict.VarDict

The pavilion provided variables. Note that these values are generated once, then reused.

day()

The current day of the month.

month()

The current month.

time()

An ‘HH:MM:SS.usec’ timestamp.

timestamp()

The current unix timestamp.

user()

The current user’s login name.

weekday()

The current weekday.

year()

The current year.

System Variable Plugins

System Variables provide a way for pavilion users to add additional variables for Pavilion tests to use. In particular, these are useful for gathering site-specific information for your tests.

class pavilion.system_variables.SysVarDict(defer=False, unique=False)

Bases: collections.UserDict

This dictionary based object provides lazy, cached lookups of all system variable values according to what system variable plugins are actually loaded. The values, once retrieved, are thus static for a given run of the pavilion command.

classmethod get_obj(name)

Return the corresponding object without invoking the .get method.

static help(key)

Return help information for the given key.

items()

As per dict.items()

keys()

As per dict.keys() (except we’re really listing the loaded plugins.)

values()

As per dict.values()

class pavilion.system_variables.SystemPlugin(name, description, priority=10, is_deferable=False, sub_keys=None)

Bases: yapsy.IPlugin.IPlugin

Each system variable plugin provides a key and value for the system variables dictionary. These are only evaluated if asked for, and generally only once.

NAME_VERS_RE = re.compile('^[a-zA-Z0-9_.-]+$')
PRIO_COMMON = 10
PRIO_CORE = 0
PRIO_USER = 20
activate()

Add this plugin to the system plugin list.

deactivate()

Remove this plugin from the system plugin list.

get(defer)

Get the value for this system variable.

Params bool defer:
 If the variable is deferable, return a DeferredVariable object instead.
exception pavilion.system_variables.SystemPluginError

Bases: RuntimeError

Error thrown when a system plugin encounters an error.

pavilion.system_variables.get_vars(defer)

Get the dictionary of system plugins.

Parameters:defer (bool) – Whether the deferable plugins should be deferred.
Return type:SysVarDict