Variable Components

Variable Inner Workings (test_config.variables)

This module contains functions and classes for building variable sets for string insertion.

There are three layers to every variable:

  • A list of variable values

  • A dictionary of sub-keys

  • The values of those sub keys. ie: [{key:value},…]

From the user perspective, however, all but the value itself is optional.

While variables are stored in this manner, these layers are automatically resolved in the trivial cases such as when there is only one element, or a single value instead of a set of key/value pairs.

There are expected to be multiple variable sets: plain variables (var), plugin provided via sys_vars (sys), core pavilion provided (pav), and scheduler provided (sched).

class pavilion.variables.DeferredVariable

The value for some variables may not be available until a test is actually running. Deferred variables act as a placeholder in such circumstances, and output an escape sequence when converted to a str.

__init__()
get(index, sub_var)

Deferred variables should never have their value retrieved.

class pavilion.variables.SubVariable(value_pairs=None)

Bases: object

The final variable tier. Variables with no sub-var end up with a dict with a single None: value pair.

get(sub_var)

Gets the actual variable value.

set_value(sub_var, value)

Set the value at the given location to value.

exception pavilion.variables.VariableError(message='', var_set=None, var=None, index=None, sub_var=None, prior_error=None)

Bases: PavilionError

This error should be thrown when processing variable data, and something goes wrong.

property msg

Just return msg. This exists to be overridden in order to allow for dynamically generated messages.

class pavilion.variables.VariableList(values=None)

Bases: object

Wraps a list of SubVariable objects. Even variables with a single value end up as a list (of one).

get(index, sub_var)

Return the variable value at the given index and sub_var.

set_value(index, sub_var, value)

Set the value at the given location to value.

class pavilion.variables.VariableSet(name, value_dict=None)

Bases: object

A set of of variables. Essentially a wrapper around a mapping of var names to VariableList objects.

get(var, index, sub_var)

Return the value of the var given the var name, index, and sub_var name.

set_value(var, index, sub_var, value)

Set the value at the given location to value.

class pavilion.variables.VariableSetManager

Bases: object

This class manages the various sets of variables, provides complex key based lookups, manages conflict resolution, and so on. Anything that works with pavilion variables should do so through an instance of this class.

Usage:

var_man = VariableSetManager()
# pav_vars and sys_vars should be dictionary like objects
var_man.add_var_set('sys', sys_vars)
var_man.add_var_set('pav', pav_vars)

var_man['sys.sys_name']
var_man['sys_name']
VAR_SETS = ('var', 'sys', 'pav', 'sched')
add_var_set(name, value_dict)

Add a new variable set to this variable set manager. Variables in the set can then be retrieved by complex key.

Parameters

  • name (str) – The name of the var set. Must be one of the reserved keys.

  • value_dict (Union(dict,collections.UserDict)) – A dictionary of values to populate the var set.

Returns

None

Raises

VariableError – On problems with the name or data.

any_deferred(key: Union[str, tuple]) → bool

Return whether any members of the given variable are deferred.

as_dict()

Return the all variable sets as a single dictionary. This will be structured as the config data is expected to be received. (A dict of variables where the values are lists of either structs or string values).

Return type

dict

get(key: str, default=None) → str

Get an item, or the provided default, as per dict.get.

get_permutations(used_per_vars: List[Tuple[str, str]]) → List[VariableSetManager]

For every combination of permutation variables (that were used), return a new var_set manager that contains only a single value (possibly a complex one) for each permutation var, in every possible permutation.

Parameters

used_per_vars (list[(str, str)]) – A set of permutation variable names that were used, as a tuple of (var_set, var_name).

Returns

A list of permuted variable managers.

is_deferred(key)

Return whether the given variable is deferred. Fully specified variables (with idx and sub_var set) may be deferred specifically or in general at various levels.

Return type

bool

static key_as_dotted(key)

Turn a tuple based key reference back into a dotted string.

len(var_set, var)

Get the length of the given key.

Parameters

  • var_set (str) – The var set to fetch from.

  • var (str) – The variable to fetch.

Return type

int

Returns

The number of items in the found ‘var_set.var’.

Raises

KeyError – When the key has problems, or can’t be found.

classmethod load(path)

Load a saved variable set.

Parameters

path (pathlib.Path) – The variable file to load.

classmethod parse_key(key)

Parse the given complex key, and return a reasonable (var_set, var, index, sub_var) tuple.

Parameters

key (Union(str,list,tuple)) – A 1-4 part key. These may either be given as a list/tuple of strings, or dot separated in a string. The components are var_set, var, index, and sub_var. Var is required, the rest are optional. Index is expected to be an integer, and var_set is expected to be a key category.

Raises

  • KeyError – For bad keys.

  • TypeError – When the key isn’t a string.

Returns

A (var_set, var, index, sub_var) tuple. Any component except var may be None.

resolve_key(key)

Resolve the given key using this known var sets. Unlike parse_key, the var_set returned will never be None, as the key must correspond to a found variable in a var_set. In case of conflicts, the var_set will be resolved in order.

Parameters

key (Union(str,list,tuple)) – A 1-4 part key. These may either be given as a list/tuple of strings, or dot separated in a string. The components are var_set, var, index, and sub_var. Var is required, the rest are optional. Index is expected to be an integer, and var_set is expected to be a key category.

Raises

KeyError – For bad keys, and when the var_set can’t be found.

Returns

A tuple of (var_set, var, index, sub_var), index and sub_var may be None.

resolve_references(partial=False, skip_deps: Optional[List] = None) → Tuple[List[str], List[str]]

Resolve all variable references that are within variable values defined in the ‘variables’ section of the test config.

Parameters

  • partial – If true, will ignore errors from variables that can’t resolve.

  • skip_deps – Variables (a list of variable names) consider unresolvable by default, which forces anything that depends on them to not resolve. This allows for delaying resolution of such variables (primarily so we can permute on them first).

Raises

VariableError – When reference loops are found.

Returns

The list of resolved variable names and a list of variable names that would have been resolved if not for dependencies in ‘skip_deps’.

save(path)

Save the variable set to the given stream as JSON.

Parameters

path (pathlib.Path) – The file path to write to.

set_deferred(var_set, var, idx=None, sub_var=None)

Set the given variable as deferred. Variables may be deferred as a whole, or as individual list or sub_var items.

Parameters

  • var_set (str) – The var_set of the deferred var.

  • var (str) – The variable name.

  • idx (Union(int, None)) – The idx of the deferred var. If set to None, the variable is deferred for all indexes. Note that single valued variables have an index of zero.

  • sub_var (Union(str, None)) – The sub_variable name that is deferred.

undefer(new_vars)

Get non-deferred values for all the deferred values in this variable set, leaving the non-deferred values intact.

Parameters

new_vars (VariableSetManager) – A completely non-deferred variable set manager.

Variable Dict Class (var_dict)

This provides a dictionary class that can register functions to dynamically provide <function_name>:<return value> key:value pairs. The functions are lazily executed, and the results are cached.

class pavilion.var_dict.VarDict(name, deferred=True)

Bases: UserDict

A dictionary for defining dynamic variables in Pavilion.

Usage: To add a variable, create a method and decorate it with either @var_method or @dfr_var_method(). The method name will be the variable name, and the method will be called to resolve the variable value. Methods that start with ‘_’ are ignored.

DEFER_ERRORS = False
add_errors(errors: List[str])

Add the given errors to the error list.

errors()

Return the list of retrieval errors encountered when using this var_dict. Key errors are not included.

get(key, default=None)

As per the dict class.

info(key)

Get an info dictionary about the given key.

items()

As per the dict class.

keys()

As per the dict class.

values()

As per the dict class.

pavilion.var_dict.dfr_var_method(*sub_keys)

This decorator marks the following function as a deferred variable. It can optionally be given sub_keys for the variable as positional arguments.

Parameters

sub_keys (list(str)) – The variable sub-keys.

pavilion.var_dict.normalize_value(value, level=0)

Normalize a value to one compatible with Pavilion variables. This means it must be a dict of strings, a list of strings, a list of dicts of strings, or just a string. Returns None on failure. :param value: The value to normalize. :param level: Controls what structures are allowed as this is called recursively.

pavilion.var_dict.var_method(func)

This decorator marks the given function as a scheduler variable. The function must take no arguments (other than self).

Pavilion Defined Variables (pav_vars)

This module provides the ‘pav’ variables, hardcoded into a VarDict object.

class pavilion.pavilion_variables.PavVars

Bases: VarDict

The pavilion provided variables. Note that these values are generated once, then reused.

day()

The current day of the month.

month()

The current month.

time()

An ‘HH:MM:SS.usec’ timestamp.

timestamp()

The current unix timestamp.

user()

The current user’s login name.

user_id()

The current user’s id.

version()

The current version of Pavilion.

weekday()

The current weekday.

year()

The current year.

System Variable Plugins

Builtin system variable plugins and utilities. These are loaded manually for speed.

pavilion.sys_vars.register_core_plugins()

Add all builtin plugins and activate them.