Test Configuration

Test File Format

Pavilion Test configurations, like the base Pavilion configuration, utilize the YamlConfig library to define the config structure. Because of the dynamic nature of test configs, there are a few extra complications this module handles that are documented below.

class pavilion.test_config.file_format.TestConfigLoader

Bases: yaml_config.loaders.YamlConfigLoader

This class describes a test section in a Pavilion config file. It is expected to be added to by various plugins.

Variables:ELEMENTS (list(yc.YamlConfig)) – Each YamlConfig instance in this list defines a key for the test config.
  • Each element must result in a string (which is why you see a lot of StrElem below), or a structure that contains only strings at the lowest layer.
    • So lists of dicts of strings are fine, etc.
    • yc.RegexElem also produces a string.
  • Everything should have a sensible default.
    • An empty config should be a valid test.
  • For bool values, accept [‘true’, ‘false’, ‘True’, ‘False’].
    • They should be checked with val.lower() == ‘true’, etc.
  • Every element must have a useful ‘help_text’.
classmethod add_result_parser_config(name, config_items)

Add the given list of config items as a result parser configuration named ‘name’. Throws errors for invalid configuraitons.

classmethod add_subsection(subsection)

Use this method to add additional sub-sections to the config.

Parameters:subsection (yc.ConfigElem) – A yaml config element to add. Keyed elements are expected, though any ConfigElem based instance (whose leave elements are StrElems) should work.
classmethod check_leaves(elem)

Make sure all of the config elements have a string element or equivalent as the final node.

Parameters:elem (yc.ConfigElement) –
classmethod remove_result_parser_config(name)

Remove the given result parser from the result parser configuration section.

Parameters:name (str) – The name of the parser to remove.
classmethod remove_subsection(subsection_name)

Remove a subsection from the config. This is really only for use in plugin deactivate methods.

class pavilion.test_config.file_format.CondCategoryElem(name=None, sub_elem=None, defaults=None, key_case='lower', **kwargs)

Bases: yaml_config.structures.CategoryElem

Allow any key. They’ll be validated later.

class pavilion.test_config.file_format.EnvCatElem(name=None, sub_elem=None, defaults=None, key_case='lower', **kwargs)

Bases: yaml_config.structures.CategoryElem

A category element that ensures environment variables retain their order.

type

alias of collections.OrderedDict

class pavilion.test_config.file_format.PathCategoryElem(name=None, sub_elem=None, defaults=None, key_case='lower', **kwargs)

Bases: yaml_config.structures.CategoryElem

This is for category elements that need a valid unix path regex.

exception pavilion.test_config.file_format.TestConfigError

Bases: ValueError

An exception specific to errors in configuration.

pavilion.test_config.file_format.TestSuiteLoader()

Create a new test suite loader instance. This is a function masquerading as a constructor because the class has to be defined dynamically after plugins have modified the test config.

class pavilion.test_config.file_format.VarCatElem(name=None, sub_elem=None, defaults=None, key_case='lower', **kwargs)

Bases: yaml_config.structures.CategoryElem

For describing how the variables section itself works.

Just like a regular category elem (any conforming key, but values must be the same type), but with some special magic when merging values.

Variables:_NAME_RE – Unlike normal categoryElem keys, these can have dashes.
merge(old, new)

Merge, but allow for special keys that change our merge behavior.

‘key?: value’
Allows values from lower levels in the config stack to override this one. The value is only used if no other value is given.
‘key+: value/s’
The values are appended to the list of whatever is given by lower levels of the config stack.
class pavilion.test_config.file_format.VarKeyCategoryElem(name=None, sub_elem=None, defaults=None, key_case='lower', **kwargs)

Bases: yaml_config.structures.CategoryElem

Allow Pavilion variable name like keys.

class pavilion.test_config.file_format.VariableElem(name=None, **kwargs)

Bases: yaml_config.structures.CategoryElem

This is for values in the ‘variables’ section of a test config.

A variable entry can be either a single string value or an arbitrary dictionary of strings. If we get a single value, we’ll return it instead of a dict. Pavilion’s variable handling code handles the normalization of these values.

normalize(value)

Normalize to either a dict of strings or just a string.

validate(value, partial=False)

Check for a single item and return it, otherwise return a dict.

Test Setup

Pavilion has to take a bunch of raw Suite/Test configurations, incorporate various Pavilion variables, resolve test inheritance and permutations, and finally produce a bunch of TestRun objects. These steps, and more, are all handled by the TestConfigResolver

class pavilion.test_config.resolver.TestConfigResolver(pav_cfg)

Bases: object

Converts raw test configurations into their final, fully resolved form.

DEFERRED_PREFIX = '!deferred!'
NOT_OVERRIDABLE = ['name', 'suite', 'suite_path', 'scheduler', 'base_name', 'host', 'modes']
_apply_override(test_cfg, key, value)

Set the given key to the given value in test_cfg.

Parameters:
  • test_cfg (dict) – The test configuration.
  • key ([str]) – A list of key components, like [`slurm', 'num_nodes']
  • value (str) – The value to assign. If this looks like a json structure, it will be decoded and treated as one.
_find_config(conf_type, conf_name)

Search all of the known configuration directories for a config of the given type and name.

Parameters:
  • conf_type (str) – ‘host’, ‘mode’, or ‘test’
  • conf_name (str) – The name of the config (without a file extension).
Return type:

Path
Returns:

The path to the first matching config found, or None if one wasn’t found.
apply_host(test_cfg, host)

Apply the host configuration to the given config.

apply_modes(test_cfg, modes)

Apply each of the mode files to the given test config. :param dict test_cfg: A raw test configuration. :param list modes: A list of mode names.

apply_overrides(test_cfg, overrides)

Apply overrides to this test.

Parameters:
  • test_cfg (dict) – The test configuration.
  • overrides (list) – A list of raw overrides in a.b.c=value form.
Raises:

(ValueError,KeyError)
build_variable_manager(raw_test_cfg)

Get all of the different kinds of Pavilion variables into a single variable set manager for this test.

Parameters:raw_test_cfg – A raw test configuration. It should be from before any variables are resolved.
Return type:variables.VariableSetManager
check_version_compatibility()

Returns a bool on if the test is compatible with the current version of pavilion.

find_all_tests()

Find all the tests within known config directories.

Returns:Returns a dictionary of suite names to an info dict.
Return type:dict(dict)

The returned data structure looks like:

suite_name -> {
    'path': Path to the suite file.
    'err': Error loading suite file.
    'supersedes': [superseded_suite_files]
    'tests': name -> {
            'conf': The full test config (inheritance resolved),
            'summary': Test summary string,
            'doc': Test doc string,
    }
load(tests: List[str], host: str = None, modes: List[str] = None, overrides: List[str] = None, output_file: IO[str] = None) → List[Tuple[dict, pavilion.test_config.variables.VariableSetManager]]

Load the given tests, updated with their host and mode files.

Parameters:
  • tests – A list of test names to load.
  • host – The host to load tests for. Defaults to the value of the ‘sys_name’ variable.
  • modes – A list of modes to load.
  • overrides – A dict of key:value pairs to apply as overrides.
  • output_file – Where to write status output.
Returns:

A list test_config dict and var_man tuples
Return type:

[(dict, VariableSetManager)]
load_raw_configs(tests, host, modes)

Get a list of raw test configs given a host, list of modes, and a list of tests. Each of these configs will be lightly modified with a few extra variables about their name, suite, and suite_file, as well as guaranteeing that they have ‘variables’ and ‘permutations’ sections.

Parameters:
  • tests (list) – A list (possibly empty) of tests to load. Each test can be either a ‘<test_suite>.<test_name>’, ‘<test_suite>’, or ‘<test_suite>.*’. A test suite by itself (or with a .*) get every test in a suite.
  • None) host (Union(str,) – The host the test is running on.
  • modes (list) – A list (possibly empty) of modes to layer onto the test.
Return type:

list(dict)
Returns:

A list of raw test_cfg dictionaries.
normalize_override_value(value)

Normalize a value to one compatible with Pavilion configs. It can be any structure of dicts and lists, as long as the leaf values are strings.

Parameters:value – The value to normalize.
Returns:A string or a structure of dicts/lists whose leaves are strings.
classmethod resolve_config(config, var_man)

Recursively resolve the variables in the value strings in the given configuration.

Deferred Variable Handling
When a config value references a deferred variable, it is left unresolved and prepended with the DEFERRED_PREFIX. To complete these, use resolve_deferred().
Parameters:
  • config (dict) – The config dict to resolve recursively.
  • var_man (variables.VariableSetManager) – A variable manager. ( Presumably a permutation of the base var_man)
Returns:

The resolved config,
classmethod resolve_deferred(config, var_man)

Resolve only those values prepended with the DEFERRED_PREFIX. All other values are presumed to be resolved already.

Parameters:
  • config (dict) – The configuration
  • var_man (variables.VariableSetManager) – The variable manager. This must not contain any deferred variables.
static resolve_inheritance(base_config, suite_cfg, suite_path)

Resolve inheritance between tests in a test suite. There’s potential for loops in the inheritance hierarchy, so we have to be careful of that.

Parameters:
  • base_config – Forms the ‘defaults’ for each test.
  • suite_cfg – The suite configuration, loaded from a suite file.
  • suite_path – The path to the suite file.
Returns:

A dictionary of test configs.
Return type:

dict(str,dict)
classmethod resolve_keys(base_dict, var_man, section_name, deferred_only=False) → dict

Some sections of the test config can have Pavilion Strings for keys. Resolve the keys of the given dict.

Parameters:
  • base_dict (dict[str,str]) – The dict whose keys need to be resolved.
  • var_man (variables.VariableSetManager) – The variable manager to use to resolve the keys.
  • section_name (str) – The name of this config section, for error reporting.
  • deferred_only (bool) – Resolve only deferred keys, otherwise mark deferred keys as deferred.
Returns:

A new dictionary with the updated keys.
static resolve_permutations(test_cfg, base_var_man)

Resolve permutations for all used permutation variables, returning a variable manager for each permuted version of the test config. We use this opportunity to populate the variable manager with most other variable types as well.

Parameters:
  • test_cfg (dict) – The raw test configuration dictionary.
  • base_var_man (variables.VariableSetManager) – The variables for this config (absent the scheduler variables).
Returns:

The modified configuration, and a list of variable set managers, one for each permutation.
Return type:

(dict, [variables.VariableSetManager])
Raises:

TestConfigError – When there are problems with variables or the permutations.
classmethod resolve_section_values(component, var_man, allow_deferred=False, deferred_only=False, key_parts=None)

Recursively resolve the given config component’s value strings using a variable manager.

Parameters:
  • component (Union[dict,list,str]) – The config component to resolve.
  • var_man – A variable manager. (Presumably a permutation of the base var_man)
  • allow_deferred (bool) – Allow deferred variables in this section.
  • deferred_only (bool) – Only resolve values prepended with the DEFERRED_PREFIX, and throw an error if such values can’t be resolved. If this is True deferred values aren’t allowed anywhere.
  • key_parts (Union[tuple[str],None]) – A list of the parts of the config key traversed to get to this point.
Returns:

The component, resolved.
Raises:

RuntimeError, TestConfigError
verify_version(comp_versions)

Ensures version was provided in the correct format, and returns the version as a list of digits.

verify_version_range()
classmethod was_deferred(val)

Return true if config item val was deferred when we tried to resolve the config.

Parameters:val (str) – The config value to check.
Return type:bool