Test Configuration

Test File Format

Pavilion Test configurations, like the base Pavilion configuration, utilize the YamlConfig library to define the config structure. Because of the dynamic nature of test configs, there are a few extra complications this module handles that are documented below.

class pavilion.test_config.file_format.TestConfigLoader

Bases: yaml_config.loaders.YamlConfigLoader

This class describes a test section in a Pavilion config file. It is expected to be added to by various plugins.

Variables:ELEMENTS (list(yc.YamlConfig)) – Each YamlConfig instance in this list defines a key for the test config.
  • Each element must result in a string (which is why you see a lot of StrElem below), or a structure that contains only strings at the lowest layer.
    • So lists of dicts of strings are fine, etc.
    • yc.RegexElem also produces a string.
  • Everything should have a sensible default.
    • An empty config should be a valid test.
  • For bool values, accept [‘true’, ‘false’, ‘True’, ‘False’].
    • They should be checked with val.lower() == ‘true’, etc.
  • Every element must have a useful ‘help_text’.
classmethod add_result_parser_config(name, config_items)

Add the given list of config items as a result parser configuration named ‘name’. Throws errors for invalid configuraitons.

classmethod add_subsection(subsection)

Use this method to add additional sub-sections to the config.

Parameters:subsection (yc.ConfigElem) – A yaml config element to add. Keyed elements are expected, though any ConfigElem based instance (whose leave elements are StrElems) should work.
classmethod check_leaves(elem)

Make sure all of the config elements have a string element or equivalent as the final node.

Parameters:elem (yc.ConfigElement) –
classmethod remove_result_parser_config(name)

Remove the given result parser from the result parser configuration section.

Parameters:name (str) – The name of the parser to remove.
classmethod remove_subsection(subsection_name)

Remove a subsection from the config. This is really only for use in plugin deactivate methods.

class pavilion.test_config.file_format.EnvCatElem(name=None, sub_elem=None, defaults=None, key_case='lower', **kwargs)

Bases: yaml_config.structures.CategoryElem

A category element that ensures environment variables retain their order.

type

alias of collections.OrderedDict

exception pavilion.test_config.file_format.TestConfigError

Bases: ValueError

An exception specific to errors in configuration.

pavilion.test_config.file_format.TestSuiteLoader()

Create a new test suite loader instance. This is a function masquerading as a constructor because the class has to be defined dynamically after plugins have modified the test config.

class pavilion.test_config.file_format.VarCatElem(name=None, sub_elem=None, defaults=None, key_case='lower', **kwargs)

Bases: yaml_config.structures.CategoryElem

For describing how the variables section itself works.

Just like a regular category elem (any conforming key, but values must be the same type), but with some special magic when merging values.

Variables:_NAME_RE – Unlike normal categoryElem keys, these can have dashes.
merge(old, new)

Merge, but allow for special keys that change our merge behavior.

‘key?: value’
Allows values from lower levels in the config stack to override this one. The value is only used if no other value is given.
‘key+: value/s’
The values are appended to the list of whatever is given by lower levels of the config stack.
class pavilion.test_config.file_format.VariableElem(name=None, **kwargs)

Bases: yaml_config.structures.CategoryElem

This is for values in the ‘variables’ section of a test config.

A variable entry can be either a single string value or an arbitrary dictionary of strings. If we get a single value, we’ll return it instead of a dict. Pavilion’s variable handling code handles the normalization of these values.

normalize(value)

Normalize to either a dict of strings or just a string.

validate(value, partial=False)

Check for a single item and return it, otherwise return a dict.

Test Setup

Pavilion has to take a bunch of raw Suite/Test configurations, incorporate various Pavilion variables, resolve test inheritance and permutations, and finally produce a bunch of TestRun objects. These steps, and more, are all handled by functions in this module.

pavilion.test_config.setup._apply_overrides(test_cfg, overrides, _first_level=True)

Apply overrides recursively.

pavilion.test_config.setup._find_config(pav_cfg, conf_type, conf_name)

Search all of the known configuration directories for a config of the given type and name.

Parameters:
  • pav_cfg – The pavilion config data.
  • conf_type (str) – ‘host’, ‘mode’, or ‘test’
  • conf_name (str) – The name of the config (without a file extension).
Return type:

Path
Returns:

The path to the first matching config found, or None if one wasn’t found.
pavilion.test_config.setup.apply_overrides(test_cfg, overrides)

Apply overrides to this test.

Parameters:
  • test_cfg (dict) – The test configuration.
  • overrides (dict) – A dictionary of values to override in all configs. This occurs at the highest level, after inheritance is resolved.
pavilion.test_config.setup.find_all_tests(pav_cfg)

Find all the tests within known config directories.

Parameters:pav_cfg – The pavilion configuration.
Returns:Returns a dictionary of suite names to an info dict.
Return type:dict(dict)

The returned data structure looks like:

suite_name -> {
    'path': Path to the suite file.
    'err': Error loading suite file.
    'supersedes': [superseded_suite_files]
    'tests': name -> {
            'conf': The full test config (inheritance resolved),
            'summary': Test summary string,
            'doc': Test doc string,
    }
pavilion.test_config.setup.load_test_configs(pav_cfg, host, modes, tests)

Get a list of raw test configs given a host, list of modes, and a list of tests. Each of these configs will be lightly modified with a few extra variables about their name, suite, and suite_file, as well as guaranteeing that they have ‘variables’ and ‘permutations’ sections.

Parameters:
  • pav_cfg – The pavilion config data
  • None) host (Union(str,) – The host the test is running on.
  • modes (list) – A list (possibly empty) of modes to layer onto the test.
  • tests (list) – A list (possibly empty) of tests to load. Each test can be either a ‘<test_suite>.<test_name>’, ‘<test_suite>’, or ‘<test_suite>.*’. A test suite by itself (or with a .*) get every test in a suite.
Return type:

list(dict)
Returns:

A list of raw test_cfg dictionaries.
pavilion.test_config.setup.resolve_config(config, var_man, no_deferred_allowed)

Recursively resolve the variables in the value strings in the given configuration.

Deferred Variable Handling
When a config value references a deferred variable, it is left unresolved and prepended with the DEFERRED_PREFIX. To complete these, use resolve_deferred().
Parameters:
  • config (dict) – The config dict to resolve recursively.
  • var_man (variables.VariableSetManager) – A variable manager. ( Presumably a permutation of the base var_man)
  • no_deferred_allowed (list) – Do not allow deferred variables in sections of with these names.
Returns:

The resolved config,
pavilion.test_config.setup.resolve_deferred(config, var_man)

Resolve only those values prepended with the DEFERRED_PREFIX. All other values are presumed to be resolved already.

Parameters:
  • config (dict) – The configuration
  • var_man (variables.VariableSetManager) – The variable manager. The must not contain any deferred variables.
pavilion.test_config.setup.resolve_inheritance(base_config, suite_cfg, suite_path)

Resolve inheritance between tests in a test suite. There’s potential for loops in the inheritance hierarchy, so we have to be careful of that.

pavilion.test_config.setup.resolve_permutations(raw_test_cfg, pav_vars, sys_vars)

Resolve permutations for all used permutation variables, returning a variable manager for each permuted version of the test config. We use this opportunity to populate the variable manager with most other variable types as well.

Parameters:
  • raw_test_cfg (dict) – The raw test configuration dictionary.
  • pav_vars (dict) – The pavilion provided variable set.
  • pavilion.system_variables.SysVarDict) sys_vars (Union(dict,) – The system plugin provided variable set.
Returns:

The modified configuration, and a list of variable set managers, one for each permutation. These will already contain all the var, sys, pav, and (resolved) permutation (per) variable sets. The ‘sched’ variable set will have to be added later.
Return type:

(dict, [variables.VariableSetManager])
Raises:

TestConfigError – When there are problems with variables or the permutations.
pavilion.test_config.setup.resolve_section_vars(component, var_man, allow_deferred, deferred_only)
Recursively resolve the given config component’s variables, using a
variable manager.
Parameters:
  • component (dict) – The config component to resolve.
  • var_man – A variable manager. (Presumably a permutation of the base var_man)
  • allow_deferred (bool) – Do not allow deferred variables in this section.
  • deferred_only (bool) – Only resolve values prepended with the DEFERRED_PREFIX, and throw an error if such values can’t be resolved.
Returns:

The component, resolved.
pavilion.test_config.setup.was_deferred(val)

Return true if config item val was deferred when we tried to resolve the config.

Parameters:val (str) – The config value to check.
Return type:bool