Test Configuration

Test File Format

Pavilion Test configurations, like the base Pavilion configuration, utilize the YamlConfig library to define the config structure. Because of the dynamic nature of test configs, there are a few extra complications this module handles that are documented below.

class pavilion.test_config.file_format.TestConfigLoader

Bases: yaml_config.loaders.YamlConfigLoader

This class describes a test section in a Pavilion config file. It is expected to be added to by various plugins.

ELEMENTS = [<yaml_config StrElem name>, <yaml_config StrElem suite>, <yaml_config StrElem suite_path>, <yaml_config StrElem host>, <yaml_config ListElem modes>, <yaml_config RegexElem inherits_from>, <yaml_config StrElem subtitle>, <yaml_config StrElem group>, <yaml_config RegexElem umask>, <yaml_config KeyedElem maintainer>, <yaml_config StrElem summary>, <yaml_config StrElem doc>, <yaml_config ListElem permute_on>, <yaml_config VarCatElem variables>, <yaml_config RegexElem scheduler>, <yaml_config CondCategoryElem only_if>, <yaml_config CondCategoryElem not_if>, <yaml_config StrElem compatible_pav_versions>, <yaml_config StrElem test_version>, <yaml_config KeyedElem spack>, <yaml_config KeyedElem build>, <yaml_config KeyedElem run>, <yaml_config EvalCategoryElem result_evaluate>, <yaml_config KeyedElem result_parse>]

Each YamlConfig instance in this list defines a key for the test config.

  • Each element must result in a string (which is why you see a lot of StrElem below), or a structure that contains only strings at the lowest layer.
    • So lists of dicts of strings are fine, etc.
    • yc.RegexElem also produces a string.
  • Everything should have a sensible default.
    • An empty config should be a valid test.
  • For bool values, accept [‘true’, ‘false’, ‘True’, ‘False’].
    • They should be checked with val.lower() == ‘true’, etc.
  • Every element must have a useful ‘help_text’.
classmethod add_result_parser_config(name, config_items)

Add the given list of config items as a result parser configuration named ‘name’. Throws errors for invalid configurations.

classmethod add_subsection(subsection)

Use this method to add additional sub-sections to the config.

Parameters:subsection (yc.ConfigElem) – A yaml config element to add. Keyed elements are expected, though any ConfigElem based instance (whose leave elements are StrElems) should work.
classmethod check_leaves(elem)

Make sure all of the config elements have a string element or equivalent as the final node.

Parameters:elem (yc.ConfigElement) –
classmethod remove_result_parser_config(name)

Remove the given result parser from the result parser configuration section.

Parameters:name (str) – The name of the parser to remove.
classmethod remove_subsection(subsection_name)

Remove a subsection from the config. This is really only for use in plugin deactivate methods.

class pavilion.test_config.file_format.CondCategoryElem(name=None, sub_elem=None, defaults=None, key_case='lower', **kwargs)

Bases: yaml_config.structures.CategoryElem

Allow any key. They’ll be validated later.

class pavilion.test_config.file_format.EnvCatElem(name=None, sub_elem=None, defaults=None, key_case='lower', **kwargs)

Bases: yaml_config.structures.CategoryElem

A category element that ensures environment variables retain their order.

type

alias of collections.OrderedDict

class pavilion.test_config.file_format.EvalCategoryElem(name=None, sub_elem=None, defaults=None, key_case='lower', **kwargs)

Bases: yaml_config.structures.CategoryElem

Allow keys that start with underscore. Lowercase only.

class pavilion.test_config.file_format.PathCategoryElem(name=None, sub_elem=None, defaults=None, key_case='lower', **kwargs)

Bases: yaml_config.structures.CategoryElem

This is for category elements that need a valid unix path regex.

class pavilion.test_config.file_format.ResultParserCatElem(name=None, sub_elem=None, defaults=None, key_case='lower', **kwargs)

Bases: yaml_config.structures.CategoryElem

class pavilion.test_config.file_format.TestCatElem(name=None, sub_elem=None, defaults=None, key_case='lower', **kwargs)

Bases: yaml_config.structures.CategoryElem

A category element that ensures order of Keyed Elems retain order.

type

alias of collections.OrderedDict

exception pavilion.test_config.file_format.TestConfigError

Bases: ValueError

An exception specific to errors in configuration.

pavilion.test_config.file_format.TestSuiteLoader()

Create a new test suite loader instance. This is a function masquerading as a constructor because the class has to be defined dynamically after plugins have modified the test config.

class pavilion.test_config.file_format.VarCatElem(name=None, sub_elem=None, defaults=None, key_case='lower', **kwargs)

Bases: yaml_config.structures.CategoryElem

For describing how the variables section itself works.

Just like a regular category elem (any conforming key, but values must be the same type), but with some special magic when merging values.

Variables:_NAME_RE – Unlike normal categoryElem keys, these can have dashes.
merge(old, new)

Merge, but allow for special keys that change our merge behavior.

‘key?: value’
Allows values from lower levels in the config stack to override this one. The value is only used if no other value is given.
‘key+: value/s’
The values are appended to the list of whatever is given by lower levels of the config stack.
class pavilion.test_config.file_format.VarKeyCategoryElem(name=None, sub_elem=None, defaults=None, key_case='lower', **kwargs)

Bases: yaml_config.structures.CategoryElem

Allow Pavilion variable name like keys.

class pavilion.test_config.file_format.VariableElem(name=None, **kwargs)

Bases: yaml_config.structures.CategoryElem

This is for values in the ‘variables’ section of a test config.

A variable entry can be either a single string value or an arbitrary dictionary of strings. If we get a single value, we’ll return it instead of a dict. Pavilion’s variable handling code handles the normalization of these values.

normalize(value)

Normalize to either a dict of strings or just a string.

validate(value, partial=False)

Check for a single item and return it, otherwise return a dict.

Test Setup

Pavilion has to take a bunch of raw Suite/Test configurations, incorporate various Pavilion variables, resolve test inheritance and permutations, and finally produce a bunch of TestRun objects. These steps, and more, are all handled by the TestConfigResolver

class pavilion.test_config.resolver.ProtoTest(config, var_man)

Bases: object

An simple object that holds the pair of a test config and its variable manager.

class pavilion.test_config.resolver.TestConfigResolver(pav_cfg)

Bases: object

Converts raw test configurations into their final, fully resolved form.

DEFERRED_PREFIX = '!deferred!'
NOT_OVERRIDABLE = ['name', 'suite', 'suite_path', 'scheduler', 'base_name', 'host', 'modes']
_apply_override(test_cfg, key, value)

Set the given key to the given value in test_cfg.

Parameters:
  • test_cfg (dict) – The test configuration.
  • key ([str]) – A list of key components, like [`slurm', 'num_nodes']
  • value (str) – The value to assign. If this looks like a json structure, it will be decoded and treated as one.
apply_host(test_cfg, host)

Apply the host configuration to the given config.

apply_modes(test_cfg, modes)

Apply each of the mode files to the given test config. :param dict test_cfg: A raw test configuration. :param list modes: A list of mode names.

apply_overrides(test_cfg, overrides)

Apply overrides to this test.

Parameters:
  • test_cfg (dict) – The test configuration.
  • overrides (list) – A list of raw overrides in a.b.c=value form.
Raises:

(ValueError,KeyError)
build_variable_manager(raw_test_cfg)

Get all of the different kinds of Pavilion variables into a single variable set manager for this test.

Parameters:raw_test_cfg – A raw test configuration. It should be from before any variables are resolved.
Return type:variables.VariableSetManager
check_version_compatibility(test_cfg)

Returns a bool on if the test is compatible with the current version of pavilion.

classmethod finalize(test_run, new_vars)

Finalize the given test run object with the given new variables.

find_all_configs(conf_type)

Find all configs (host/modes) within known config directories.

Returns:Returns a dictionary of suite names to an info dict.
Return type:dict(dict)

The returned data structure looks like:

config_name -> {
    'path': Path to the suite file.
    'config': Full config file, loaded as a dict.
    'status': Nothing if successful, 'Loading the config failes.'
              if TectConfigError.
    'error': Detailed error if applicable.
    }
find_all_tests()

Find all the tests within known config directories.

Returns:Returns a dictionary of suite names to an info dict.
Return type:dict(dict)

The returned data structure looks like:

suite_name -> {
    'path': Path to the suite file.
    'err': Error loading suite file.
    'supersedes': [superseded_suite_files]
    'tests': name -> {
            'conf': The full test config (inheritance resolved),
            'summary': Test summary string,
            'doc': Test doc string,
    }
find_config(conf_type, conf_name)

Search all of the known configuration directories for a config of the given type and name.

Parameters:
  • conf_type (str) – ‘host’, ‘mode’, or ‘test’
  • conf_name (str) – The name of the config (without a file extension).
Return type:

Path
Returns:

The path to the first matching config found, or None if one wasn’t found.
load(tests: List[str], host: str = None, modes: List[str] = None, overrides: List[str] = None, conditions=None, output_file: IO[str] = None) → List[pavilion.test_config.resolver.ProtoTest]

Load the given tests, updated with their host and mode files. Returns ‘ProtoTests’, a simple object with ‘config’ and ‘var_man’ attributes for each resolved test.

Parameters:
  • tests – A list of test names to load.
  • host – The host to load tests for. Defaults to the value of the ‘sys_name’ variable.
  • modes – A list of modes to load.
  • overrides – A dict of key:value pairs to apply as overrides.
  • conditions – A dict containing the only_if and not_if conditions.
  • output_file – Where to write status output.
load_raw_configs(tests, host, modes)

Get a list of raw test configs given a host, list of modes, and a list of tests. Each of these configs will be lightly modified with a few extra variables about their name, suite, and suite_file, as well as guaranteeing that they have ‘variables’ and ‘permutations’ sections.

Parameters:
  • tests (list) – A list (possibly empty) of tests to load. Each test can be either a ‘<test_suite>.<test_name>’, ‘<test_suite>’, or ‘<test_suite>.*’. A test suite by itself (or with a .*) get every test in a suite.
  • None) host (Union(str,) – The host the test is running on.
  • modes (list) – A list (possibly empty) of modes to layer onto the test.
Return type:

list(dict)
Returns:

A list of raw test_cfg dictionaries.
normalize_override_value(value)

Normalize a value to one compatible with Pavilion configs. It can be any structure of dicts and lists, as long as the leaf values are strings.

Parameters:value – The value to normalize.
Returns:A string or a structure of dicts/lists whose leaves are strings.
resolve_cmd_inheritance(test_cfg)

Extend the command list by adding any prepend or append commands, then clear those sections so they don’t get added at additional levels of config merging.

classmethod resolve_deferred(config, var_man)

Resolve only those values prepended with the DEFERRED_PREFIX. All other values are presumed to be resolved already.

Parameters:
  • config (dict) – The configuration
  • var_man (variables.VariableSetManager) – The variable manager. This must not contain any deferred variables.
resolve_inheritance(base_config, suite_cfg, suite_path)

Resolve inheritance between tests in a test suite. There’s potential for loops in the inheritance hierarchy, so we have to be careful of that.

Parameters:
  • base_config – Forms the ‘defaults’ for each test.
  • suite_cfg – The suite configuration, loaded from a suite file.
  • suite_path – The path to the suite file.
Returns:

A dictionary of test configs.
Return type:

dict(str,dict)
classmethod resolve_keys(base_dict, var_man, section_name, deferred_only=False) → dict

Some sections of the test config can have Pavilion Strings for keys. Resolve the keys of the given dict.

Parameters:
  • base_dict (dict[str,str]) – The dict whose keys need to be resolved.
  • var_man (variables.VariableSetManager) – The variable manager to use to resolve the keys.
  • section_name (str) – The name of this config section, for error reporting.
  • deferred_only (bool) – Resolve only deferred keys, otherwise mark deferred keys as deferred.
Returns:

A new dictionary with the updated keys.
resolve_permutations(test_cfg, base_var_man)

Resolve permutations for all used permutation variables, returning a variable manager for each permuted version of the test config. We use this opportunity to populate the variable manager with most other variable types as well.

Parameters:
  • test_cfg (dict) – The raw test configuration dictionary.
  • base_var_man (variables.VariableSetManager) – The variables for this config (absent the scheduler variables).
Returns:

The modified configuration, and a list of variable set managers, one for each permutation.
Return type:

(dict, [variables.VariableSetManager])
Raises:

TestConfigError – When there are problems with variables or the permutations.
classmethod resolve_section_values(component, var_man, allow_deferred=False, deferred_only=False, key_parts=None)

Recursively resolve the given config component’s value strings using a variable manager.

Parameters:
  • component (Union[dict,list,str]) – The config component to resolve.
  • var_man – A variable manager. (Presumably a permutation of the base var_man)
  • allow_deferred (bool) – Allow deferred variables in this section.
  • deferred_only (bool) – Only resolve values prepended with the DEFERRED_PREFIX, and throw an error if such values can’t be resolved. If this is True deferred values aren’t allowed anywhere.
  • key_parts (Union[tuple[str],None]) – A list of the parts of the config key traversed to get to this point.
Returns:

The component, resolved.
Raises:

RuntimeError, TestConfigError
classmethod resolve_test_vars(config, var_man)

Recursively resolve the variables in the value strings in the given configuration.

Deferred Variable Handling
When a config value references a deferred variable, it is left unresolved and prepended with the DEFERRED_PREFIX. To complete these, use resolve_deferred().
Parameters:
  • config (dict) – The config dict to resolve recursively.
  • var_man (variables.VariableSetManager) – A variable manager. ( Presumably a permutation of the base var_man)
Returns:

The resolved config,
verify_version(version_str, comp_versions)

Ensures version was provided in the correct format, and returns the version as a list of digits.

verify_version_range(comp_versions)

Validate a version range value.

classmethod was_deferred(val)

Return true if config item val was deferred when we tried to resolve the config.

Parameters:val (str) – The config value to check.
Return type:bool