Suite Organization

This section details the organization of test suites.

Suite Directories

In addition to single-file test suites, Pavilion provides the option to organize more complex suites using a directory structure. Under this structure, each suite directory contains not only the suite config YAML file, but also any host, mode, and OS configs associated with the suite, along with any source code required by the suite’s tests. This provides a convenient way of collecting the test code and data in a single location. This is useful, for instance, if the user wishes to version control each test separately and use it as a Git submodule in a larger project.

Two Ways of Organizing Suites

All test suites reside in the suites directory under one of the user’s configuration directories. There are two ways of organizing these suites, depending on the user’s needs and the complexity of the suite:

1. A simple suite may consist of a single suite config, <suite_name>.yaml, placed directly in the suites directory. This method is useful for tests that do not require source code or other files.

2. Suites consisting of multiple files, including additional configs and source code, should be placed under a subdirectory of suites, suites/<suite_name>/. Arbitrary files may be placed in this subdirectory, but the subdirectory must contain at minimum a suite config named suite.yaml.

In both cases, the suite config follows the format described in the Test Format section. Note the difference in file name between the two organization methods: in the first, the name of the file is the name of the suite; in the second, the file name is the generic suite.yaml, and Pavilion derives the suite name from the file’s parent directory.

Deprecation Warning

Pavilion previously used the tests and test_src subdirectories to store suite configs and test source code. These directories are now deprecated in favor of the single suites directory, and support for them is planned for eventual removal. While the older organization structure continues to be supported, users should plan on reorganizing existing tests using the new structure.

Host, OS, and Mode Configs

When using the first organization method, host, OS, and mode configs must be placed under their respective subdirectories (hosts, os, or modes) under the user’s config directory and must be named according to their associated host, operating system, or mode.

When using the second, suite directory method of organization, suite-specific auxiliary configs may optionally be placed in the suite directory alongside the suite config itself. These configs may be used to define test-specific options and variables that vary by host or OS, without cluttering global host, OS, or mode configs with test-specific variables. These configs extend, rather than override, configs placed in the hosts, os, and modes directories.

This method is especially useful in combination with the submodule-based test structure described above, since it allows host-specific test options and variables to be version controlled along with the suite config while maintaining independence from global configs.

If used, these suite-specific configs must be named hosts.yaml, os.yaml, or modes.yaml, depending on their config type. Note that these file names are plural.

The directory method of organization allows multiple hosts, OSs, or modes to be specified in a single config file. Pavilion discovers host and OS config files automatically based on the detected host and OS, but specific hosts and OSs, as well as modes, can be selected on test invocation by passing the name of the host, OS, or mode to pav run using the appropriate flag (either -H, -o, or -m). The following is an example of a host file containing multiple host entries (OS and mode formats are analogous):

host1:
    scheduler: slurm
    slurm:
        partition: user
        qos: user
host2:
    scheduler: raw
    variables:
        foo: bar

Note that this format differs from the format used by configs located in the hosts, os, and modes directories; in this case, the config file consists of a series of key–value pairs, where the keys are the names of hosts (or OSs or modes) and the values are individual host configs. This is necessary even in configs containing only a single host, since configs derive their names from the top-level keys.