Advanced Features

This page is an overview of some of the advanced features of Pavilion, to give you a better idea of what it’s capable of. See Tutorial Contents: for full tutorials on using these and other features.

Mode Configs

Full Docs: Mode Configs

In addition to host config files, you can provide mode config files that you can apply to any test when you run it. They have the same format as the host configs, but multiple can be provided per test.

Unlike host configs, mode configs apply _last_ in the process, overriding values set be host configs and the test itself.

For example, the following mode file could be used to set a particular set of slurm vars:

schedule:
    account: tester
    reservation: post-dst

$ pav run -m tester -f post_dst_tests.txt

Test Series

A test series is a well defined group of tests that are designated to be run together. A series is created automatically whenever you use pav run, but you can also explicitly define series using a configs/series/<series_name>.yaml file. This allows you defined relationships and dependencies between the tests, among other things.

See Writing a Series File.

Advanced Test Configs

Test configs aren’t just static files. They can be re-shaped dynamically through Pavilion variable substitution, module file loads, and environment variables.

Variables

Full Docs: Pavilion Test Variables

Test configs can contain expressions within their config values that reference and manipulate variables.

These variables come from a variety of sources (this is also the resolution order):

  • The test config’s variables section (var)

  • System Plugins (sys)

  • Pavilion hardcoded variables (pav)

  • The selected scheduler (sched)

Variable names must be in lowercase and start with a letter, but may contain number and underscores.

mytest:
  scheduler: slurm

  variables:
    sleep_time: 24

  run:
    cmds:
      - "sleep {{var.sleep_time + 12}}"
      - 'echo "Slept {{sleep_time + 12}} seconds on node
         {{sched.node_num}}."'

  • Use double curly brackets {{var.myvar}}.

  • Variable category is optional. {{myvar}} is fine.

  • Name conflicts are resolved in the order of categories listed above.

  • In fact, it’s recommended to not use the category component unless you need to make the reference explicit.

  • You’ll also see {{myvar.2}} list references, {{myvar.foo}} attribute references, and the combination of the two {{myvar.1.bar}}.

Listing Variables

Use the pav show commands to display what variables are available from various sources.

# pav show sched --vars slurm
# pav show pav_vars

pav show sys_vars

 Available System Variables
-----------+-------------------------------------+---------------------------------------------
 Name      | Value                               | Description
-----------+-------------------------------------+---------------------------------------------
 host_arch | <deferred>                          | The current host's architecture.
 host_name | <deferred>                          | The target host's hostname.
 host_os   | <deferred>                          | The target host's OS info (name, version).
 sys_arch  | x86_64                              | The system architecture.
 sys_host  | myhost                              | The system (kickoff) hostname.
 sys_name  | myhost                              | The system name (not necessarily hostname).
 sys_os    | {'name': 'sles', 'version': '12.3'} | The system os info (name, version).

Deferred Variables

Deferred variables are those that can’t be resolved at test kickoff time. They need to know something about the nodes the test is being started on (which we won’t know till the scheduler gives us nodes), or something about the allocation.

Because some parts of the test are resolved at kickoff time (on a front-end) rather than on the nodes, deferred variables aren’t allowed in those sections. Namely, this includes the build and various scheduler config sections, as well as root level config values. Pavilion will tell you when you make this mistake.

Expressions

Full Docs: Mathematical Expressions

The double curly brace sections that can contain variables are really fully capable Mathematical Expressions, and can contain math operations and function calls. Functions are provided via plugins.

mytest:
  variables:
    sleep_time: 24

  run:
    cmds:
      - "sleep {{ max([var.sleep_time/4, 1, sleep_time + 1]) }}"

Inheritance

Full Docs: Inheritance

Tests within a single test suite file can inherit from each other.

test_a:
    variables:
        key1: "apple"
        key2: "pear"

    run:
        cmds: 'echo "{{key1}} {{key2}}"'

test_b:
    inherits_from: test_a

    variables:
        key2: "banana"

The first test, ‘test_a’, would echo “apple pear”, while the second would echo “apple banana”.

Rules of Inheritance

  1. Every field in a test config can be inherited (except for inherits_from).

  2. A field that takes a list (modules, cmds, etc.) are always completely overwritten by a new list. (In the above example, the single command in the fs test command list overwrites the entire original command list.)

  3. A test can inherit from a test, which inherits from a test, and so on.

  4. Inheritance is resolved before permutations or any variable substitutions.

Permutations

Let’s say you want to create ten mostly identical tests, but each test takes slightly different input. In Pavilion, you can assign those different input values to a variable, and then create test ‘permutations’ over those values. Each permutation of a test is an instance of that test where that variable takes on just one of the values from your variable.

nbodies:

    variables:
        bodies: [2, 3, 10, 1000, 10000, 100000]
    permute_on: bodies

    run:
        cmds:
            - "nbodies -n {{bodies}} -s 1000"

    build:
        ...
This will create six test configurations (and thus six test runs), one for each

of the values of bodies with run commands that look like:

  • nbodies -n 2 -s 1000

  • nbodies -n 3 -s 1000

  • nbodies -n 10 -s 1000

  • etc.

You also can permute over multiple variables at once, producing a test run for each possible permutation of values. See Test Permutations for more info.

Skip Conditions

Full Docs: Skip Conditions

The only_if and not_if sections of the test config allow users to specify the conditions under which a test should run. Tests are ‘SKIPPED’ unless each of their only_if conditions (and none if their not_if conditions) match. The conditions are key:value/s pairs; the key is a Pavilion variable, and the value/s are one or more items that the ‘resolved’ value of the Pavilion variable might match to.

test: # This test uses the directives only_if and not_if.
    only_if:
        # For this test to run, 'user' must be one of the values below.
        "{{user}}": ['calvin', 'paul', 'nick', 'francine']
    not_if:
        # For this test to run 'sys_arch' must not be x86_64
        "{{sys_arch}}": 'x86_64'
    run:
        cmds:
            - 'echo "Helloworld"'

Environment

Pavilion provides means to alter environment variables and load environment (or lmod) modules.

Environment Variables

Full Docs: Environment Variables

You can set environment variables in your test scripts using the ‘env’ section under both ‘run’ and ‘build’. This will cause the variables to be exported within the generated run or build script, where they can be used by commands run as part of that script. Note that environment variables are only usable in the cmds and env sections, as these are written directly into the build and run scripts.

python_test:
    run:
      env:
        # Unset the python path environment variable.
        PYTHONPATH:
        # Use a different python home
        PYTHONHOME: /home/mario/python_root/
        # Specify a python version
        PY_VERS: 3
      cmds:
        - python${PY_VERS} -c "print('hello world')"

This will result in a run script that looks like:

#!/bin/bash

unset PYTHONPATH
export PYTHONHOME=/home/mario/python_root
export PY_VERS=3

python${PY_VERS} -c "print ('hello world')"

Modules

Full Docs: Modules

You can have pavilion load module files automatically for each test or build. This assumes the modules (and module build combinations) are available on your system. If the test can’t load a module, the test will report a ENV_FAILED status and fail.

super_magic:
    scheduler: slurm
    build:
      modules:
        - gcc/7.4.0
        - openmpi
      cmds:
        - mpicc -o super_magic super_magic.c
    run:
      # This runs as a separate script from the build, so you
      # have to specify modules for both the build and run.
      modules:
        - gcc/7.4.0
        - openmpi
      cmds:
        - srun ./super_magic -a

Pavilion assumes everything is starting from a clean system state in regards to modules, which is essentially the environment you get by default when logging in. That state may include modules that you don’t want loaded, so Pavilion provides a means for removing and swapping modules as well.

super_magic:
    build:
      modules:
        # Swap the gcc module for the intel module.
        - 'gcc -> intel/18.0.3'
        # Remove the python module
        - '-python'
      ...

Module Wrappers

When tell pavilion to load/remove/swap modules, the code to do this is added to the test or build script automatically using Module Wrapper Plugins. The default module wrapper performs the module command, and then verifies that the module is actually loaded.

More complicated setups are possible by adding additional plugins that replace this default behaviour for particular modules or module versions. You could, for instance, wrap all your compiler modules to set a consistent compiler wrapper environment variable.

openmp_test:
    build:
      modules:
        # Normally intel-mpi would require that we use mpiicc to build.
        # In our case though, we use module_wrappers (not shown) to set the
        # $MPICC env variable consistently across different MPI modules.
        # We also set $OPENMP_FLAG to value, as it varies across compilers.
        - intel
        - intel-mpi
      cmds:
        - '$MPICC $OPENMP_FLAG -o openmp_test openmp_test.c

# This test will use the same command, but it will work thanks to our
# module wrapper plugins.
openmp_test2:
    inherits_from: openmp_test
    build:
      modules:
        - gcc
        - openmpi

Module wrappers are also useful for smoothing the differences between clusters that have distinct module setups. For instance, one might wrap the gcc module such that it loads normally on some systems, but it performs a module swap on an odd system that loads a different compiler by default. This can allow for a single, host-agnostic set of tests.

Spack Packages

Full Docs: Spack Packages

Pavilion can be configured to use Spack to build code or provide modules. This requires a working instance of Spack to be configured globally for Pavilion.

build:
    spack:
        install:
            - ember
        load:
            - gcc
run:
    spack:
        load:
            - ember

Schedulers

An HPC testing framework wouldn’t be complete without allowing you to schedule your tests. Most of the above example tests reference a scheduler, but don’t configure one. It’s time to rectify that.

super_magic:
    scheduler: slurm
    schedule:
      # We can ask for all the nodes that match our filter parameters
      nodes: all
      # We can also set a minimum number of nodes
      min_nodes: 4
      tasks_per_node: 3
      # Most additional parameters denote how to filter the nodes down to
      # just those you want.
      partition: test_partition
      reservation: testing
      qos: test

    build:
      modules: [gcc, openmpi]
      cmds:
        - mpicc -o super_magic super_magic.c

    run:
      modules: [gcc, openmpi]
      cmds:
        # Regardless of scheduler used, scheduler vars are in the 'sched'
        # category. This var generates an srun command based on the slurm args
        # given above. Assuming we got 50 nodes, it will look like:
        # srun -N 50 -n 150 ./supermagic -a
        # Note that this would run in an sbatch script within an allocation
        # that conforms to the rest of the slurm settings.
        - {sched.test_cmd} ./supermagic -a

  • See pav show sched --vars <sched_name> for a listing of what variables are available for a given scheduler.

  • See pav show sched --config for full scheduler configuration documentation.

An Automated Workflow

Pavilion is often run manually, but can be used to run tests automatically through cron jobs or CI.

A typical sequence in an automated setup looks like this:

# Run a script that adds Pavilion to the path, and sets PAV_CONFIG_DIR in the environment.
source activate.sh

# Run a collection file of tests.
# Ignore build and test generation errors (we'll find those later).
pav run -f my_collection --ignore-errors

# Wait for those tests to complete.
pav wait

# Get the results (--all-passed checks that all tests passed).
pav result --all-passed

The above assumes the script exits/fails if any command exits non-zero.