Test Permutations
Permutations allow you to create a ‘virtual’ test for each permutation of the values of one or more variables.
Contents
Overview
The permute_on test attribute defines a list of variables to permute over. These can come from any variable set, but they cannot be deferred variables.
permuted_test:
permute_on: [msg, person, date]
subtitle: "{{person}}-{{msg}}"
variables:
msg: ['hello', 'goodbye']
person: ['Paul', 'Nick']
run:
cmds: 'echo "{{msg}} {{person}} - {{date}}"'
The above would result in four virtual tests, each one echoing a different message.
That’s 2 users * 2 people * 1 date
echo "hello Paul - 07/14/19"
echo "hello Nick - 07/14/19"
echo "goodbye Paul - 07/14/19"
echo "goodbye Nick - 07/14/19"
The tests are scheduled independently when using
pav run
.The subtitle attribute allows for adding a permutation based value to the test name. If it is not defined, all permutations of a test will have identical names, making it difficult to tell them apart.
The subtitle attribute also allows for running specific permutations. For example, you can run
pav run suite.permuted_test.Paul-hello
, which will only run the permutation with that subtitle. You can also use standard filename globs, such as ‘suite.permuted_test.Paul-*’, which will run all the ‘Paul-’ permutations.
Limitations
You can’t permute on ‘sched’ variables. They don’t exist until after permutations are generated.
You can’t permute on Deferred variables. They can only have one value, and we won’t know what that is until right before the test runs.
No attempt is made to remove duplicate tests, so if you permute on a variable you don’t use it will create some identical test runs.
Complex Variables in Permutations
Complex variables are a useful way to group variables together in a permutation.
mytest:
permute_on: compiler
variables:
compiler:
- {name: 'gcc', mpi: 'openmpi', cmd: 'mpicc', openmp: '-fopenmp'}
- {name: 'intel', mpi: 'intel-mpi', cmd: 'mpiicc', openmp: '-qopenmp'}
subtitle: '{{compiler.name}}'
build:
# Will result in `mpiicc -qopenmp mysrc.c`
cmds: '{{compiler.cmd}} {{compiler.openmp}} mysrc.c'
...
This would create two virtual tests, one built with gcc and one with
intel. - The subtitle
test attribute lets us give each a specific
name. In this case mytest.gcc
and mytest.intel
. - Note that
using a variable multiple times never creates additional
permutations.
Permutations vs Combinations
Are they ‘permutations’ or ‘combinations’? These words have very specific meaning in a mathematical sense, and our usage here can be a bit confusing. We are permuting over the combinations of multiple sets, and in neither case are we are we using the word in a purely combinatorial sense.