Test Permutations

Permutations allow you to create a ‘virtual’ test for each permutation of the values of one or more variables.

Overview

The permute_on test attribute defines a list of variables to permute over. These can come from any variable set, but they cannot be deferred variables.

permuted_test:
    permute_on: [msg, person, date]
    subtitle: "{{person}}-{{msg}}"

    variables:
      msg: ['hello', 'goodbye']
      person: ['Paul', 'Nick']
    run:
      cmds: 'echo "{{msg}} {{person}} - {{date}}"'

The above would result in four virtual tests, each one echoing a different message.

• That’s 2 users * 2 people * 1 date

  • echo "hello Paul - 07/14/19"

  • echo "hello Nick - 07/14/19"

  • echo "goodbye Paul - 07/14/19"

  • echo "goodbye Nick - 07/14/19"

• The tests are scheduled independently when using pav run.

• The subtitle attribute allows for adding a permutation-based value to the test name. If it is not defined, all permutations of a test will have identical names, making it difficult to tell them apart.

• The subtitle attribute also allows for running specific permutations. For example, you can run pav run suite.permuted_test.Paul-hello, which will only run the permutation with that subtitle. You can also use standard filename globs, such as suite.permuted_test.Paul-\*, which will run all the Paul- permutations.

Limitations

• You can’t permute on sched variables. They don’t exist until after permutations are generated.

• You can’t permute on deferred variables. They can only have one value, and we won’t know what that is until right before the test runs.

• No attempt is made to remove duplicate tests, so if you permute on a variable you don’t use it will create some identical test runs.

Complex Variables in Permutations

Complex variables are a useful way to group variables together in a permutation.

mytest:
    permute_on: compiler
    variables:
      compiler:
        - {name: 'gcc',   mpi: 'openmpi',   cmd: 'mpicc',  openmp: '-fopenmp'}
        - {name: 'intel', mpi: 'intel-mpi', cmd: 'mpiicc', openmp: '-qopenmp'}

    subtitle: '{{compiler.name}}'

    build:
      # Will result in `mpiicc -qopenmp mysrc.c`
      cmds: '{{compiler.cmd}} {{compiler.openmp}} mysrc.c'
    ...

This would create two virtual tests, one built with gcc and one with intel. The subtitle test attribute lets us give each a specific name. In this case mytest.gcc and mytest.intel. Note that using a variable multiple times never creates additional permutations.

Permutations vs Combinations

Are they ‘permutations’ or ‘combinations’? These words have very specific meaning in a mathematical sense, and our usage here can be a bit confusing. We are permuting over the combinations of multiple sets, and in neither case are we are we using the word in a purely combinatorial sense.