Writing a Unit Test 

This tutorial is for developers writing unittests of the Pavilion code itself.

Contents

Writing a Unit Test

What You Need To Test 

Here are some basic tenets:

Test the end results, not intermediate results.
- Trust that Python works. Test what a setting/argument effects rather than the fact that the variable got set.
Exercise each code path.
- Don’t test every combination of code path.
When reasonable test that bad values are handled sanely.
- You can assume a reasonable type is always passed.
Your test should be fast.
- If waiting for something, sleep in .1s increments.
You can combine multiple ‘subtests’ into a single test method, but at least separate them by ‘theme’.

Unit Test Files 

The unit test directory has the following structure.

test/run_tests: The command run the unittests. See Link to Running Unit Tests.
test/tests: Unit test python modules go here.
test/data: Unit test data goes here.
test/data/pav_config_dir: Tests generated by unittests are by default configured via this directory. If you want to add a working plugin, add it here.
test/working_dir/: Unit tests run using this as the working directory.

Unit test python modules must end in _test.py and be in the test/tests directory for run_tests to find them.

Each Pavilion Unit Test Suite is a class that must inherit from PavTestCase. This provides a variety of useful helper methods and data, enables selective test skipping, and consistently sets up Pavilion correctly for unittests. It inherits from the base unittest.TestCase object, and which provides all the general python unittest functionality.

from pavilion.unittest import PavTestCase
from pavilion import plugins
from pavilion import commands

class MyTests(PavTestCase):

    # This method is run before each test in this class. You can override it
    # to do things that each test requires.
    def set_up(self):
        # The default verson of this initializes plugins
        plugins.initialize_plugins(self.pav_config)
        # It's also a good place to pre-load commands.
        commands.load('run')

    # This method is run after each test in this class.
    # By default it resets plugins.
    def tear_down(self):
        plugins._reset_plugins()

    # Each method that starts with 'test_' is a unittest.
    def test_stuff()

        # You should generally check your test results with the builtin
        # .assert* methods. You don't have to give a custom msg, but it
        # can often be helpful with debugging if what you're checking for
        # isn't obvious.
        self.assertEqual(2*2, 4,
                         msg="OMG the universe is broken.")

        if 3**2 != 9:
            # You can also do a hard fail with the fail method.
            self.fail(msg="Oh noes!")

        # You can make sure an exception is raised with assertRaises method.
        # The cleanest format is to use it in a 'with' context.
        with self.assertRaises(ValueError, msg="Not again!")
            # This should always raise a ValueError.
            # Our context will catch it.
            c = int('abc')

        # Finally, if the test returns, then the test passes.

Test Results 

As mentioned a test PASSES if the test method returns without an exception.

A test FAILS if any of the .assert* methods don’t evaluate to True.

A test is an ERROR if any exception is raised. (Technically, the asserts will raise an exception too, but those are handled specially.)

Using Pavilion Components in Tests 

Most components in Pavilion is designed to be used independently of the the others. The two big exceptions are the Pavilion configuration (which almost everything needs), and the plugin system (which anything that uses a plugin needs).

Pavilion Config 

The PavTestCase object provides a pavilion config object as a instance variable. This has been specially configured for unit tests.

Sets working_dir to test/working_dir
Sets the pavilion config dir to test/data/pav_config_dir.
Sets exception and result logs to point to our working_dir.

It’s accessible via self.pav_cfg from within any test.

Always use this pavilion config anytime you create a Pavilion object that takes a Pavilion configuration as an argument.

Modifying `self.pav_cfg`

If you ever need to modify self.pav_cfg, do a deep copy of it first.

import copy
from pavilion import unittest

class MyTests(unittest.PavTestCase):

    def test_more_stuff(self):

        my_pav_cfg = copy.deepcopy(self.pav_cfg)
        my_pav_cfg.config_dirs.append(self.TEST_DATA_ROOT/'pav_config_dir2')

Plugins 

Your tests will probably need plugins, and may even need custom test plugins to work with. Any such test needs to initialize the plugin system and reset it when you’re done.

By design, the PavTestCase object does both of these things for you in the default set_up() and tear_down() methods.

Getting Plugins 

Each plugin type in Pavilion provides a function to find a plugin by name (and sometimes additional information).

from pavilion.unittest import PavTestCase
from pavilion import plugins
import pavilion

class MyTests(PavTestCase):

    def test_plugins(self):
        run_cmd = pavilion.commands.get_plugin('run')
        slurm = pavilion.schedulers.get_plugin('slurm')
        regex_parser = pavilion.result_parsers.get_plugin('regex')

        # System Variable Plugins simply provide values through the
        # sys_vars dict.
        sys_vars = pavilion.system_variables.get_vars(defer=True)
        sys_vars['sys_name']

Test Run Objects 

It’s very likely that your test will require one or more test run objects. Your PavTestCase can help with that via the _quick_test() and _quick_test_cfg() methods.

from pavilion.unittest import PavTestCase
from pavilion import plugins
import pavilion

class MyTests(PavTestCase):
    def test_foo(self):
        # This will create a test run object, along with its run directory.

        # The default test is essentially a 'hello world'.
        test = self._quick_test()
        test.run()

        # If you need to modify the config, first get the default.
        test_cfg = self._quick_test_cfg()
        # Note that you're working with a raw config after it's been
        # loaded and all 'magic' applied. So things that end up as lists
        # should be given as lists, and you shouldn't use pavilion
        # variables.
        test_cfg['run']['cmds'] = ['sleep 5']
        test2 = self._quick_test(cfg=test_cfg, build=False, finalize=False)

You must be cognizant of the test lifecycle. Before a test can be run, it must be built and finalized. The ._quick_test() method does this for you by default, but it can be turned off through the build and finalize options to ._quick_test().

Testing Commands 

Commands need an argument object, which we can get from the pavilion argument parser using the .get_parser() method. The parser returned is just a standard Python argparse.ArgumentParser object.

HOWEVER - by default the argument parser only knows about commands that have already been loaded. A command is loaded when its plugin is found, or (for builtin commands) when it has been ‘gotten’ with commands.get_plugin or preloaded with commands.load.

from pavilion import unittests
from pavilion import arguments
from pavilion import commands
import io

class LogTests(unittests.PavTestCase):

    def test_log_cmd(self):

        # To check the logs, we need a test to check the logs of.
        test = self._quick_test()
        test.run()

        # Get the command itself.
        # Now that it's loaded, the argparse will know how to parse its arguments.
        log_cmd = commands.get_plugin('log')
        # Set the command's output streams to memory buffers
        log_cmd.silence()

        # Get the argument parser.
        arg_parser = arguments.get_parser()

        # We can test a whole bunch of argument combinations at once by
        # iterating over them.
        arg_sets = (
            ['log', 'kickoff', str(test.id)],
            ['log', 'run', str(test.id)],
            ['log', 'build', str(test.id)],
        )

        for arg_set in arg_sets:
            # Parse the arguments
            args = arg_parser.parse_args(arg_set)
            # Run the command with the given args.
            log_cmd.run(self.pav_cfg, args)

            # Get the contents of the output streams and clear them.
            out, err_out = log_cmd.clear_output()
            self.assertContains("foo", out)