Writing a Unit Test¶
This tutorial is for developers writing unittests of the Pavilion code itself.
Contents
What You Need To Test¶
Here are some basic tenants:
- Test the end results, not intermediate results.
- Trust that Python works. Test what a setting/argument effects rather than the fact that the variable got set.
- Exercise each code path.
- Don’t test every combination of code path.
- When reasonable test that bad values are handled sanely.
- You can assume a reasonable type is always passed.
- Your test should be fast.
- If waiting for something, sleep in .1s increments.
- You can combine multiple ‘subtests’ into a single test method, but at least separate them by ‘theme’.
Unit Test Files¶
The unit test directory has the following structure.
- test/run_tests
- The command run the unittests. See Link to Running Unit Tests.
- test/tests
- Unit test python modules go here.
- test/data
- Unit test data goes here.
- test/data/pav_config_dir
- Tests generated by unittests are by default configured via this directory. If you want to add a working plugin, add it here.
- test/working_dir/
- Unit tests run using this as the working directory.
Unit test python modules must end in _test.py
and be in the test/tests
directory for run_tests
to find them.
Using the PavTestCase class¶
Each Pavilion Unit Test Suite is a class that must inherit from PavTestCase. This provides a variety of useful helper methods and data, enables selective test skipping, and consistently sets up Pavilion correctly for unittests. It inherits from the base unittest.TestCase object, and which provides all the general python unittest functionality.
from pavilion.unittest import PavTestCase
from pavilion import plugins
class MyTests(PavTestCase):
# This method is run before each test in this class. You can override it
# to do things that each test requires.
def setUp(self):
# One typical thing to do here is initialize the Pavilion plugin
# system. More on that later.
plugins.initialize_plugins(self.pav_config)
# This method is run after each test in this class.
def tearDown(self):
# If you initialize plugins before each test, you must also reset
# them afterwards.
plugins._reset_plugins()
# Each method that starts with 'test_' is a unittest.
def test_stuff()
# You should generally check your test results with the builtin
# .assert* methods. You don't have to give a custom msg, but it
# can often be helpful with debugging if what you're checking for
# isn't obvious.
self.assertEqual(2*2, 4,
msg="OMG the universe is broken.")
if 3**2 != 9:
# You can also do a hard fail with the fail method.
self.fail(msg="Oh noes!")
# You can make sure an exception is raised with assertRaises method.
# The cleanest format is to use it in a 'with' context.
with self.assertRaises(ValueError, msg="Not again!")
# This should always raise a ValueError.
# Our context will catch it.
c = int('abc')
# Finally, if the test returns, then the test passes.
Test Results¶
As mentioned a test PASSES if the test method returns without an exception.
A test FAILS if any of the .assert* methods don’t evaluate to True.
A test is an ERROR if any exception is raised. (Technically, the asserts will raise an exception too, but those are handled specially.)
Using Pavilion Components in Tests¶
Most components in Pavilion is designed to be used independently of the the others. The two big exceptions are the Pavilion configuration (which almost everything needs), and the plugin system (which anything that uses a plugin needs).
Pavilion Config¶
The PavTestCase object provides a pavilion config object as a instance variable. This has been specially configured for unit tests.
- Sets working_dir to
test/working_dir
- Sets the pavilion config dir to
test/data/pav_config_dir
. - Sets exception and result logs to point to our working_dir.
It’s accessible via self.pav_cfg
from within any test.
Always use this pavilion config anytime you create a Pavilion object that takes a Pavilion configuration as an argument.
Modifying self.pav_cfg
¶
If you ever need to modify self.pav_cfg
, do a deep copy of it first.
import copy
from pavilion import unittest
class MyTests(unittest.PavTestCase):
def test_more_stuff(self):
my_pav_cfg = copy.deepcopy(self.pav_cfg)
my_pav_cfg.config_dirs.append(self.TEST_DATA_ROOT/'pav_config_dir2')
Plugins¶
Your tests will probably need plugins, and may even need custom test plugins to work with. Any such test needs to initialize the plugin system and reset it when you’re done.
You can generally do this in the setUp()
and tearDown
methods. This
isn’t done by default, because quite a few tests don’t need it or need to do
this multiple times in a single test.
from pavilion.unittest import PavTestCase
from pavilion import plugins
class MyTests(PavTestCase):
# This method is run before each test in this class.
def setUp(self):
# Given the default Pavilion config, this will find all the plugins
# that come with Pavilion, and all the plugins in
# test/data/pav_config_dir/plugins
plugins.initialize_plugins(self.pav_cfg)
# This method is run after each test in this class.
def tearDown(self):
# Unload all of the plugins. Don't worry, the plugins are designed
# to be loaded/unloaded multiple times.
plugins._reset_plugins()
Our examples below all initialize plugins in the test method itself, but just for brevity.
Getting Plugins¶
Each plugin type in Pavilion provides a function to find a plugin by name (and sometimes additional information).
from pavilion.unittest import PavTestCase
from pavilion import plugins
import pavilion
class MyTests(PavTestCase):
def test_plugins(self):
plugins.initialize_plugins(self.pav_cfg)
run_cmd = pavilion.commands.get_plugin('run')
slurm = pavilion.schedulers.get_plugin('slurm')
regex_parser = pavilion.get_plugin('regex')
# System Variable Plugins simply provide values through the
# sys_vars dict.
sys_vars = pavilion.system_variables.get_vars(defer=True)
sys_vars['sys_name']
plugins._reset_plugins()
Test Run Objects¶
It’s very likely that your test will require one or more test run objects. Your PavTestCase can help with that via the _quick_test() and _quick_test_cfg() methods.
from pavilion.unittest import PavTestCase
from pavilion import plugins
import pavilion
class MyTests(PavTestCase):
def test_foo(self):
plugins.initialize_plugins(self.pav_cfg)
# This will create a test run object, along with its run directory.
# The test is essentially a 'hello world'.
test = self._quick_test()
test.run()
# If you need to modify the config, first get the default.
test_cfg = self._quick_test_cfg()
# Note that you're working with a raw config after it's been
# loaded and all 'magic' applied. So things that end up as lists
# should be given as lists, and you shouldn't use pavilion
# variables.
test_cfg['run']['cmds'] = ['sleep 5']
test2 = self._quick_test(cfg=test_cfg, build=False, finalize=False)
plugins._reset_plugins()
You must be cognizant of the
test lifecycle.
Before a test can be run, it must be built and finalized. The ._quick_test()
method does this for you by default, but it can be turned off through the
build
and finalize
options to ._quick_test()
.
Testing Commands¶
Commands need an argument object, which we can get from the pavilion argument
parser using the .get_parser()
method. The parser returned is just a
standard Python
argparse.ArgumentParser
object.
from pavilion import unittests
from pavilion import arguments
from pavilion import commands
import io
class LogTests(unittests.PavTestCase):
def test_log_cmd(self):
plugins.initialize_plugins(self.pav_config)
# To check the logs, we need a test to check the logs of.
test = self._quick_test()
test.run()
# Get the command itself.
log_cmd = commands.get_plugin('log')
# Instead of printing to stdout and stderr, we should capture the
# command output. Remember, we'll reload the plugins for each test,
# so this change won't be permanent.
log_cmd.outfile = io.StringIO()
log_cmd.errfile = io.StringIO()
# Get the argument parser.
arg_parser = arguments.get_parser()
# We can test a whole bunch of argument combinations at once by
# iterating over them.
arg_sets = (
['log', 'kickoff', str(test.id)],
['log', 'run', str(test.id)],
['log', 'build', str(test.id)],
)
for arg_set in arg_sets:
# Parse the arguments
args = arg_parser.parse_args(arg_set)
# Run the command with the given args.
log_cmd.run(self.pav_cfg, args)
# We could check that the output is sane here (in this case
# we can do so easily, so we should). The StringIO objects
# we used above would make that fairly easy.
plugins._reset_plugins()