Installing Pavilion
Installing Pavilion is mostly a matter of placing it’s source somewhere, providing its (few) dependencies, and creating a pavilion.yaml config file.
Contents
Requirements
Pavilion has very few dependencies and requirements:
Python 3.6 or newer
A writeable space on a filesystem shared across all hosts (to be tested) in each cluster. (Assuming you’re scheduling jobs across a cluster).
The path to this directory must be consistent across all cluster hosts.
It must support atomic file creation and appends of < 4kb.
Lmod or ‘environment modules’ is recommended.
Filesystems
Pavilion works by recursively running itself in different modes at different points in the testing process. This means certain paths, like the Pavilion source directory, working directory, and used config directories must have paths that are consistent across the nodes and front-ends of any given system.
Pavilion places all builds, test working spaces, and lockfiles in a
working directory specified in the pavilion configuration (defaults
to ~/.pavilion/
).
Atomic (O_EXCL) file creation is needed here for the creation of lock files.
Atomic small appends are needed for writing to the status file. Not having this has a small chance of resulting in corrupted test status files.
Both of these requirements are probably already satisfied by one or more of your cluster NFS partitions. Lustre filesystems are not recommended, mostly due to the type of load Pavilion presents to these.
Testing Filesystems
If you’re unsure if your shared filesystem is reliable, there’s a test for that in test/utils.
$ python3 lock_test.py --help
Result Log
The result log can be configured to write to an arbitrary filesystem. That filesystem should be shared and have consistent paths as well, as the log is written as the final on-node step in tests.
Recommended Directory Structure
We recommend the following directory structure for Pavilion.
<root>/pavilion/src
- Clone/install pavilion here.<root>/pavilion/config
- Your pavilion configuration files. See Configuring Pavilion.<root>/pavilion/working_dir
- Where test runs and builds will be written.
The <root>/pavilion/config
directory will contain all of your site
specific tests, plugins, and test source files. If you organize your tests in
separate repositories, you may check them out here, and simply link the
appropriate files into config/tests
and config/test_src
.
Alternatively, you can make the entire config directory, source tarballs and
all, its own repository.
Install
Pavilion installs are meant to be dropped into place as a complete directory that contains the source and any dependencies missing on your system. This generally starts with a git pull of the latest release of Pavilion.
git clone https://github.com/hpc/pavilion2.git
cd pavilion2
# You can also, optionally, check out a specific release version.
git checkout tags/v2.4
You can also simply download and extract the source.
Releases
You should probably pick the latest Pavilion release when installing Pavilion for a couple reasons.
While we try to maintain backwards compatibility as much as possible, the reality is that every release contains several major compatibility breaks both for test configurations and plugins. These are documented per-release in the RELEASE.txt file.
We run a large bevy of unit tests against every change in Pavilion, but each release is used in production before it is actually tagged. This often reveals bugs, regressions, and practical usage issues. We fix those issues, then tag the release. Bugfix releases are provided as needed.
Dependencies
Pavilion has a few dependencies, and most aren’t required. Pavilion was
designed and tested against fairly recent (as of 2022-01) versions of
these, but it’s likely that older, system provided versions may work
just as well. Conversely, the latest version should be fine as well. The
supported and tests versions for each are recorded in requirements.txt
.
yaml_config (required) - Used to define the test and pavilion configurations.
yapsy (required) - The basis for Pavilion’s plugin architecture.
lark (required) - Used for Pavilion string value and expression parsing.
matplotlib - Needed for the graph command.
requests - Used for automatic downloads of test source files. This feature is disabled in the absence of this library, and tests that use it will fail with an error. The remaining dependencies are needed by requests. The following are sub-dependencies of requests.
Installing Dependencies
Dependencies auto-install when you run the pav command for the first time. If you’ve checked out Pavilion as a git repo, it will even auto-update dependencies if they change.
Environment Modules
Pavilion uses the module
command to load modules for tests. It will work
with either lmod or the tcl based ‘environment modules’ systems. This is
generally only needed if your cluster/s have a complex software environment
that supports multiple compilers and conflicting builds of libraries.
It is assumed that the module environment is set up before you run Pavilion. If you need to set up this environment separately on allocations, use the ‘env_setup’ option in the Configuring Pavilion to add the commands to do so.