Dir Database

Table of Contents

Manage ‘id’ directories. The name of the directory is an integer, which essentially serves as a filesystem primary key.

class pavilion.dir_db.SelectItems(data, paths)
data

Alias for field number 0

paths

Alias for field number 1

pavilion.dir_db.create_id_dir(id_dir: pathlib.Path, group: str, umask: int) -> (<class 'int'>, <class 'pathlib.Path'>)

In the given directory, create the lowest numbered (positive integer) directory that doesn’t already exist.

Parameters:
  • id_dir – Path to the directory that contains these ‘id’ directories
  • group – The group owner for this path.
  • umask – The umask to apply to this path.
Returns:

The id and path to the created directory.
Raises:
  • OSError – on directory creation failure.
  • TimeoutError – If we couldn’t get the lock in time.
pavilion.dir_db.default_filter(_: pathlib.Path) → bool

Pass every path.

pavilion.dir_db.delete(id_dir: pathlib.Path, filter_func: Callable[[pathlib.Path], bool] = <function default_filter>, transform: Callable[[pathlib.Path], Any] = None, verbose: bool = False)

Delete all id directories in a given path that match the given filter. :param id_dir: The directory to iterate through. :param filter_func: A passed filter function, to be passed to select. :param transform: As per ‘select_from’ :param verbose: Verbose output. :return int count: The number of directories removed. :return list msgs: Any messages generated during removal.

pavilion.dir_db.index(id_dir: pathlib.Path, idx_name: str, transform: Callable[[pathlib.Path], Dict[str, Any]], complete_key: str = 'complete', refresh_period: int = 1, remove_missing: bool = False, verbose: IO[str] = None, fn_base: int = 10) → NewType.<locals>.new_type

Load and/or update an index of the given directory for the given transform, and return it. The returned index is a dictionary by id of the transformed data.

Parameters:
  • id_dir – The directory to index.
  • idx_name – The name of the index.
  • transform – A transformation function that produces a json compatible dictionary.
  • complete_key – The key in the transformed dictionary that marks a record as complete. If not given, the record is always assumed to be complete. Incomplete records are recompiled every time the index is updated (hopefully they will be complete eventually).
  • refresh_period – Only update the index if this much time (in seconds) has passed since the last update.
  • remove_missing – Remove items that no longer exist from the index.
  • fn_base – The integer base for dir_db.
pavilion.dir_db.make_id_path(base_path, id_) → pathlib.Path

Create the full path to an id directory given its base path and the id.

Parameters:
  • base_path (Path) – The path to where id directories are stored.
  • id (int) – The id number
Return type:

Path
pavilion.dir_db.paths_to_ids(paths: List[pathlib.Path]) → List[int]

Convert a list of list of dir_db paths to ids.

Parameters:paths – A list of id paths.
Raises:ValueError – For invalid paths
pavilion.dir_db.reset_pkey(id_dir: pathlib.Path) → None

Reset the the ‘next_id’ for the given directory by deleting the pkey file (‘next_id’) if present.

pavilion.dir_db.select(id_dir: pathlib.Path, filter_func: Callable[[Any], bool] = <function default_filter>, transform: Callable[[pathlib.Path], Any] = None, order_func: Callable[[Any], Any] = None, order_asc: bool = True, fn_base: int = 10, idx_complete_key: str = 'complete', use_index: Union[bool, str] = True, verbose: IO[str] = None, limit: int = None) -> (typing.List[typing.Any], typing.List[pathlib.Path])

Filter and order found paths in the id directory based on the filter and other parameters. If a transform is given, this will create an index of the data returned by the transform to hasten this process.

Parameters:
  • id_dir – The director
  • transform – Function to apply to each path before applying filters or ordering. The filter and order functions should expect the type returned by this.
  • filter_func – A function that takes a directory, and returns whether to include that directory. True -> include, False -> exclude
  • order_func – A function that returns a comparable value for sorting, as per the list.sort keys argument. Items for which this returns None are removed.
  • order_asc – Whether to sort in ascending or descending order.
  • use_index – The name of (and whether to use) an index. When this is the literal ‘True’, the index name is pulled from the transform function name. A string can also be given to manually specify the name.
  • idx_complete_key – The key used to identify directories as ‘complete’ for indexing purposes. Incomplete directories will be re-indexed until complete.
  • fn_base – Number base for file names. 10 by default, ensure dir name is a valid integer.
  • limit – The max items to return. None denotes return all.
  • verbose – A file like object to print status info to.
Returns:

A filtered, ordered list of transformed objects, and the list of untransformed paths.
pavilion.dir_db.select_from(paths: Iterable[pathlib.Path], filter_func: Callable[[Any], bool] = <function default_filter>, transform: Callable[[pathlib.Path], Any] = None, order_func: Callable[[Any], Any] = None, order_asc: bool = True, fn_base: int = 10, limit: int = None) -> (typing.List[typing.Any], typing.List[pathlib.Path])

Filter, order, and truncate the given paths based on the filter and other parameters.

Parameters:
  • paths – A list of paths to filter, order, and limit.
  • transform – Function to apply to each path before applying filters or ordering. The filter and order functions should expect the type returned by this.
  • filter_func – A function that takes a directory, and returns whether to include that directory. True -> include, False -> exclude
  • order_func – A function that returns a comparable value for sorting, as per the list.sort keys argument. Items for which this returns None are removed.
  • order_asc – Whether to sort in ascending or descending order.
  • fn_base – Number base for file names. 10 by default, ensure dir name is a valid integer.
  • limit – The max items to return. None denotes return all.
Returns:

A filtered, ordered list of transformed objects, and the list of untransformed paths.

Dir DB Filters

This module contains functions to generate filter functions for handing to dir_db commands.

pavilion.filters.add_common_filter_args(target: str, arg_parser: argparse.ArgumentParser, defaults: dict, sort_options: List[str])

Add common arguments for all filters.

Parameters:
  • target – The name of what is being filtered, to be inserted in documentation. Should be plural.
  • arg_parser – The argparser to add arguments to.
  • defaults – A dictionary of default values for all arguments.
  • sort_options – A list of possible sort options.
Returns:
pavilion.filters.add_series_filter_args(arg_parser: argparse.ArgumentParser, default_overrides: Dict[str, Any] = None, sort_functions: Dict[str, Callable] = None) → None

Add a common set of arguments for filtering series (those supported by make_series_filter below).

Arguments and defaults:

{}

Parameters:
  • arg_parser – The arg parser (or sub-parser) to add arguments to.
  • default_overrides – A dictionary of defaults to override.
  • sort_functions – A dict of sort-by names and sorting functions. The functions don’t matter at this point. If empty, sorting is disabled.
pavilion.filters.add_test_filter_args(arg_parser: argparse.ArgumentParser, default_overrides: Dict[str, Any] = None, sort_functions: Dict[str, Callable] = None) → None

Add a common set of arguments for filtering tests (those supported by make_test_run_filter below).

Arguments and defaults:

{}

Parameters:
  • arg_parser – The arg parser (or sub-parser) to add arguments to.
  • default_overrides – A dictionary of defaults to override.
  • sort_functions – A dict of sort-by names and sorting functions. The functions don’t matter at this point. If empty, sorting is disabled.
pavilion.filters.get_sort_opts(sort_name: str, choices: Dict[str, Callable[[Any], Any]]) -> (typing.Callable[[pathlib.Path], typing.Any], <class 'bool'>)

Return a sort function and the sort order.

Parameters:
  • sort_name – The name of the sort, possibly prepended with ‘-‘.
  • choices – A dictionary of sort order names and key functions (ala list.sort). Defaults to TEST_SORT_FUNCS
Returns:

A tuple of the sort function and ascending boolean
pavilion.filters.make_series_filter(user: str = None, sys_name: str = None, newer_than: datetime.datetime = None, older_than: datetime.datetime = None, complete: bool = False, incomplete: bool = False) → Callable[[Dict[str, Any]], bool]

Generate a filter for using with dir_db functions to filter series. This is expected to operate on series.SeriesInfo objects, so make sure to pass Series info as the dir_db transform function.

Parameters:
  • complete – Only accept series for which all tests are complete.
  • incomplete – Only accept series for which not all tests are complete.
  • newer_than – Only accept series created after this time.
  • older_than – Only accept series created before this time.
  • sys_name – Only accept series created on this system.
  • user – Only accept series created by this user.
pavilion.filters.make_test_run_filter(complete: bool = False, failed: bool = False, incomplete: bool = False, name: str = None, newer_than: float = None, older_than: float = None, passed: bool = False, result_error: bool = False, show_skipped: bool = False, sys_name: str = None, user: str = None)

Generate a filter function for use by dir_db.select and similar functions. This operates on TestAttribute objects, so make sure to pass the TestAttribute class as the transform to dir_db functions.

Parameters:
  • complete – Only accept complete tests
  • failed – Only accept failed tests
  • incomplete – Only accept incomplete tests
  • name – Only accept names that match this glob.
  • newer_than – Only accept tests that are more recent than this date.
  • older_than – Only accept tests older than this date.
  • passed – Only accept passed tests
  • result_error – Only accept tests with a result error.
  • show_skipped – Accept skipped tests.
  • sys_name – Only accept tests with a matching sys_name.
  • user – Only accept tests started by this user.
Returns: