Dir Database

Table of Contents

Manage ‘id’ directories. The name of the directory is an integer, which essentially serves as a filesystem primary key.

class pavilion.dir_db.SelectItems(data, paths)
data

Alias for field number 0

paths

Alias for field number 1

pavilion.dir_db.create_id_dir(id_dir: pathlib.Path) -> (<class 'int'>, <class 'pathlib.Path'>)

In the given directory, create the lowest numbered (positive integer) directory that doesn’t already exist.

Parameters:

id_dir – Path to the directory that contains these ‘id’ directories
Returns:

The id and path to the created directory.
Raises:
  • OSError – on directory creation failure.
  • TimeoutError – If we couldn’t get the lock in time.
pavilion.dir_db.default_filter(_: pathlib.Path) → bool

Pass every path.

pavilion.dir_db.delete(pav_cfg, id_dir: pathlib.Path, filter_func: Callable[[pathlib.Path], bool] = <function default_filter>, transform: Callable[[pathlib.Path], Any] = None, verbose: bool = False)

Delete all id directories in a given path that match the given filter.

Parameters:
  • pav_cfg – The pavilion config.
  • id_dir – The directory to iterate through.
  • filter_func – A passed filter function, to be passed to select.
  • transform – As per ‘select_from’
  • verbose – Verbose output.
Return int count:
 

The number of directories removed.
Return list msgs:
 

Any messages generated during removal.
pavilion.dir_db.identity(value)

Because lambdas can’t be pickled.

pavilion.dir_db.index(pav_cfg, id_dir: pathlib.Path, idx_name: str, transform: Callable[[pathlib.Path], Dict[str, Any]], complete_key: str = 'complete', refresh_period: int = 1, verbose: IO[str] = None, fn_base: int = 10) → NewType.<locals>.new_type

Load and/or update an index of the given directory for the given transform, and return it. The returned index is a dictionary by id of the transformed data.

Parameters:
  • pav_cfg – The pavilion config.
  • id_dir – The directory to index.
  • idx_name – The name of the index.
  • transform – A transformation function that produces a json compatible dictionary.
  • complete_key – The key in the transformed dictionary that marks a record as complete. If not given, the record is always assumed to be complete. Incomplete records are recompiled every time the index is updated (hopefully they will be complete eventually).
  • refresh_period – Only update the index if this much time (in seconds) has passed since the last update.
  • verbose – Print status information during indexing.
  • fn_base – The integer base for dir_db.
pavilion.dir_db.make_id_path(base_path, id_) → pathlib.Path

Create the full path to an id directory given its base path and the id.

Parameters:
  • base_path (Path) – The path to where id directories are stored.
  • id (int) – The id number
Return type:

Path
pavilion.dir_db.paths_to_ids(paths: List[pathlib.Path]) → List[int]

Convert a list of list of dir_db paths to ids.

Parameters:paths – A list of id paths.
Raises:ValueError – For invalid paths
pavilion.dir_db.reset_pkey(id_dir: pathlib.Path) → None

Reset the the ‘next_id’ for the given directory by deleting the pkey file (‘next_id’) if present.

pavilion.dir_db.select(pav_cfg, id_dir: pathlib.Path, filter_func: Callable[[Any], bool] = <function default_filter>, transform: Callable[[pathlib.Path], Any] = None, order_func: Callable[[Any], Any] = None, order_asc: bool = True, fn_base: int = 10, idx_complete_key: str = 'complete', use_index: Union[bool, str] = True, verbose: IO[str] = None, limit: int = None) -> (typing.List[typing.Any], typing.List[pathlib.Path])

Filter and order found paths in the id directory based on the filter and other parameters. If a transform is given, this will create an index of the data returned by the transform to hasten this process.

Parameters:
  • pav_cfg – The pavilion config.
  • id_dir – The director
  • transform – Function to apply to each path before applying filters or ordering. The filter and order functions should expect the type returned by this.
  • filter_func – A function that takes a directory, and returns whether to include that directory. True -> include, False -> exclude
  • order_func – A function that returns a comparable value for sorting, as per the list.sort keys argument. Items for which this returns None are removed.
  • order_asc – Whether to sort in ascending or descending order.
  • use_index – The name of (and whether to use) an index. When this is the literal ‘True’, the index name is pulled from the transform function name. A string can also be given to manually specify the name.
  • idx_complete_key – The key used to identify directories as ‘complete’ for indexing purposes. Incomplete directories will be re-indexed until complete.
  • fn_base – Number base for file names. 10 by default, ensure dir name is a valid integer.
  • limit – The max items to return. None denotes return all.
  • verbose – A file like object to print status info to.
Returns:

A filtered, ordered list of transformed objects, and the list of untransformed paths.
pavilion.dir_db.select_from(pav_cfg, paths: Iterable[pathlib.Path], filter_func: Callable[[Any], bool] = <function default_filter>, transform: Callable[[pathlib.Path], Any] = None, order_func: Callable[[Any], Any] = None, order_asc: bool = True, fn_base: int = 10, limit: int = None) -> (typing.List[typing.Any], typing.List[pathlib.Path])

Filter, order, and truncate the given paths based on the filter and other parameters.

Parameters:
  • pav_cfg – The pavilion config.
  • paths – A list of paths to filter, order, and limit.
  • transform – Function to apply to each path before applying filters or ordering. The filter and order functions should expect the type returned by this.
  • filter_func – A function that takes a directory, and returns whether to include that directory. True -> include, False -> exclude
  • order_func – A function that returns a comparable value for sorting, as per the list.sort keys argument. Items for which this returns None are removed.
  • order_asc – Whether to sort in ascending or descending order.
  • fn_base – Number base for file names. 10 by default, ensure dir name is a valid integer.
  • limit – The max items to return. None denotes return all.
Returns:

A filtered, ordered list of transformed objects, and the list of untransformed paths.
pavilion.dir_db.select_one(path, ffunc, trans, ofunc, fnb)

Allows the objects to be filtered and transformed in parallel with map.

Parameters:
  • path – Path to filter and transform (input to reduced function)
  • ffunc – (filter function) Function that takes a directory, and returns whether to include that directory. True -> include, False -> exclude
  • trans – Function to apply to each path before applying filters or ordering. The filter and order functions should expect the type returned by this.
  • ofunc – A function that returns a comparable value for sorting validate against output.
  • fnb – Number base for file names. 10 by default, ensure dir name is a valid integer.
Returns:

A filtered, transformed object.

Dir DB Filters

This module contains functions to generate filter functions for handing to dir_db commands.

pavilion.filters.add_common_filter_args(target: str, arg_parser: argparse.ArgumentParser, defaults: dict, sort_options: List[str])

Add common arguments for all filters.

Parameters:
  • target – The name of what is being filtered, to be inserted in documentation. Should be plural.
  • arg_parser – The argparser to add arguments to.
  • defaults – A dictionary of default values for all arguments.
  • sort_options – A list of possible sort options.
Returns:
pavilion.filters.add_series_filter_args(arg_parser: argparse.ArgumentParser, default_overrides: Dict[str, Any] = None, sort_functions: Dict[str, Callable] = None) → None

Add a common set of arguments for filtering series (those supported by make_series_filter below).

Arguments and defaults:

{}

Parameters:
  • arg_parser – The arg parser (or sub-parser) to add arguments to.
  • default_overrides – A dictionary of defaults to override.
  • sort_functions – A dict of sort-by names and sorting functions. The functions don’t matter at this point. If empty, sorting is disabled.
pavilion.filters.add_test_filter_args(arg_parser: argparse.ArgumentParser, default_overrides: Dict[str, Any] = None, sort_functions: Dict[str, Callable] = None) → None

Add a common set of arguments for filtering tests (those supported by make_test_run_filter below).

Arguments and defaults:

{}

Parameters:
  • arg_parser – The arg parser (or sub-parser) to add arguments to.
  • default_overrides – A dictionary of defaults to override.
  • sort_functions – A dict of sort-by names and sorting functions. The functions don’t matter at this point. If empty, sorting is disabled.
pavilion.filters.filter_test_run(test_attrs: Dict[KT, VT], complete: bool, failed: bool, has_state: str, incomplete: bool, name: str, newer_than: float, older_than: float, passed: bool, result_error: bool, state: str, sys_name: str, user: str)

Determine whether the test run at the given path should be included in the set. This function with test_attrs as the sole input is returned by make_test_run_filter.

Parameters:
  • test_attrs – Dict of attributes filtered to determine whether to keep or discard test.
  • complete – Only accept complete tests
  • failed – Only accept failed tests
  • has_state – Only accept tests that have had the given state.
  • incomplete – Only accept incomplete tests
  • name – Only accept names that match this glob.
  • newer_than – Only accept tests that are more recent than this date.
  • older_than – Only accept tests older than this date.
  • passed – Only accept passed tests
  • result_error – Only accept tests with a result error.
  • state – Only accept tests whose state is the one given.
  • sys_name – Only accept tests with a matching sys_name.
  • user – Only accept tests started by this user.
Returns:
pavilion.filters.get_sort_opts(sort_name: str, stype: str) -> (typing.Callable[[pathlib.Path], typing.Any], <class 'bool'>)

Return a sort function and sort order.

Parameters:
  • sort_name – The name of the sort, possibly prepended with -.
  • stype – TEST or SERIES to select the list of options available for sort_name.
pavilion.filters.make_series_filter(complete: bool = False, has_state: str = None, incomplete: bool = False, name: str = None, newer_than: datetime.datetime = None, older_than: datetime.datetime = None, state: str = None, sys_name: str = None, user: str = None) → Callable[[pavilion.series.info.SeriesInfo], bool]

Generate a filter for using with dir_db functions to filter series. This is expected to operate on series.SeriesInfo objects, so make sure to pass Series info as the dir_db transform function.

Parameters:
  • complete – Only accept series for which all tests are complete.
  • has_state – Only accept tests that have had the given state.
  • incomplete – Only accept series for which not all tests are complete.
  • name – Only accept series whose name matches the given glob.
  • newer_than – Only accept series created after this time.
  • older_than – Only accept series created before this time.
  • state – Only accept series with the given state.
  • sys_name – Only accept series created on this system.
  • user – Only accept series created by this user.
pavilion.filters.make_test_run_filter(complete: bool = False, failed: bool = False, has_state: str = None, incomplete: bool = False, name: str = None, newer_than: float = None, older_than: float = None, passed: bool = False, result_error: bool = False, state: str = None, sys_name: str = None, user: str = None)

Generate a filter function for use by dir_db.select and similar functions. This operates on TestAttribute objects, so make sure to pass the TestAttribute class as the transform to dir_db functions.

Parameters:
  • complete – Only accept complete tests
  • failed – Only accept failed tests
  • has_state – Only accept tests that have had this state at some point.
  • incomplete – Only accept incomplete tests
  • name – Only accept names that match this glob.
  • newer_than – Only accept tests that are more recent than this date.
  • older_than – Only accept tests older than this date.
  • passed – Only accept passed tests
  • result_error – Only accept tests with a result error.
  • state – Only accept tests with this as the current state.
  • sys_name – Only accept tests with a matching sys_name.
  • user – Only accept tests started by this user.
Returns:
pavilion.filters.sort_func(test, choice, sort_type)

Use partial to reduce inputs and use as key in sort function. :param test: Dict within list to sort on. :param choice: Key in dict to sort by. :param sort_type: Type of list of dicts to sort by, check for key.