Dir Database

Table of Contents

Manage ‘id’ directories. The name of the directory is an integer, which essentially serves as a filesystem primary key.

class pavilion.dir_db.SelectItems(data, paths)
property data

Alias for field number 0

property paths

Alias for field number 1

pavilion.dir_db.create_id_dir(id_dir: ~pathlib.Path) -> (<class 'int'>, <class 'pathlib.Path'>)

In the given directory, create the lowest numbered (positive integer) directory that doesn’t already exist.

Parameters

id_dir – Path to the directory that contains these ‘id’ directories

Returns

The id and path to the created directory.

Raises

  • OSError – on directory creation failure.

  • TimeoutError – If we couldn’t get the lock in time.

pavilion.dir_db.default_filter(_: Path) → bool

Pass every path.

pavilion.dir_db.delete(pav_cfg, id_dir: ~pathlib.Path, filter_func: ~typing.Callable[[~pathlib.Path], bool] = <function default_filter>, transform: ~typing.Optional[~typing.Callable[[~pathlib.Path], ~typing.Any]] = None, verbose: bool = False)

Delete all id directories in a given path that match the given filter.

Parameters

  • pav_cfg – The pavilion config.

  • id_dir – The directory to iterate through.

  • filter_func – A passed filter function, to be passed to select.

  • transform – As per ‘select_from’

  • verbose – Verbose output.

Return int count

The number of directories removed.

Return list msgs

Any messages generated during removal.

pavilion.dir_db.identity(value)

Because lambdas can’t be pickled.

pavilion.dir_db.index(pav_cfg, id_dir: Path, idx_name: str, transform: Callable[[Path], Dict[str, Any]], complete_key: str = 'complete', refresh_period: int = 1, verbose: Optional[IO[str]] = None, fn_base: int = 10) → Index

Load and/or update an index of the given directory for the given transform, and return it. The returned index is a dictionary by id of the transformed data.

Parameters

  • pav_cfg – The pavilion config.

  • id_dir – The directory to index.

  • idx_name – The name of the index.

  • transform – A transformation function that produces a json compatible dictionary.

  • complete_key – The key in the transformed dictionary that marks a record as complete. If not given, the record is always assumed to be complete. Incomplete records are recompiled every time the index is updated (hopefully they will be complete eventually).

  • refresh_period – Only update the index if this much time (in seconds) has passed since the last update.

  • verbose – Print status information during indexing.

  • fn_base – The integer base for dir_db.

pavilion.dir_db.make_id_path(base_path, id_) → Path

Create the full path to an id directory given its base path and the id.

Parameters

  • base_path (Path) – The path to where id directories are stored.

  • id (int) – The id number

Return type

Path

pavilion.dir_db.paths_to_ids(paths: List[Path]) → List[int]

Convert a list of list of dir_db paths to ids.

Parameters

paths – A list of id paths.

Raises

ValueError – For invalid paths

pavilion.dir_db.reset_pkey(id_dir: Path) → None

Reset the the ‘next_id’ for the given directory by deleting the pkey file (‘next_id’) if present.

pavilion.dir_db.select(pav_cfg, id_dir: ~pathlib.Path, filter_func: ~typing.Callable[[~typing.Any], bool] = <function default_filter>, transform: ~typing.Optional[~typing.Callable[[~pathlib.Path], ~typing.Any]] = None, order_func: ~typing.Optional[~typing.Callable[[~typing.Any], ~typing.Any]] = None, order_asc: bool = True, fn_base: int = 10, idx_complete_key: str = 'complete', use_index: ~typing.Union[bool, str] = True, verbose: ~typing.Optional[~typing.IO[str]] = None, limit: ~typing.Optional[int] = None) -> (typing.List[typing.Any], typing.List[pathlib.Path])

Filter and order found paths in the id directory based on the filter and other parameters. If a transform is given, this will create an index of the data returned by the transform to hasten this process.

Parameters

  • pav_cfg – The pavilion config.

  • id_dir – The director

  • transform – Function to apply to each path before applying filters or ordering. The filter and order functions should expect the type returned by this.

  • filter_func – A function that takes a directory, and returns whether to include that directory. True -> include, False -> exclude

  • order_func – A function that returns a comparable value for sorting, as per the list.sort keys argument. Items for which this returns None are removed.

  • order_asc – Whether to sort in ascending or descending order.

  • use_index – The name of (and whether to use) an index. When this is the literal ‘True’, the index name is pulled from the transform function name. A string can also be given to manually specify the name.

  • idx_complete_key – The key used to identify directories as ‘complete’ for indexing purposes. Incomplete directories will be re-indexed until complete.

  • fn_base – Number base for file names. 10 by default, ensure dir name is a valid integer.

  • limit – The max items to return. None denotes return all.

  • verbose – A file like object to print status info to.

Returns

A filtered, ordered list of transformed objects, and the list of untransformed paths.

pavilion.dir_db.select_from(pav_cfg, paths: ~typing.Iterable[~pathlib.Path], filter_func: ~typing.Callable[[~typing.Any], bool] = <function default_filter>, transform: ~typing.Optional[~typing.Callable[[~pathlib.Path], ~typing.Any]] = None, order_func: ~typing.Optional[~typing.Callable[[~typing.Any], ~typing.Any]] = None, order_asc: bool = True, fn_base: int = 10, limit: ~typing.Optional[int] = None) -> (typing.List[typing.Any], typing.List[pathlib.Path])

Filter, order, and truncate the given paths based on the filter and other parameters.

Parameters

  • pav_cfg – The pavilion config.

  • paths – A list of paths to filter, order, and limit.

  • transform – Function to apply to each path before applying filters or ordering. The filter and order functions should expect the type returned by this.

  • filter_func – A function that takes a directory, and returns whether to include that directory. True -> include, False -> exclude

  • order_func – A function that returns a comparable value for sorting, as per the list.sort keys argument. Items for which this returns None are removed.

  • order_asc – Whether to sort in ascending or descending order.

  • fn_base – Number base for file names. 10 by default, ensure dir name is a valid integer.

  • limit – The max items to return. None denotes return all.

Returns

A filtered, ordered list of transformed objects, and the list of untransformed paths.

pavilion.dir_db.select_one(path, ffunc, trans, ofunc, fnb)

Allows the objects to be filtered and transformed in parallel with map.

Parameters

  • path – Path to filter and transform (input to reduced function)

  • ffunc – (filter function) Function that takes a directory, and returns whether to include that directory. True -> include, False -> exclude

  • trans – Function to apply to each path before applying filters or ordering. The filter and order functions should expect the type returned by this.

  • ofunc – A function that returns a comparable value for sorting validate against output.

  • fnb – Number base for file names. 10 by default, ensure dir name is a valid integer.

Returns

A filtered, transformed object.

Dir DB Filters

This module contains functions to generate filter functions for handing to dir_db commands.

pavilion.filters.add_common_filter_args(target: str, arg_parser: ArgumentParser, defaults: dict, sort_options: List[str], disable_opts: List[str])

Add common arguments for all filters.

Parameters

  • target – The name of what is being filtered, to be inserted in documentation. Should be plural.

  • arg_parser – The argparser to add arguments to.

  • defaults – A dictionary of default values for all arguments.

  • sort_options – A list of possible sort options.

  • disable_opts – Options to disable.

Returns

pavilion.filters.add_series_filter_args(arg_parser: ArgumentParser, default_overrides: Optional[Dict[str, Any]] = None, sort_keys: Optional[List[str]] = None, disable_opts: Optional[List[str]] = None) → None

Add a common set of arguments for filtering series (those supported by make_series_filter below).

Arguments and defaults:

{}

Parameters

  • arg_parser – The arg parser (or sub-parser) to add arguments to.

  • default_overrides – A dictionary of defaults to override.

  • sort_keys – A list of sort-by names.

  • disable_opts – Don’t include the listed options.

pavilion.filters.add_test_filter_args(arg_parser: ArgumentParser, default_overrides: Optional[Dict[str, Any]] = None, sort_keys: Optional[List[str]] = None, disable_opts: Optional[List[str]] = None) → None

Add a common set of arguments for filtering tests (those supported by make_test_run_filter below).

Arguments and defaults:

{}

Parameters

  • arg_parser – The arg parser (or sub-parser) to add arguments to.

  • default_overrides – A dictionary of defaults to override.

  • sort_keys – A list of sort-by names, corresponding to keys in the data being sorted.

  • disable_opts – Options to disable (not attach to the arg_parser).

pavilion.filters.filter_test_run(test_attrs: Dict, complete: bool, failed: bool, has_state: str, incomplete: bool, name: str, newer_than: float, older_than: float, passed: bool, result_error: bool, state: str, sys_name: str, user: str)

Determine whether the test run at the given path should be included in the set. This function with test_attrs as the sole input is returned by make_test_run_filter.

Parameters

  • test_attrs – Dict of attributes filtered to determine whether to keep or discard test.

  • complete – Only accept complete tests

  • failed – Only accept failed tests

  • has_state – Only accept tests that have had the given state.

  • incomplete – Only accept incomplete tests

  • name – Only accept names that match this glob.

  • newer_than – Only accept tests that are more recent than this date.

  • older_than – Only accept tests older than this date.

  • passed – Only accept passed tests

  • result_error – Only accept tests with a result error.

  • state – Only accept tests whose state is the one given.

  • sys_name – Only accept tests with a matching sys_name.

  • user – Only accept tests started by this user.

Returns

pavilion.filters.get_sort_opts(sort_name: str, stype: str) -> (typing.Callable[[pathlib.Path], typing.Any], <class 'bool'>)

Return a sort function and sort order.

Parameters

  • sort_name – The name of the sort, possibly prepended with -.

  • stype – TEST or SERIES to select the list of options available for sort_name.

pavilion.filters.make_series_filter(complete: bool = False, has_state: Optional[str] = None, incomplete: bool = False, name: Optional[str] = None, newer_than: Optional[float] = None, older_than: Optional[float] = None, state: Optional[str] = None, sys_name: Optional[str] = None, user: Optional[str] = None) → Callable[[SeriesInfo], bool]

Generate a filter for using with dir_db functions to filter series. This is expected to operate on series.SeriesInfo objects, so make sure to pass Series info as the dir_db transform function.

Parameters

  • complete – Only accept series for which all tests are complete.

  • has_state – Only accept tests that have had the given state.

  • incomplete – Only accept series for which not all tests are complete.

  • name – Only accept series whose name matches the given glob.

  • newer_than – Only accept series created after this time.

  • older_than – Only accept series created before this time.

  • state – Only accept series with the given state.

  • sys_name – Only accept series created on this system.

  • user – Only accept series created by this user.

pavilion.filters.make_test_run_filter(complete: bool = False, failed: bool = False, has_state: Optional[str] = None, incomplete: bool = False, name: Optional[str] = None, newer_than: Optional[float] = None, older_than: Optional[float] = None, passed: bool = False, result_error: bool = False, state: Optional[str] = None, sys_name: Optional[str] = None, user: Optional[str] = None)

Generate a filter function for use by dir_db.select and similar functions. This operates on TestAttribute objects, so make sure to pass the TestAttribute class as the transform to dir_db functions.

Parameters

  • complete – Only accept complete tests

  • failed – Only accept failed tests

  • has_state – Only accept tests that have had this state at some point.

  • incomplete – Only accept incomplete tests

  • name – Only accept names that match this glob.

  • newer_than – Only accept tests that are more recent than this date.

  • older_than – Only accept tests older than this date.

  • passed – Only accept passed tests

  • result_error – Only accept tests with a result error.

  • state – Only accept tests with this as the current state.

  • sys_name – Only accept tests with a matching sys_name.

  • user – Only accept tests started by this user.

Returns

pavilion.filters.sort_func(test, choice)

Use partial to reduce inputs and use as key in sort function. Sort by default key if given key is invalid at this stage. :param test: Dict within list to sort on. :param choice: Key in dict to sort by.