Dir Database

Manage ‘id’ directories. The name of the directory is an integer, which essentially serves as a filesystem primary key.

class pavilion.dir_db.SelectItems(data, paths)
data: List[Dict[str, Any]]

Alias for field number 0

paths: List[Path]

Alias for field number 1

pavilion.dir_db.default_filter(_: Path) → bool

Pass every path.

pavilion.dir_db.delete(pav_cfg, id_dir: ~pathlib.Path, filter_func: ~typing.Callable[[~pathlib.Path], bool] = <function default_filter>, transform: ~typing.Callable[[~pathlib.Path], ~typing.Any] | None = None, verbose: bool = False)

Delete all id directories in a given path that match the given filter.

Parameters:

  • pav_cfg – The pavilion config.

  • id_dir – The directory to iterate through.

  • filter_func – A passed filter function, to be passed to select.

  • transform – As per ‘select_from’

  • verbose – Verbose output.

Return int count:

The number of directories removed.

Return list msgs:

Any messages generated during removal.

pavilion.dir_db.identity(value: T) → T

Because lambdas can’t be pickled.

pavilion.dir_db.index(pav_cfg: PavConfig, id_dir: Path, idx_name: str, transform: Callable[[Path], Dict[str, Any]], complete_key: str = 'complete', refresh_period: int = 1, verbose: IO[str] | None = None) → Index

Load and/or update an index of the given directory for the given transform, and return it. The returned index is a dictionary by id of the transformed data.

Parameters:

  • pav_cfg – The pavilion config.

  • id_dir – The directory to index.

  • idx_name – The name of the index.

  • transform – A transformation function that produces a json compatible dictionary.

  • complete_key – The key in the transformed dictionary that marks a record as complete. If not given, the record is always assumed to be complete. Incomplete records are recompiled every time the index is updated (hopefully they will be complete eventually).

  • refresh_period – Only update the index if this much time (in seconds) has passed since the last update.

  • verbose – Print status information during indexing.

  • fn_base – The integer base for dir_db.

pavilion.dir_db.paths_to_ids(paths: List[Path]) → List[int]

Convert a list of list of dir_db paths to ids.

Parameters:

paths – A list of id paths.

Raises:

ValueError – For invalid paths

pavilion.dir_db.reset_pkey(id_dir: Path) → None

Reset the the ‘next_id’ for the given directory by deleting the pkey file (‘next_id’) if present.

pavilion.dir_db.select(pav_cfg: ~pavilion.config.PavConfig, id_dir: ~pathlib.Path, filter_func: ~typing.Callable[[~typing.Any], bool] = <function default_filter>, transform: ~typing.Callable[[~pathlib.Path], ~typing.Any] | None = None, order_func: ~typing.Callable[[~typing.Dict[str, ~typing.Any]], ~typing.Any] | None = None, order_asc: bool = True, idx_complete_key: str = 'complete', use_index: bool | str = True, verbose: ~typing.IO[str] | None = None, limit: int | None = None) → SelectItems

Filter and order found paths in the id directory based on the filter and other parameters. If a transform is given, this will create an index of the data returned by the transform to hasten this process.

Parameters:

  • pav_cfg – The pavilion config.

  • id_dir – The director

  • transform – Function to apply to each path before applying filters or ordering. The filter and order functions should expect the type returned by this.

  • filter_func – A function that takes a directory, and returns whether to include that directory. True -> include, False -> exclude

  • order_func – A function that returns a comparable value for sorting, as per the list.sort keys argument. Items for which this returns None are removed.

  • order_asc – Whether to sort in ascending or descending order.

  • use_index – The name of (and whether to use) an index. When this is the literal ‘True’, the index name is pulled from the transform function name. A string can also be given to manually specify the name.

  • idx_complete_key – The key used to identify directories as ‘complete’ for indexing purposes. Incomplete directories will be re-indexed until complete.

  • limit – The max items to return. None denotes return all.

  • verbose – A file like object to print status info to.

Returns:

A filtered, ordered list of transformed objects, and the list of untransformed paths.

pavilion.dir_db.select_from(pav_cfg: ~pavilion.config.PavConfig, paths: ~typing.Iterable[~pathlib.Path], filter_func: ~typing.Callable[[~pavilion.dir_db.T], bool] = <function default_filter>, transform: ~typing.Callable[[~pathlib.Path], ~pavilion.dir_db.T] | None = None, order_func: ~typing.Callable[[~pavilion.dir_db.T], ~typing.Any] | None = None, order_asc: bool = True, limit: int | None = None) → SelectItems

Filter, order, and truncate the given paths based on the filter and other parameters.

Parameters:

  • pav_cfg – The pavilion config.

  • paths – A list of paths to filter, order, and limit.

  • transform – Function to apply to each path before applying filters or ordering. The filter and order functions should expect the type returned by this.

  • filter_func – A function that takes a directory, and returns whether to include that directory. True -> include, False -> exclude

  • order_func – A function that returns a comparable value for sorting, as per the list.sort keys argument. Items for which this returns None are removed.

  • order_asc – Whether to sort in ascending or descending order.

  • limit – The max items to return. None denotes return all.

Returns:

A filtered, ordered list of transformed objects, and the list of untransformed paths.

pavilion.dir_db.select_one(path: Path, ffunc: Callable[[Path], bool] | None, trans: Callable[[Path], T] | None, ofunc: Callable[[T], Any]) → T | None

Allows the objects to be filtered and transformed in parallel with map.

Parameters:

  • path – Path to filter and transform (input to reduced function)

  • ffunc – (filter function) Function that takes a directory, and returns whether to include that directory. True -> include, False -> exclude

  • trans – Function to apply to each path before applying filters or ordering. The filter and order functions should expect the type returned by this.

  • ofunc – A function that returns a comparable value for sorting validate against output.

Returns:

A filtered, transformed object.

Dir DB Filters