Reading POD5 Datasets

Continuing from the Reading POD5 Files introduction

The DatasetReader aims to provide a reader interface for a collection of POD5 files which is iterable (see reads()) and indexable by read_id (see get_read()).

Loading a Dataset

The DatasetReader parses all directory and file paths supplied to the class constructor. This argument supports strings, paths or a collection of both.

from pathlib import Path
import pod5

# Load a single POD5 file
with pod5.DatasetReader("string.pod5") as dataset:
    assert dataset.paths == [Path("string.pod5")]

# Load a directory of POD5 files (without searching sub-directories)
with pod5.DatasetReader(Path("/my/dataset")) as dataset:
    ...

# Recursively load multiple directories and a specific file
paths = [Path("/my/dataset"), Path("../other/dataset"), "extra.pod5"]
with pod5.DatasetReader(paths, recursive=True) as dataset:
    ...

Iterate Over Reads

As shown in the introduction the DatasetReader class implements the __iter__ method which yields all ReadRecord’s in the dataset. This method is equivalent to calling reads() with no parameters. A selection of read_ids can be passed to reads() if only some records are required.

This is the most efficient way to inspect a large number of records.

Note

The order of records returned by Reader iterators is always the order on-disk even when specifying a selection of read_ids.

import pod5

with pod5.DatasetReader("./dataset/", recursive=True) as dataset:
    # Iterate over every `ReadRecord` in the dataset
    for read_record in dataset.reads():
        ...

    # This is equivalent to calling .reads()
    for read_record in dataset:
        ...

    # Example: Iterate over a random selection of records
    selection = random.sample(dataset.read_ids, 5)
    for read_record in dataset.reads(selection=selection)
        # Order of read_records is the on-disk record order
        ...

Random Access to Records

The get_read() method allows users to select any record in the dataset by read_id.

To efficiently access records from a POD5 dataset, the DatasetReader will index the read_ids of all records and cache POD5 Reader instances.

import pod5

with pod5.DatasetReader("./dataset/") as dataset:
    read_id = "00445e58-3c58-4050-bacf-3411bb716cc3"

    # Dataset indexing will take place here
    read_record = dataset.get_read(read_id)

    # Returned object might be None if read_id is not found
    if read_record is None:
        print(f"dataset does not contain read_id: {read_id}")

    assert str(read_record.read_id) == read_id

Indexing Records

If necessary, the DatasetReader will index every record in the dataset. This may consume a significant amount of memory depending on the size of the dataset. Indexing is only done when a call to a function which requires the index is made. The functions which require the index are get_read(), get_path(), and has_duplicate(). To clear the index, freeing the memory, call clear_index().

Cached Readers

The DatasetReader class uses an LRU cache of Reader instances to reduce the overhead of repeatedly re-opening POD5 files. The size of this cache can be controlled by setting max_cached_readers during initialisation.

Where possible, users should maximise the likelihood of a Reader cache hit by ensuring that successive calls to get_read() access records in no more POD5s than the size of the LRU cache. Randomly indexing read_ids into many files will result in repeatedly opening the underlying files which will severely affect performance.

Duplicate Records

For a typical dataset sourced from a sequencer, it is vanishingly unlikely that there will be a single duplicate UUID read_id. It is much more likely that some POD5 files in a dataset may be copied, merged, subset etc and that loading all files in a dataset especially using the recursive mode will find duplicate read_ids.

The DatasetReader handles duplicate records in iterators by returning all copies as they appear on disk. Similarly, having duplicate read_ids in the selection will repeatedly return duplicates.

The DatasetReader handles duplicate records when indexing by returning a random valid ReadRecord if it exists or None. The ReadRecord instance returned is random because the indexing process is multi-threaded.

If a duplicate read_id is detected, the API will issue a warning. This can be disabled with warn_duplicate_indexing=False in the initialiser.

DatasetReader Reference

class DatasetReader(paths: Union[PathLike, str, Collection[Union[PathLike, str]]], recursive: bool = False, pattern: str = '*.pod5', index: bool = False, threads: int = 2, max_cached_readers: Optional[int] = 16, warn_duplicate_indexing: bool = True)[source]

__init__(paths: Union[PathLike, str, Collection[Union[PathLike, str]]], recursive: bool = False, pattern: str = '*.pod5', index: bool = False, threads: int = 2, max_cached_readers: Optional[int] = 16, warn_duplicate_indexing: bool = True) → None[source]

Reads pod5 files and/or directories of pod5 files as a dataset.

Parameters:

paths (PathOrStr | Collection[PathOrStr]) – One or more files or directories to load
recursive (bool) – Search directories in paths recursively
pattern (str) – A glob expression to match against file names
index (bool) – Promptly index the dataset instead of deferring until required
threads (int) – The number of threads to use
max_cached_readers (Optional[int]) – The maximum size of the Reader LRU cache. Set to None for an unlimited cache size.
warn_duplicate_indexing (bool) – Issue warnings when duplicate read_ids are detected and indexing by read_id is attempted

Note

Random record access is implemented by creating an index of read_id to file path. This can consume a large amount of memory. Methods that generate an index have this noted in their docstring.

Warning

If duplicate read_ids are present in the dataset, iterator methods such as reads() will yield all copies. Indexing methods such as get_read return one chosen randomly and issue a warning which can be suppressed by setting warn_duplicate_indexing=False

clear_index() → None[source]: Clears the read_id to file path index

clear_readers() → None[source]: Clears the readers LRU cache

get_path(read_id: str) → Optional[Path][source]

Get the pod5 Path for a given read_id or None if it was not found

Parameters:: read_id (str) – The read_id (UUID) string in this dataset

Note

This method will index the dataset

Warning

Issues a warning if duplicate read_ids are detected in this dataset. The returned path is a always valid file which contains this read_id but this may be random between instances.

Return type:: A Path or None

get_read(read_id: str) → Optional[ReadRecord][source]

Get a ReadRecord by read_id or return None if it is missing

Parameters:: read_id (str) – The read_id (UUID) string in this dataset to find

Note

This method will index the dataset

Warning

Issues a warning if duplicate read_ids are detected in this dataset. The returned ReadRecord is a always valid but the source may be random between instances of a DatasetReader.

Return type:: A ReadRecord or None

get_reader(path: Union[PathLike, str]) → Reader[source]

Get a pod5 file Reader in this dataset by path

Parameters:: path (PathOrStr) – Path to a pod5 file
Return type:: A Reader

has_duplicate() → bool[source]: Returns True if there are duplicate read_ids in this dataset

Note

This method will index the dataset

property num_reads: int: Return the number of ReadRecords in this dataset.

property paths: List[Path]: Return the list of pod5 file paths in this dataset

property read_ids: Generator[str, None, None]: Yield all read_ids in this dataset

reads(selection: Optional[Iterable[str]] = None, preload: Optional[Set[str]] = None) → Generator[ReadRecord, None, None][source]

Iterate over ``ReadRecord``s in the dataset.

Parameters:

selection (iterable[str]) – The read ids to walk in the file.
preload (set[str]) – Columns to preload - “samples” and “sample_count” are valid values

Note

ReadRecord``s are yielded in on-disk record order for each file in ``self.paths.

Missing records are not detected and multiple records will be yielded if there are duplicates in either of the dataset or selection.

Yields:: ReadRecord