Skip to content

unienv_data.storages.parquet

Classes:

Name Description
ParquetStorage

Stores samples of a flat DictSpace (or a single BoxSpace / BinarySpace / TextSpace) as a directory of

Attributes:

Name Type Description
ParquetBatchType
ParquetSpaceType
ARRAY_ENCODINGS

ParquetBatchType module-attribute

ParquetBatchType = Union[Dict[str, Any], NumpyArrayType, str]

ParquetSpaceType module-attribute

ParquetSpaceType = Union[DictSpace, BoxSpace, TextSpace, BinarySpace]

ARRAY_ENCODINGS module-attribute

ARRAY_ENCODINGS = ('list', 'binary')

ParquetStorage

ParquetStorage(
    single_instance_space: ParquetSpaceType,
    storage_dir: str,
    capacity: Optional[int],
    piece_size: int,
    num_pieces: int,
    total_len: int,
    column_specs: List[Tuple[str, dtype, Tuple[int, ...]]],
    column_shapes: Dict[str, Tuple[int, ...]],
    read_only: bool = False,
    compression: Union[str, Dict[str, str], None] = "snappy",
    array_encodings: Optional[Dict[str, str]] = None,
)

Bases: SpaceStorage[ParquetBatchType, NumpyArrayType, NumpyDeviceType, NumpyDtypeType, NumpyRNGType]

Stores samples of a flat DictSpace (or a single BoxSpace / BinarySpace / TextSpace) as a directory of Parquet files ("pieces") of piece_size rows each, plus _metadata.json. Numpy backend only.

A mutable storage keeps the pieces it touches in memory and rewrites a whole piece when it flushes it: on dumps() / close() / pickling, and, while an unbounded storage grows, every piece as soon as it is complete. The last, partial piece of a growing storage is only written by dumps() / close(); until then the directory holds a loadable prefix of the data (complete pieces, or what the last dumps() wrote).

Parquet suits low-dimensional, columnar data that other tools read too. It is a poor fit for images: every flush re-encodes whole pieces of pixels, and reads decode whole columns of a piece. Store camera frames in VideoStorage / ImageStorage (e.g. through a DictStorage) and keep the low-dim keys here.

Methods:

Name Description
create

Create a storage directory at cache_path.

load_from
build_space_from_parquet_file

Infer space, count, and capacity from a parquet storage directory.

load_replay_buffer_from_raw_parquet
get
set
extend_length
shrink_length
dumps
close
get_column
get_segments
mark_segment_start
append

Append one structured sample to the storage.

mark_segment_end
abort_segment
open_segment_writer

A writer that builds one segment of this storage off to the side; see SegmentWriter.

clear

Clear all data inside the storage and set the length to 0 if the storage has unlimited capacity.

Attributes:

Name Type Description
single_file_ext Optional[str]
DEFAULT_KEY str
capacity
is_mutable bool
is_multiprocessing_safe bool
cache_filename Optional[Union[str, PathLike]]
backend ComputeBackend[BArrayType, BDeviceType, BDtypeType, BRNGType]
device Optional[BDeviceType]
single_instance_space
has_open_segment bool
pending_segment_length int

single_file_ext class-attribute instance-attribute

single_file_ext: Optional[str] = None

DEFAULT_KEY class-attribute instance-attribute

DEFAULT_KEY: str = 'data'

capacity instance-attribute

capacity = capacity

is_mutable property

is_mutable: bool

is_multiprocessing_safe property

is_multiprocessing_safe: bool

cache_filename property

cache_filename: Optional[Union[str, PathLike]]

backend property

backend: ComputeBackend[BArrayType, BDeviceType, BDtypeType, BRNGType]

device property

device: Optional[BDeviceType]

single_instance_space instance-attribute

single_instance_space = single_instance_space

has_open_segment property

has_open_segment: bool

pending_segment_length property

pending_segment_length: int

create classmethod

create(
    single_instance_space: ParquetSpaceType,
    *args,
    capacity: Optional[int] = None,
    cache_path: Optional[Union[str, PathLike]] = None,
    multiprocessing: bool = False,
    piece_size: int = 1024,
    compression: Union[str, Dict[str, str], None] = "snappy",
    array_encoding: Union[str, Mapping[str, str]] = "list",
    **kwargs,
) -> ParquetStorage

Create a storage directory at cache_path.

array_encoding sets how columns holding an array per row are stored, for all of them or per column (a mapping from column names to encodings, "*" for the rest):

  • "list" (default): a fixed-size list of numbers, which pandas, Hugging Face datasets, DuckDB and other Parquet readers load as arrays.
  • "binary": one fixed-size binary value per row holding the array's little-endian bytes. Writing and reading are ~8-10x faster for large arrays such as images (the list encoding pays per element), at the same file size, but other tools see opaque bytes, and UniEnv versions before this option cannot read the files.

The encodings are recorded in _metadata.json; load_from reads both.

load_from classmethod

load_from(
    path: Union[str, PathLike],
    single_instance_space: ParquetSpaceType,
    *,
    capacity: Optional[int] = None,
    read_only: bool = True,
    multiprocessing: bool = False,
    **kwargs,
) -> ParquetStorage

build_space_from_parquet_file staticmethod

build_space_from_parquet_file(
    path: Union[str, PathLike],
) -> Tuple[int, Optional[int], ParquetSpaceType]

Infer space, count, and capacity from a parquet storage directory. Returns (count, capacity, space).

load_replay_buffer_from_raw_parquet staticmethod

load_replay_buffer_from_raw_parquet(
    path: Union[str, PathLike],
) -> ReplayBuffer[
    ParquetBatchType, NumpyArrayType, NumpyDeviceType, NumpyDtypeType, NumpyRNGType
]

get

get(index) -> ParquetBatchType

set

set(index, value: ParquetBatchType) -> None

extend_length

extend_length(length: int) -> None

shrink_length

shrink_length(length: int) -> None

dumps

dumps(path: Union[str, PathLike]) -> None

close

close() -> None

get_column

get_column(nested_keys: Sequence[str]) -> ParquetStorage

get_segments

get_segments()

mark_segment_start

mark_segment_start(start_index: int) -> None

append

append(value) -> None

Append one structured sample to the storage.

Appended values may be immediately visible or only visible after mark_segment_end() depending on the storage implementation.

mark_segment_end

mark_segment_end() -> None

abort_segment

abort_segment() -> None

open_segment_writer

open_segment_writer() -> SegmentWriter

A writer that builds one segment of this storage off to the side; see SegmentWriter. Unlike mark_segment_start() / append(), any number of writers can be open at a time. The default keeps the samples in memory until the commit; storages override it to stream instead.

clear

clear() -> None

Clear all data inside the storage and set the length to 0 if the storage has unlimited capacity. For storages with fixed capacity, this should reset the storage to its initial state.