unienv_data.storages.parquet¶
Classes:
| Name | Description |
|---|---|
ParquetStorage |
Stores samples of a flat |
Attributes:
| Name | Type | Description |
|---|---|---|
ParquetBatchType |
|
|
ParquetSpaceType |
|
|
ARRAY_ENCODINGS |
|
ParquetSpaceType
module-attribute
¶
ParquetSpaceType = Union[DictSpace, BoxSpace, TextSpace, BinarySpace]
ParquetStorage
¶
ParquetStorage(
single_instance_space: ParquetSpaceType,
storage_dir: str,
capacity: Optional[int],
piece_size: int,
num_pieces: int,
total_len: int,
column_specs: List[Tuple[str, dtype, Tuple[int, ...]]],
column_shapes: Dict[str, Tuple[int, ...]],
read_only: bool = False,
compression: Union[str, Dict[str, str], None] = "snappy",
array_encodings: Optional[Dict[str, str]] = None,
)
Bases: SpaceStorage[ParquetBatchType, NumpyArrayType, NumpyDeviceType, NumpyDtypeType, NumpyRNGType]
Stores samples of a flat DictSpace (or a single BoxSpace / BinarySpace / TextSpace) as a directory of
Parquet files ("pieces") of piece_size rows each, plus _metadata.json. Numpy backend only.
A mutable storage keeps the pieces it touches in memory and rewrites a whole piece when it flushes it: on
dumps() / close() / pickling, and, while an unbounded storage grows, every piece as soon as it is
complete. The last, partial piece of a growing storage is only written by dumps() / close(); until then
the directory holds a loadable prefix of the data (complete pieces, or what the last dumps() wrote).
Parquet suits low-dimensional, columnar data that other tools read too. It is a poor fit for images: every
flush re-encodes whole pieces of pixels, and reads decode whole columns of a piece. Store camera frames in
VideoStorage / ImageStorage (e.g. through a DictStorage) and keep the low-dim keys here.
Methods:
| Name | Description |
|---|---|
create |
Create a storage directory at |
load_from |
|
build_space_from_parquet_file |
Infer space, count, and capacity from a parquet storage directory. |
load_replay_buffer_from_raw_parquet |
|
get |
|
set |
|
extend_length |
|
shrink_length |
|
dumps |
|
close |
|
get_column |
|
get_segments |
|
mark_segment_start |
|
append |
Append one structured sample to the storage. |
mark_segment_end |
|
abort_segment |
|
open_segment_writer |
A writer that builds one segment of this storage off to the side; see |
clear |
Clear all data inside the storage and set the length to 0 if the storage has unlimited capacity. |
Attributes:
| Name | Type | Description |
|---|---|---|
single_file_ext |
Optional[str]
|
|
DEFAULT_KEY |
str
|
|
capacity |
|
|
is_mutable |
bool
|
|
is_multiprocessing_safe |
bool
|
|
cache_filename |
Optional[Union[str, PathLike]]
|
|
backend |
ComputeBackend[BArrayType, BDeviceType, BDtypeType, BRNGType]
|
|
device |
Optional[BDeviceType]
|
|
single_instance_space |
|
|
has_open_segment |
bool
|
|
pending_segment_length |
int
|
|
create
classmethod
¶
create(
single_instance_space: ParquetSpaceType,
*args,
capacity: Optional[int] = None,
cache_path: Optional[Union[str, PathLike]] = None,
multiprocessing: bool = False,
piece_size: int = 1024,
compression: Union[str, Dict[str, str], None] = "snappy",
array_encoding: Union[str, Mapping[str, str]] = "list",
**kwargs,
) -> ParquetStorage
Create a storage directory at cache_path.
array_encoding sets how columns holding an array per row are stored, for all of them or per column
(a mapping from column names to encodings, "*" for the rest):
"list"(default): a fixed-size list of numbers, which pandas, Hugging Facedatasets, DuckDB and other Parquet readers load as arrays."binary": one fixed-size binary value per row holding the array's little-endian bytes. Writing and reading are ~8-10x faster for large arrays such as images (the list encoding pays per element), at the same file size, but other tools see opaque bytes, and UniEnv versions before this option cannot read the files.
The encodings are recorded in _metadata.json; load_from reads both.
load_from
classmethod
¶
load_from(
path: Union[str, PathLike],
single_instance_space: ParquetSpaceType,
*,
capacity: Optional[int] = None,
read_only: bool = True,
multiprocessing: bool = False,
**kwargs,
) -> ParquetStorage
build_space_from_parquet_file
staticmethod
¶
build_space_from_parquet_file(
path: Union[str, PathLike],
) -> Tuple[int, Optional[int], ParquetSpaceType]
Infer space, count, and capacity from a parquet storage directory. Returns (count, capacity, space).
load_replay_buffer_from_raw_parquet
staticmethod
¶
load_replay_buffer_from_raw_parquet(
path: Union[str, PathLike],
) -> ReplayBuffer[
ParquetBatchType, NumpyArrayType, NumpyDeviceType, NumpyDtypeType, NumpyRNGType
]
append
¶
Append one structured sample to the storage.
Appended values may be immediately visible or only visible after
mark_segment_end() depending on the storage implementation.
open_segment_writer
¶
open_segment_writer() -> SegmentWriter
A writer that builds one segment of this storage off to the side; see SegmentWriter.
Unlike mark_segment_start() / append(), any number of writers can be open at a time.
The default keeps the samples in memory until the commit; storages override it to stream instead.
clear
¶
Clear all data inside the storage and set the length to 0 if the storage has unlimited capacity. For storages with fixed capacity, this should reset the storage to its initial state.