unienv_data.storages.parquet¶
module-attribute
¶
ParquetSpaceType = Union[DictSpace, BoxSpace, TextSpace, BinarySpace]
ParquetStorage(single_instance_space: ParquetSpaceType, storage_dir: str, capacity: Optional[int], piece_size: int, num_pieces: int, total_len: int, column_specs: List[Tuple[str, dtype, Tuple[int, ...]]], column_shapes: Dict[str, Tuple[int, ...]], read_only: bool = False, compression: Union[str, Dict[str, str], None] = 'snappy')
Bases: SpaceStorage[ParquetBatchType, NumpyArrayType, NumpyDeviceType, NumpyDtypeType, NumpyRNGType]
Stores samples of a flat DictSpace (or a single BoxSpace / BinarySpace / TextSpace) as a directory of
Parquet files ("pieces") of piece_size rows each, plus _metadata.json. Numpy backend only.
A mutable storage keeps the pieces it touches in memory and rewrites a whole piece when it flushes it: on
dumps() / close() / pickling, and, while an unbounded storage grows, every piece as soon as it is
complete. The last, partial piece of a growing storage is only written by dumps() / close(); until then
the directory holds a loadable prefix of the data (complete pieces, or what the last dumps() wrote).
Parquet suits low-dimensional, columnar data that other tools read too. It is a poor fit for images: every
flush re-encodes whole pieces of pixels, and reads decode whole columns of a piece. Store camera frames in
VideoStorage / ImageStorage (e.g. through a DictStorage) and keep the low-dim keys here.
classmethod
¶
create(single_instance_space: ParquetSpaceType, *args, capacity: Optional[int] = None, cache_path: Optional[Union[str, PathLike]] = None, multiprocessing: bool = False, piece_size: int = 1024, compression: Union[str, Dict[str, str], None] = 'snappy', **kwargs) -> ParquetStorage
classmethod
¶
load_from(path: Union[str, PathLike], single_instance_space: ParquetSpaceType, *, capacity: Optional[int] = None, read_only: bool = True, multiprocessing: bool = False, **kwargs) -> ParquetStorage
staticmethod
¶
build_space_from_parquet_file(path: Union[str, PathLike]) -> Tuple[int, Optional[int], ParquetSpaceType]
Infer space, count, and capacity from a parquet storage directory. Returns (count, capacity, space).
staticmethod
¶
load_replay_buffer_from_raw_parquet(path: Union[str, PathLike]) -> ReplayBuffer[ParquetBatchType, NumpyArrayType, NumpyDeviceType, NumpyDtypeType, NumpyRNGType]
append(value) -> None
Append one structured sample to the storage.
Appended values may be immediately visible or only visible after
mark_segment_end() depending on the storage implementation.
open_segment_writer() -> SegmentWriter
A writer that builds one segment of this storage off to the side; see SegmentWriter.
Unlike mark_segment_start() / append(), any number of writers can be open at a time.
The default keeps the samples in memory until the commit; storages override it to stream instead.
clear() -> None
Clear all data inside the storage and set the length to 0 if the storage has unlimited capacity. For storages with fixed capacity, this should reset the storage to its initial state.