unienv_data.storages.flattened

FlattenedStorage

FlattenedStorage(single_instance_space: Space[BatchT, BDeviceType, BDtypeType, BRNGType], inner_storage: SpaceStorage[BArrayType, BArrayType, BDeviceType, BDtypeType, BRNGType], inner_storage_path: Union[str, PathLike], cache_filename: Optional[Union[str, PathLike]] = None)

Bases: SpaceStorage[BatchT, BArrayType, BDeviceType, BDtypeType, BRNGType]

Stores every sample flattened into one 1D BoxSpace row of an inner storage (e.g. PytorchTensorStorage), so a read or write is one row operation on a single array.

The flat row has one common dtype, promoted from all leaves: mixing uint8 camera images with float32 keys stores the images as float32 (4x the memory / disk of uint8, e.g. 1.97 GB instead of 0.49 GB for 10k 128x128x3 frames) and converts every image on every read and write. Keep images in their own storage instead, e.g. a DictStorage with a PytorchTensorStorage (or VideoStorage) per image key, and flatten only the low-dimensional keys.

single_file_ext class-attribute instance-attribute

single_file_ext = None

inner_storage instance-attribute

inner_storage = inner_storage

inner_storage_path instance-attribute

inner_storage_path = inner_storage_path

cache_filename property

cache_filename: Optional[Union[str, PathLike]]

is_mutable property

is_mutable: bool

is_multiprocessing_safe property

is_multiprocessing_safe: bool

capacity property

capacity: Optional[int]

has_open_segment property

has_open_segment: bool

pending_segment_length property

pending_segment_length: int

backend property

backend: ComputeBackend[BArrayType, BDeviceType, BDtypeType, BRNGType]

device property

device: Optional[BDeviceType]

single_instance_space instance-attribute

single_instance_space = single_instance_space

create classmethod

create(single_instance_space: Space[BatchT, BDeviceType, BDtypeType, BRNGType], inner_storage_cls: Type[SpaceStorage[BArrayType, BArrayType, BDeviceType, BDtypeType, BRNGType]], *args, capacity: Optional[int] = None, cache_path: Optional[str] = None, multiprocessing: bool = False, inner_storage_kwargs: Dict[str, Any] = {}, **kwargs) -> FlattenedStorage[BatchT, BArrayType, BDeviceType, BDtypeType, BRNGType]

load_from classmethod

load_from(path: Union[str, PathLike], single_instance_space: Space[BatchT, BDeviceType, BDtypeType, BRNGType], *, capacity: Optional[int] = None, read_only: bool = True, multiprocessing: bool = False, **kwargs) -> FlattenedStorage[BatchT, BArrayType, BDeviceType, BDtypeType, BRNGType]

extend_length

extend_length(length)

shrink_length

shrink_length(length)

mark_segment_start

mark_segment_start(start_index: int) -> None

append

append(value) -> None

open_segment_writer

open_segment_writer() -> SegmentWriter

mark_segment_end

mark_segment_end() -> None

abort_segment

abort_segment() -> None

get_segments

get_segments()

get_flattened

get_flattened(index)

get

get(index)

set_flattened

set_flattened(index, value)

set

set(index, value)

clear

clear()

dumps

dumps(path)

close

close()