unienv_data.base.storage¶
Classes:
| Name | Description |
|---|---|
SegmentWriter |
Builds one contiguous segment of a storage off to the side. |
BufferedSegmentWriter |
Keeps the appended samples in memory and writes them with one |
MappedSegmentWriter |
Forwards to a writer of an inner storage, mapping every appended sample with |
SpaceStorage |
SpaceStorage is an abstract base class for storages that hold instances of a specific space. |
SegmentWriter
¶
SegmentWriter(storage: SpaceStorage)
Bases: ABC
Builds one contiguous segment of a storage off to the side.
Values appended to a writer land at an index range that is only chosen when the writer is committed,
so several writers of one storage can be filled at the same time (e.g. one per environment of a batched rollout)
and committed in any order. Obtain one with SpaceStorage.open_segment_writer().
Methods:
| Name | Description |
|---|---|
append |
Append one structured sample to the segment. |
finish |
Declare the segment complete (no more appends) and release what only appending needs (e.g. encoders). |
commit |
Write the segment to the storage indices |
abort |
Drop the segment without writing it. |
Attributes:
| Name | Type | Description |
|---|---|---|
storage |
|
|
length |
|
|
is_finished |
|
finish
¶
Declare the segment complete (no more appends) and release what only appending needs (e.g. encoders).
Called by commit() if needed; finish early when the commit has to wait.
commit
abstractmethod
¶
Write the segment to the storage indices [start_index, start_index + len(self))
(modulo the capacity of fixed-capacity storages). Storages with unlimited capacity must already be long enough.
The writer cannot be used afterwards.
BufferedSegmentWriter
¶
BufferedSegmentWriter(storage: SpaceStorage)
Bases: SegmentWriter
Keeps the appended samples in memory and writes them with one set() on commit.
Methods:
| Name | Description |
|---|---|
commit |
|
abort |
|
append |
Append one structured sample to the segment. |
finish |
Declare the segment complete (no more appends) and release what only appending needs (e.g. encoders). |
Attributes:
| Name | Type | Description |
|---|---|---|
storage |
|
|
length |
|
|
is_finished |
|
finish
¶
Declare the segment complete (no more appends) and release what only appending needs (e.g. encoders).
Called by commit() if needed; finish early when the commit has to wait.
MappedSegmentWriter
¶
MappedSegmentWriter(
storage: SpaceStorage, inner_writer: SegmentWriter, transform: Callable[[Any], Any]
)
Bases: SegmentWriter
Forwards to a writer of an inner storage, mapping every appended sample with transform.
Methods:
| Name | Description |
|---|---|
commit |
|
abort |
|
append |
Append one structured sample to the segment. |
finish |
Declare the segment complete (no more appends) and release what only appending needs (e.g. encoders). |
Attributes:
| Name | Type | Description |
|---|---|---|
inner_writer |
|
|
transform |
|
|
storage |
|
|
length |
|
|
is_finished |
|
finish
¶
Declare the segment complete (no more appends) and release what only appending needs (e.g. encoders).
Called by commit() if needed; finish early when the commit has to wait.
SpaceStorage
¶
Bases: ABC, Generic[BatchT, BArrayType, BDeviceType, BDtypeType, BRNGType]
SpaceStorage is an abstract base class for storages that hold instances of a specific space.
It provides a common interface for creating, loading, and managing the storage of instances of a given space.
Note that if you want your space storage to support multiprocessing, you need to check / implement __getstate__ and __setstate__ methods to ensure that the storage can be pickled and unpickled correctly.
Bind the storage to the space of one stored element.
Methods:
| Name | Description |
|---|---|
create |
Create a new storage instance for samples from |
load_from |
Restore a storage instance previously written to disk. |
get_segments |
|
mark_segment_start |
|
append |
Append one structured sample to the storage. |
mark_segment_end |
|
abort_segment |
|
extend_length |
This is used by capacity = None storages to extend the length of the storage |
shrink_length |
This is used by capacity = None storages to shrink the length of the storage |
get |
Read one or more samples from storage. |
set |
Write one or more samples into storage. |
open_segment_writer |
A writer that builds one segment of this storage off to the side; see |
clear |
Clear all data inside the storage and set the length to 0 if the storage has unlimited capacity. |
dumps |
Dumps the storage to the specified path. |
close |
|
Attributes:
| Name | Type | Description |
|---|---|---|
single_file_ext |
Optional[str]
|
The total capacity (number of single instances) of the storage. |
capacity |
Optional[int]
|
The cache path for the storage. |
cache_filename |
Optional[Union[str, PathLike]]
|
Can the storage instance be safely used in multiprocessing environments after creation? |
is_multiprocessing_safe |
bool
|
Is the storage mutable? If False, the storage is read-only. |
is_mutable |
bool
|
|
backend |
ComputeBackend[BArrayType, BDeviceType, BDtypeType, BRNGType]
|
|
device |
Optional[BDeviceType]
|
|
single_instance_space |
|
|
has_open_segment |
bool
|
|
pending_segment_length |
int
|
|
single_file_ext
class-attribute
instance-attribute
¶
The total capacity (number of single instances) of the storage. If None, the storage has unlimited capacity.
capacity
class-attribute
instance-attribute
¶
The cache path for the storage. If None, the storage will not use caching.
cache_filename
class-attribute
instance-attribute
¶
Can the storage instance be safely used in multiprocessing environments after creation? If True, the storage can be used in multiprocessing environments.
is_multiprocessing_safe
class-attribute
instance-attribute
¶
Is the storage mutable? If False, the storage is read-only.
create
classmethod
¶
create(
single_instance_space: Space[BatchT, BDeviceType, BDtypeType, BRNGType],
*args,
capacity: Optional[int],
cache_path: Optional[Union[str, PathLike]] = None,
multiprocessing: bool = False,
**kwargs,
) -> SpaceStorage[BatchT, BArrayType, BDeviceType, BDtypeType, BRNGType]
Create a new storage instance for samples from single_instance_space.
load_from
classmethod
¶
load_from(
path: Union[str, PathLike],
single_instance_space: Space[BatchT, BDeviceType, BDtypeType, BRNGType],
*,
capacity: Optional[int] = None,
read_only: bool = True,
multiprocessing: bool = False,
**kwargs,
) -> SpaceStorage[BatchT, BArrayType, BDeviceType, BDtypeType, BRNGType]
Restore a storage instance previously written to disk.
append
¶
Append one structured sample to the storage.
Appended values may be immediately visible or only visible after
mark_segment_end() depending on the storage implementation.
extend_length
¶
This is used by capacity = None storages to extend the length of the storage If this is called on a storage with a fixed capacity, we will simply ignore the call.
shrink_length
¶
This is used by capacity = None storages to shrink the length of the storage If this is called on a storage with a fixed capacity, we will simply ignore the call.
get
abstractmethod
¶
get(index: Union[IndexableType, BArrayType]) -> BatchT
Read one or more samples from storage.
set
abstractmethod
¶
set(index: Union[IndexableType, BArrayType], value: BatchT) -> None
Write one or more samples into storage.
open_segment_writer
¶
open_segment_writer() -> SegmentWriter
A writer that builds one segment of this storage off to the side; see SegmentWriter.
Unlike mark_segment_start() / append(), any number of writers can be open at a time.
The default keeps the samples in memory until the commit; storages override it to stream instead.
clear
¶
Clear all data inside the storage and set the length to 0 if the storage has unlimited capacity. For storages with fixed capacity, this should reset the storage to its initial state.
dumps
abstractmethod
¶
Dumps the storage to the specified path.
This is used for storages that have a single file extension (e.g. .pt for PyTorch).