Skip to content

unienv_data.storages.pytorch

Classes:

Name Description
PytorchTensorStorage

PytorchTensorStorage

PytorchTensorStorage(
    single_instance_space: Union[
        BoxSpace[PyTorchArrayType, PyTorchDeviceType, PyTorchDtypeType, PyTorchRNGType],
        BinarySpace[
            PyTorchArrayType, PyTorchDeviceType, PyTorchDtypeType, PyTorchRNGType
        ],
    ],
    data: Union[Tensor, MemoryMappedTensor],
    mutable: bool = True,
)

Bases: SpaceStorage[PyTorchArrayType, PyTorchArrayType, PyTorchDeviceType, PyTorchDtypeType, PyTorchRNGType]

Methods:

Name Description
create
load_from
get
set
append
clear
dumps
close
get_segments
mark_segment_start
mark_segment_end
abort_segment
extend_length

This is used by capacity = None storages to extend the length of the storage

shrink_length

This is used by capacity = None storages to shrink the length of the storage

open_segment_writer

A writer that builds one segment of this storage off to the side; see SegmentWriter.

Attributes:

Name Type Description
backend
single_file_ext str
data
device Optional[PyTorchDeviceType]
cache_filename Optional[Union[str, PathLike]]
is_mutable bool
is_multiprocessing_safe bool
capacity int
is_memmap bool
single_instance_space
has_open_segment bool
pending_segment_length int

backend class-attribute instance-attribute

single_file_ext class-attribute instance-attribute

single_file_ext: str = '.memmap'

data instance-attribute

data = data

device property

device: Optional[PyTorchDeviceType]

cache_filename property

cache_filename: Optional[Union[str, PathLike]]

is_mutable property

is_mutable: bool

is_multiprocessing_safe property

is_multiprocessing_safe: bool

capacity property

capacity: int

is_memmap property

is_memmap: bool

single_instance_space instance-attribute

single_instance_space = single_instance_space

has_open_segment property

has_open_segment: bool

pending_segment_length property

pending_segment_length: int

create classmethod

create(
    single_instance_space: BoxSpace[
        PyTorchArrayType, PyTorchDeviceType, PyTorchDtypeType, PyTorchRNGType
    ],
    *,
    capacity: Optional[int],
    is_memmap: bool = True,
    cache_path: Optional[str] = None,
    multiprocessing: bool = False,
    memmap_existok: bool = True,
) -> PytorchTensorStorage

load_from classmethod

load_from(
    path: Union[str, PathLike],
    single_instance_space: Union[
        BoxSpace[PyTorchArrayType, PyTorchDeviceType, PyTorchDtypeType, PyTorchRNGType],
        BinarySpace[
            PyTorchArrayType, PyTorchDeviceType, PyTorchDtypeType, PyTorchRNGType
        ],
    ],
    *,
    is_memmap: bool = True,
    capacity: Optional[int] = None,
    read_only: bool = True,
    multiprocessing: bool = False,
    **kwargs,
) -> PytorchTensorStorage

get

get(index: Union[int, slice, Tensor]) -> Tensor

set

set(index: Union[int, slice, Tensor], value: Tensor) -> None

append

append(value: Tensor) -> None

clear

clear() -> None

dumps

dumps(path: Union[str, PathLike]) -> None

close

close()

get_segments

get_segments()

mark_segment_start

mark_segment_start(start_index: int) -> None

mark_segment_end

mark_segment_end() -> None

abort_segment

abort_segment() -> None

extend_length

extend_length(length: int) -> None

This is used by capacity = None storages to extend the length of the storage If this is called on a storage with a fixed capacity, we will simply ignore the call.

shrink_length

shrink_length(length: int) -> None

This is used by capacity = None storages to shrink the length of the storage If this is called on a storage with a fixed capacity, we will simply ignore the call.

open_segment_writer

open_segment_writer() -> SegmentWriter

A writer that builds one segment of this storage off to the side; see SegmentWriter. Unlike mark_segment_start() / append(), any number of writers can be open at a time. The default keeps the samples in memory until the commit; storages override it to stream instead.