Skip to content

unienv_data.integrations.huggingface

Classes:

Name Description
HFAsUniEnvDataset

HFAsUniEnvDataset

HFAsUniEnvDataset(
    hf_dataset: Dataset, space: Space[Any, BDeviceType, BDtypeType, BRNGType]
)

Bases: BatchBase[BatchT, BArrayType, BDeviceType, BDtypeType, BRNGType]

Methods:

Name Description
create
get_at_with_metadata
get_at
get_column
get_flattened_at

Fetch samples as flattened backend arrays.

get_flattened_at_with_metadata

Fetch flattened samples together with optional per-sample metadata.

set_flattened_at

Overwrite existing samples using flattened data.

append_flattened

Append one flattened sample to the batch.

extend_flattened

Append a batched block of flattened samples.

set_at

Overwrite existing samples using structured data.

remove_at

Remove one or more samples from the batch.

append

Append one structured sample to the batch.

extend

Append a batched block of structured samples.

extend_from

Copy data from another batch in bounded-size chunks.

get_slice

Create a lazy view over a subset of indices.

close

Attributes:

Name Type Description
BACKEND_TO_FORMAT_MAP
is_mutable
hf_dataset
single_space
single_metadata_space
backend ComputeBackend[BArrayType, BDeviceType, BDtypeType, BRNGType]
device Optional[BDeviceType]

BACKEND_TO_FORMAT_MAP class-attribute instance-attribute

BACKEND_TO_FORMAT_MAP = {'numpy': 'numpy', 'pytorch': 'torch', 'jax': 'jax'}

is_mutable class-attribute instance-attribute

is_mutable = False

hf_dataset instance-attribute

hf_dataset = hf_dataset

single_space instance-attribute

single_space = single_space

single_metadata_space instance-attribute

single_metadata_space = single_metadata_space

backend property

backend: ComputeBackend[BArrayType, BDeviceType, BDtypeType, BRNGType]

device property

device: Optional[BDeviceType]

create staticmethod

create(
    hf_dataset: Dataset,
    backend: ComputeBackend[BArrayType, BDeviceType, BDtypeType, BRNGType],
    device: Optional[BDeviceType] = None,
) -> HFAsUniEnvDataset[BArrayType, BDeviceType, BDtypeType, BRNGType]

get_at_with_metadata

get_at_with_metadata(idx)

get_at

get_at(idx)

get_column

get_column(nested_keys)

get_flattened_at

get_flattened_at(idx: Union[IndexableType, BArrayType]) -> BArrayType

Fetch samples as flattened backend arrays.

get_flattened_at_with_metadata

get_flattened_at_with_metadata(
    idx: Union[IndexableType, BArrayType],
) -> Tuple[BArrayType, Optional[Dict[str, Any]]]

Fetch flattened samples together with optional per-sample metadata.

set_flattened_at

set_flattened_at(idx: Union[IndexableType, BArrayType], value: BArrayType) -> None

Overwrite existing samples using flattened data.

append_flattened

append_flattened(value: BArrayType) -> None

Append one flattened sample to the batch.

extend_flattened

extend_flattened(value: BArrayType) -> None

Append a batched block of flattened samples.

set_at

set_at(idx: Union[IndexableType, BArrayType], value: BatchT) -> None

Overwrite existing samples using structured data.

remove_at

remove_at(idx: Union[IndexableType, BArrayType]) -> None

Remove one or more samples from the batch.

append

append(value: BatchT) -> None

Append one structured sample to the batch.

extend

extend(value: BatchT) -> None

Append a batched block of structured samples.

extend_from

extend_from(
    other: BatchBase[BatchT, BArrayType, BDeviceType, BDtypeType, BRNGType],
    chunk_size: int = 8,
    tqdm: bool = False,
) -> None

Copy data from another batch in bounded-size chunks.

get_slice

get_slice(
    idx: Union[IndexableType, BArrayType],
) -> BatchBase[BatchT, BArrayType, BDeviceType, BDtypeType, BRNGType]

Create a lazy view over a subset of indices.

close

close() -> None