unienv_data.samplers.step_sampler¶
Classes:
| Name | Description |
|---|---|
StepSampler |
Samples mini-batches of individual steps (uniformly, with replacement) from |
StepSampler
¶
StepSampler(
data: BatchBase[BatchT, BArrayType, BDeviceType, BDtypeType, BRNGType],
batch_size: int,
seed: Optional[int] = None,
device: Optional[BDeviceType] = None,
)
Bases: BatchSampler[BatchT, BArrayType, BDeviceType, BDtypeType, BRNGType, BatchT, BArrayType, BDeviceType, BDtypeType, BRNGType]
Samples mini-batches of individual steps (uniformly, with replacement) from data.
Performance notes:
- For data held in RAM (e.g. an in-memory
ReplayBufferand views over it), sampling in-process is usually faster thanMultiprocessingSampler: worker processes only pay off when producing a batch is expensive (decoding, transformations, storage reads), since every batch is copied to the main process. sample_flat()/get_flattened_at()flatten a sample into one array of a common dtype, souint8images next tofloat32fields are cast tofloat32(4x the bytes and a conversion per batch). Prefersample()for image data.- Read single fields through
data.get_column(keys)rather than sampling whole samples.
Methods:
| Name | Description |
|---|---|
get_flattened_at |
|
get_flattened_at_with_metadata |
|
get_at |
|
get_at_with_metadata |
|
set_flattened_at |
Overwrite existing samples using flattened data. |
append_flattened |
Append one flattened sample to the batch. |
extend_flattened |
Append a batched block of flattened samples. |
set_at |
Overwrite existing samples using structured data. |
remove_at |
Remove one or more samples from the batch. |
append |
Append one structured sample to the batch. |
extend |
Append a batched block of structured samples. |
extend_from |
Copy data from another batch in bounded-size chunks. |
get_slice |
Create a lazy view over a subset of indices. |
get_column |
Create a lazy view over a nested field inside each sample. |
close |
|
manual_seed |
Reset sampler RNG state, including the optional data-index RNG. |
sample_index |
Draw random indices for one batch. |
sample_flat |
Sample a batch and return it in flattened form. |
sample_flat_with_metadata |
Sample a flattened batch together with optional metadata. |
sample |
Sample a batch in its structured form. |
sample_with_metadata |
Sample a structured batch together with optional metadata. |
epoch_iter |
Iterate once over a random permutation of the source data. |
epoch_iter_with_metadata |
Iterate once over shuffled structured batches plus metadata. |
epoch_flat_iter |
Iterate once over shuffled batches in flattened form. |
epoch_flat_iter_with_metadata |
Iterate once over shuffled flattened batches plus metadata. |
Attributes:
| Name | Type | Description |
|---|---|---|
data |
|
|
batch_size |
|
|
data_rng |
|
|
backend |
ComputeBackend[BArrayType, BDeviceType, BDtypeType, BRNGType]
|
|
device |
Optional[BDeviceType]
|
|
is_mutable |
bool
|
|
single_space |
|
|
single_metadata_space |
|
|
rng |
Optional[SamplerRNGType]
|
|
sampled_space |
Space[SamplerBatchT, SamplerDeviceType, SamplerDtypeType, SamplerRNGType]
|
|
sampled_metadata_space |
Optional[DictSpace[SamplerDeviceType, SamplerDtypeType, SamplerRNGType]]
|
|
data_rng
instance-attribute
¶
sampled_space
property
¶
sampled_space: Space[SamplerBatchT, SamplerDeviceType, SamplerDtypeType, SamplerRNGType]
sampled_metadata_space
property
¶
sampled_metadata_space: Optional[
DictSpace[SamplerDeviceType, SamplerDtypeType, SamplerRNGType]
]
set_flattened_at
¶
set_flattened_at(idx: Union[IndexableType, BArrayType], value: BArrayType) -> None
Overwrite existing samples using flattened data.
append_flattened
¶
Append one flattened sample to the batch.
extend_flattened
¶
Append a batched block of flattened samples.
set_at
¶
set_at(idx: Union[IndexableType, BArrayType], value: BatchT) -> None
Overwrite existing samples using structured data.
remove_at
¶
remove_at(idx: Union[IndexableType, BArrayType]) -> None
Remove one or more samples from the batch.
extend_from
¶
extend_from(
other: BatchBase[BatchT, BArrayType, BDeviceType, BDtypeType, BRNGType],
chunk_size: int = 8,
tqdm: bool = False,
) -> None
Copy data from another batch in bounded-size chunks.
get_slice
¶
get_slice(
idx: Union[IndexableType, BArrayType],
) -> BatchBase[BatchT, BArrayType, BDeviceType, BDtypeType, BRNGType]
Create a lazy view over a subset of indices.
get_column
¶
get_column(
nested_keys: Sequence[str],
) -> BatchBase[Any, BArrayType, BDeviceType, BDtypeType, BRNGType]
Create a lazy view over a nested field inside each sample.
manual_seed
¶
Reset sampler RNG state, including the optional data-index RNG.
sample_flat_with_metadata
¶
sample_flat_with_metadata() -> Tuple[SamplerArrayType, Optional[Dict[str, Any]]]
Sample a flattened batch together with optional metadata.
sample_with_metadata
¶
sample_with_metadata() -> Tuple[SamplerBatchT, Optional[Dict[str, Any]]]
Sample a structured batch together with optional metadata.
epoch_iter
¶
epoch_iter() -> Iterator[SamplerBatchT]
Iterate once over a random permutation of the source data.
epoch_iter_with_metadata
¶
epoch_iter_with_metadata() -> Iterator[Tuple[SamplerBatchT, Optional[Dict[str, Any]]]]
Iterate once over shuffled structured batches plus metadata.
epoch_flat_iter
¶
epoch_flat_iter() -> Iterator[SamplerArrayType]
Iterate once over shuffled batches in flattened form.
epoch_flat_iter_with_metadata
¶
epoch_flat_iter_with_metadata() -> Iterator[
Tuple[SamplerArrayType, Optional[Dict[str, Any]]]
]
Iterate once over shuffled flattened batches plus metadata.