unienv_data.storages.video_storage¶
PyAvVideoReader(backend: ComputeBackend, filename: str, buffer_pixel_format: Optional[str] = None, hwaccel: Optional[Union[HWAccel, Literal['auto']]] = None, seek_mode: Literal['exact', 'approximate'] = 'exact', device: Optional[BDeviceType] = None)
keyframe_indices() -> np.ndarray
Frame indices of the keyframes, from a demux pass over the packets (no decoding).
TorchCodecVideoReader(backend: ComputeBackend, filename: str, buffer_pixel_format: Optional[str] = None, hwaccel: Optional[Union[HWAccel, Literal['auto']]] = None, seek_mode: Literal['exact', 'approximate'] = 'exact', device: Optional[BDeviceType] = None)
VideoStorage(single_instance_space: BoxSpace[BArrayType, BDeviceType, BDtypeType, BRNGType], cache_filename: Union[str, PathLike], seek_mode: Literal['exact', 'approximate'] = 'exact', hardware_acceleration: Optional[Union[Any, Literal['auto']]] = 'auto', codec: Union[str, Literal['auto']] = 'auto', decode_backend: Literal['torchcodec', 'pyav', 'auto'] = 'auto', cache_decoders: bool = False, file_ext: str = 'mp4', file_pixel_format: Optional[str] = None, buffer_pixel_format: str = 'rgb24', fps: int = 15, encoder_options: Optional[Dict[str, str]] = None, mutable: bool = True, capacity: Optional[int] = None, length: int = 0)
Bases: EpisodeStorageBase[BArrayType, BArrayType, BDeviceType, BDtypeType, BRNGType]
A storage for RGB or depth video data using video files
If encoding RGB video
- Set buffer_pixel_format to rgb24
- Set file_pixel_format to None (especially when running with nvenc codec)
- Set file_ext to anything you like (e.g., "mp4", "avi", "mkv", etc.)
If encoding depth video
- Set buffer_pixel_format to gray16le (You can use rescale transform inside a TransformedStorage to convert depth values to this format, where dtype should be np.uint16) - if in meters, set min to 0 and max to 65.535 as the multiplication factor is 1000 (i.e., depth in mm)
- Set file_pixel_format to gray16le
- Set codec to a lossless codec that supports gray16le, e.g., ffv1
- Set file_ext to mkv
encoder_options are passed to the encoder (e.g. {"crf": "28", "g": "15"}) and persisted in the metadata,
so a loaded storage keeps encoding new segments the same way.
Random access: reading frame i decodes from the keyframe at or before i, so the cost of a random read
grows with the GOP size (keyframe interval). Encoders default to long GOPs (libx264: 250 frames, i.e. one
keyframe per episode); for buffers sampled at random positions set a short one, e.g.
encoder_options={"g": "15"} (larger files, much cheaper random reads). Sequential and dense reads are not
affected, and sparse reads within one file decode forward instead of seeking when no keyframe lies between
two requested frames (the PyAV reader indexes each file's keyframes once).
instance-attribute
¶
encoder_options = None if encoder_options is None else {str(k): str(v) for k, v in encoder_options.items()}
classmethod
¶
create(single_instance_space: BoxSpace[BArrayType, BDeviceType, BDtypeType, BRNGType], *args, seek_mode: Literal['exact', 'approximate'] = 'exact', hardware_acceleration: Optional[Union[Any, Literal['auto']]] = 'auto', codec: Union[str, Literal['auto']] = 'auto', decode_backend: Literal['torchcodec', 'pyav', 'auto'] = 'auto', cache_decoders: bool = False, file_ext: str = 'mp4', file_pixel_format: Optional[str] = None, buffer_pixel_format: str = 'rgb24', fps: int = 15, encoder_options: Optional[Dict[str, str]] = None, capacity: Optional[int] = None, cache_path: Optional[str] = None, multiprocessing: bool = False, **kwargs) -> VideoStorage[BArrayType, BDeviceType, BDtypeType, BRNGType]
classmethod
¶
load_from(path: Union[str, PathLike], single_instance_space: BoxSpace[BArrayType, BDeviceType, BDtypeType, BRNGType], *, seek_mode: Literal['exact', 'approximate'] = 'exact', hardware_acceleration: Optional[Union[Any, Literal['auto']]] = 'auto', codec: Union[str, Literal['auto']] = 'auto', decode_backend: Literal['torchcodec', 'pyav', 'auto'] = 'auto', cache_decoders: bool = False, capacity: Optional[int] = None, read_only: bool = True, multiprocessing: bool = False, **kwargs) -> VideoStorage[BArrayType, BDeviceType, BDtypeType, BRNGType]
get_from_file(filename: str, index: Union[IndexableType, BArrayType], total_length: int) -> BArrayType
get_start_end_filename_iter() -> Iterable[Tuple[int, int, str]]
Iterate over (start_idx, end_idx, filename) tuples, constructing filenames on the fly.
convert_read_index_to_filenames_and_offsets(index: Union[IndexableType, BArrayType]) -> Tuple[int, List[Tuple[Union[str, os.PathLike], Union[int, BArrayType], Union[int, BArrayType]]]]
Convert an index (which can be an integer, slice, list of integers, or backend array) to a list of filenames and offsets. Each filename corresponds to an episode file, and the offset indicates the position within that episode.
set_to_file_safe(filename: str, batched_value: BatchT)
Atomically write to a temporary file and rename it to the target filename.