UniEnv
on PyPI · Apache-2.0 · Python 3.10+

Robot environments, policies and data, unified.

Simulators, real robots and datasets describe their data with the same spaces. Policies, buffers and tools written against those spaces work with every source.

$pip install unienv

Three sources

One observation space

observation_spacetorch · cuda:0
robot
 ├ arm
 │ ├ joint_position  (7,)float32rad
 │ ├ joint_velocity  (7,)float32rad/s
 │ ├ eef_position    (3,)float32m
 │ └ eef_quaternion  (4,)float32quat_wxyz
 └ gripper …
external_cam_0
 ├ rgb               (H,W,3)uint8image · HWC
 ├ depth             (H,W)float32depth · m
 └ …
✓Shared leaves keep the same keys, shapes and dtypes on every source.

Unchanged code

PoliciesPolicy.get_action
Replay bufferDictStorage by metadata
Remote servingRemoteServer
Evaluationsame step() loop

A Franka FR3 with a Robotiq gripper: simulated with unienv-genesis, driven live with unienv-droid, or replayed from the DROID dataset. Select a source.

Space metadata

Spaces say what the data means, not just its shape.

Every leaf can carry a small shared vocabulary: kind, units, frame, rotation, names, source, fps. Tools read it, so you stop passing per-key configuration around.

BoxSpace(backend, 0, 255, np.uint8, shape=(480, 640, 3),
         metadata={"kind": "image", "format": "rgb",
                   "layout": "HWC", "source": "wrist_cam",
                   "fps": 15})

BoxSpace(backend, -3.14, 3.14, np.float32, shape=(7,),
         metadata={"kind": "proprio", "units": "rad",
                   "names": ["fr3_joint1", ...]})
rgbimage depthdepth joint_posproprio imagesJPEG or video HDF5everything else

Buffers lay themselves out

dict_storage_kwargs_from_space routes image leaves to image or video storage at the declared fps, and everything else to HDF5.

simreal joint_posradrad✓ eef_posm · basem · base✓ eef_quatquat_wxyzquat_wxyz✓ joint_velrad/sdeg/s✗ spaces_equal_with_metadata(sim, real) → False

Parity you can check

Compare two spaces leaf by leaf, units and frames included, before a policy crosses from sim to the robot.

What's inside

Five layers, each usable on its own.

Use only the layer you need. They share the same space definitions, so they compose.

Spaces

Typed, nested data contracts with metadata. Any space converts between NumPy, PyTorch and JAX, and between devices.

BoxSpaceDictSpacespace.to()

Environments & worlds

Stateful and functional envs, plus worlds assembled from reusable robot, sensor and scene nodes.

EnvFuncEnvWorldWorldNode

Policies

Stateful and functional policies that act against the env's spaces, with wrappers that stack like env wrappers.

PolicyFuncPolicy

Data

Replay buffers on Parquet, HDF5, image and video storages, with samplers and LeRobot / Hugging Face adapters.

ReplayBufferTransformedBatch

Remote

Host any env, world, node or policy over TCP, Unix sockets or WebSocket. Clients get proxies with the same interface.

RemoteServerRemoteClient

Quick start

From a space to a running loop.

  1. Describe your dataBuild the observation and action spaces once.
  2. Wrap the sourceA simulator, robot or dataset adaptor exposes those spaces.
  3. Write everything else oncePolicies, buffers and remote serving read the spaces, not the source.
import numpy as np
from unienv_interface.backends.numpy import NumpyComputeBackend as B
from unienv_interface.space.spaces import BoxSpace, DictSpace
from unienv_data.replay_buffer import ReplayBuffer
from unienv_data.storages.parquet import ParquetStorage

space = DictSpace(B, {
    "obs":    BoxSpace(B, 0.0, 1.0, np.float32, shape=(4,)),
    "action": BoxSpace(B, -1.0, 1.0, np.float32, shape=(2,)),
})

buffer = ReplayBuffer.create(
    ParquetStorage, space,
    cache_path="cache/demo", capacity=10_000,
)
import numpy as np
from unienv_interface.backends.numpy import NumpyComputeBackend as B
from unienv_interface.space import BoxSpace
from unienv_interface.policy import Policy

class RandomPolicy(Policy):
    # the policy declares what it reads and what it outputs
    backend, device = B, None
    observation_space = BoxSpace(B, -np.pi, np.pi, np.float32, shape=(7,))
    action_space = BoxSpace(B, -1.0, 1.0, np.float32, shape=(7,))

    def __init__(self, seed=0):
        self.rng = B.random.random_number_generator(seed)

    # state comes from reset / observe_transition; a random policy keeps none
    def reset(self, context, observation, **kwargs): pass
    def observe_transition(self, action_taken, new_observation): pass

    def get_action(self, **kwargs):
        return self.sample_action()

policy = RandomPolicy()  # or one served remotely: client.policy("pi")
context, obs, info = env.reset(seed=0)
policy.reset(context, obs)
for _ in range(100):
    action = policy.get_action()
    obs, reward, terminated, truncated, info = env.step(action)
    policy.observe_transition(action, obs)
from unienv_interface.remote import RemoteServer, RemoteClient

# on the robot
with RemoteServer() as server:
    server.register("robot", make_env())
    server.serve("tcp://0.0.0.0:7000")

# on the workstation: a proxy with the same Env interface
with RemoteClient.connect("tcp://robot-pc:7000") as client:
    env = client.env("robot")
    context, obs, info = env.reset(seed=42)
from unienv_interface.backends.pytorch import PyTorchComputeBackend

# the same space, now producing torch tensors on the GPU
torch_space = space.to(PyTorchComputeBackend, device="cuda")
rng, sample = torch_space.sample(rng)

Performance

Zero-copy where data moves, tuned where it is stored.

Tensors cross frameworks and processes without being copied, and the storages are benchmarked on robot-sized data: camera frames, proprioception, long episodes.

NumPy PyTorch JAX one buffer · same pointer

Zero-copy across backends

space.data_to(x, backend=...) hands arrays between frameworks through DLPack: the result is a view of the same memory.

0 copiesNumPy ↔ PyTorch on the CPU, PyTorch ↔ JAX on the GPU
env 0 env 1 env 2 env 3 shared memory parentview

Shared-memory vectorized envs

AsyncVecEnv workers write observations into one shared buffer that the parent reads in place; CUDA envs share GPU memory over CUDA IPC.

20–35×more steps/s than through pipes (PyTorch CPU, 8 workers)
dump 10k 128×128 frames before 9.5 s now 0.46 s HDF5 reads ~200× · LeRobot reads ~100×

Storage tuned on robot data

HDF5, Parquet, video and LeRobot storages are profiled on camera frames and long episodes, and read only the rows and columns a batch needs.

~20×faster Parquet dumps with array_encoding="binary"

Speedups are against UniEnv's previous implementation of the same feature, measured on one workstation; absolute times depend on the machine. Reproduce them with the scripts in benchmarks/.

Cite

Using UniEnv in research?

Please cite the software. XBArray is developed as part of UniEnv and shares this citation.