Isaac Capture offers two collection routes for robot learning. Isaac Teleop captures what a robot actually observes and executes while an operator drives it. The ego pipeline records what a person sees and reconstructs the person’s motion for downstream processing. The routes share device and data utilities but produce different training inputs.
How does a person’s hand become a robot demonstration?
- InputTrack the operator
device pose + timestampThe device reports hand or controller motion.
- ProcessRetarget the motion
device coordinates → robot commandsRetargeting adapts motion to the robot’s joints and limits.
- OutputSave a demonstration
aligned observations and actionsCheck coordinate frames and timestamp alignment before training.
Retarget the motion into the robot’s controls
A tracking device reports hand or controller position, orientation, and a timestamp. Retargeting converts that motion into robot joint targets or tool poses. If a person moves a hand 3 cm to the right in the tracking frame, first identify the corresponding direction in the robot frame and check whether the target fits its joint limits.
Synchronize video and actions
An illustrative sample might record image_time_s=1.200, state_time_s=1.199, and command_time_s=1.200. These are teaching fields. A shared array index can still pair an image of an open gripper with later joint state if the streams use different clocks. Replay the demonstration and check that the visual observation, state, and command line up. Isaac Capture project
Architecture and data flow
| Component | Inputs, processing and outputs |
|---|---|
| Device plug-ins | Tracking devices provide streams such as hand, wrist or headset poses. The plug-in exposes device data to the runtime so retargeting can be changed without rewriting device integration. |
| Teleop and retargeting | Robot-specific retargeters transform tracking samples into target commands for simulation or hardware. The robot environment supplies its own observations, robot state and control interface. |
| Recording and playback | Capture can record device streams to MCAP for replay. Dataset integrations record environment observations and actions as required by their respective learning workflow, such as LeRobot episodes for the SO-101 example. |
| Ego pipeline | Calibrated head-mounted footage feeds hand reconstruction and optional camera localization. The output is estimated human hand motion in camera or world coordinates for inspection and subsequent retargeting. |

NVIDIA’s overview places shared device, retargeting and data utilities underneath the teleoperation and ego pipelines. The integration boxes show how the two routes can connect to robot development stacks. Official figure source
The Teleop diagram zooms in on the live control path. Follow device tracking through retargeting to the robot integration and distinguish live input from replayed input. Official figure source
Worked example
- Use the official SO-101 integration to calibrate the robot and configure the supported operator device. Define a short, repeatable tabletop task.
- Collect a few episodes with the same camera layout. Record the operator stream separately if you need to reproduce the retargeting path.
- Replay one episode, then inspect synchronized observations, commands and the end of the trial before collecting a larger dataset.
What to verify
- Check pose coordinate frames, orientation conventions, missing device samples and whether gripper commands map in the expected direction.
- Inspect recorded device data and the training dataset independently. Successful MCAP replay alone does not establish camera/action alignment in the dataset.
Official references and terms
The repository defaults to Apache-2.0. EgoPost uses OpenMDW-1.1. CloudXR and optional dependencies have separate terms.