Browser exercise · No setup
Did the robot complete the task?
Read the first three Arena trials. A lift is an intermediate milestone. The task succeeds only when the cube is left in the bowl.
| Trial | Lifted | Placed in bowl |
|---|---|---|
| 1 | Yes | No |
| 2 | Yes | No |
| 3 | No | No |
Compare with the other recording
Placement succeeded in 0 of 3 trials with the 24-second settings and 2 of 2 with the 70-second settings. The starting layouts also differed, so these runs do not isolate the effect of time.
How it works
- InputWhat the robot sees
camera views + joint state + instructionArena packages the simulated observations for the GR00T DROID checkpoint.
- ProcessWhat the policy predicts
sequence of joint and gripper targetsThe runner executes a portion of the prediction before requesting a new one.
- OutputWhat the simulator records
episode_results_*.jsonlIsaac Sim applies the targets. Arena writes one trial per line, with separate lift and placement events.
The robot moves, the cameras see the new scene, and the policy gets a new observation.
How does a camera image become a robot action?
Isaac Sim produces the virtual camera views and robot state. Arena packages them in the format expected by the GR00T N1.6-DROID checkpoint and adds the instruction to put the cube in the bowl.
| Data passed along | What it tells the next stage | Consumer |
|---|---|---|
| Camera views | Where the hand, cube, and bowl appear relative to each other. | GR00T vision input |
| Robot state and instruction | How the arm is posed and what task it should complete. | DROID state and language inputs |
| Predicted action sequence | Arm-joint and gripper targets for several future control steps. | Isaac Sim controller through the Arena adapter |
What happens inside GR00T?
The lab uses the DROID checkpoint, a version of GR00T N1.6-3B fine-tuned on DROID. A vision-language model first turns the camera views and instruction into numeric representations. Conditioned on those representations and the robot state, the action head starts with sampled noise and updates it iteratively to produce a sequence of continuous commands. The training method for learning that transformation is called flow matching. Those internal updates produce one action prediction. The steps in the log below count actions applied in the simulated control loop. NVIDIA N1.6 architecture · DROID checkpoint
The policy proposes a sequence of actions. The runner applies a configured portion, reads a fresh observation, and queries the policy again. The camera views and joint state change after the arm moves, so subsequent actions can respond to that change. Match the arm’s joint order and normalization to the checkpoint and adapter. Arena GR00T walkthrough · GR00T modality configuration
Read “lift” and “placement” in a saved episode
Here is a projection of a few fields from the first 70-second trial. JSONL stores one complete trial per line. Within the trial, progress.events records which checks passed and at which control steps.
{
"success": true,
"episode_length": 521,
"progress": { "events": [{ "step": 1 }, { "step": 442 }, { "step": 521 }] }
}
Step 1 marks the objects settling, step 442 marks the cube reaching the lift height, and step 521 marks placement on the destination. Convert steps to time using the control rate. The checker uses 15 Hz, so 442 ÷ 15 ≈ 29.47 seconds. In the shorter 24-second runs, a successful lift could still end without a placement. Check for object_on_destination and the final outcome when you replay your own episode. Result checker
First time? Get the lab files
Open a terminal on the Isaac Sim machine for GPU labs, or on your own computer for CPU-only steps. Press Enter after each line. You can copy a multi-line command block as a whole.
cd "$HOME"
git clone https://github.com/james-joobs/awesome-nvidia-physical-ai.git
cd awesome-nvidia-physical-ai
pwd
This example clones into your home folder. If you already cloned the repo, skip git clone and enter your project folder. In each new terminal, check that pwd ends in awesome-nvidia-physical-ai so commands can find the labs/ files.
Run it
Before you start · Use an RTX Linux machine with a desktop session and uv. Follow the Isaac Sim setup and prepare the native uv runtime in the official Arena GR00T guide. This example uses Arena c7edf3a and GR00T submodule e29d8fc. The commands assume both repos are in your home folder. If you cloned them elsewhere, change the cd paths and ARENA_DIR.
1. Start GR00T in terminal A
Move into Arena, fetch its GR00T submodule, and start the model. You can paste the whole block. The first run downloads the model files. Keep this terminal open once the model is ready.
cd "$HOME/IsaacLab-Arena"
git submodule update --init submodules/Isaac-GR00T
cd submodules/Isaac-GR00T
uv run python gr00t/eval/run_gr00t_server.py --model-path nvidia/GR00T-N1.6-DROID \
--embodiment-tag OXE_DROID --device cuda --host 127.0.0.1 --port 5555
2. Run the robot in terminal B on the same machine
Activate Arena’s Python environment, then enter this repository to run its script. The 70 selects the 70-second trial configuration. The Isaac Sim window opens and the arm starts moving. Arena asks the model in terminal A for actions at 127.0.0.1:5555.
ARENA_DIR="$HOME/IsaacLab-Arena"
source "$ARENA_DIR/.venv/bin/activate"
cd "$HOME/awesome-nvidia-physical-ai"
bash labs/gr00t-placement/run_arena.sh "$ARENA_DIR" 70 ./runs/gr00t
If Arena gets a 404 from its staging asset URL, apply the asset URL patch in the Arena checkout with git apply --unidiff-zero, then retry step 2.
3. Read your run log in terminal B
When the window closes and the terminal is ready for input again, run the commands below. If the filename differs, use the path printed by find in the second line. lift is when the robot picked the cube up. placed is when it left the cube in the bowl. no means the trial did not reach that milestone.
find ./runs/gr00t -name 'episode_results_*.jsonl'
python labs/gr00t-placement/check_result.py ./runs/gr00t/droid_pnp_gr00t/episode_results_rebuild0.jsonl
Record your own live GUI run
Keep the policy server in terminal A running. In terminal B, use the command below instead of step 2 to save the native Isaac Sim window, an initial USD scene, and object-position samples. This recorder needs an X11 desktop, ffmpeg, and xwininfo; it does not replay a previously successful trajectory.
export GR00T_GUI_FFMPEG="$(command -v ffmpeg)"
bash labs/gr00t-placement/run_recorded.sh "$ARENA_DIR" 70 ./runs/gr00t-gui
python labs/gr00t-placement/check_result.py \
./runs/gr00t-gui/evaluation/droid_pnp_gr00t/episode_results_rebuild0.jsonl
The GUI recorder writes gr00t-gui.mp4, gr00t-arena.usda, viewport-timing.jsonl, and capture.json. The USD captures the initial scene and can reference official external assets; it does not include the policy weights or a standalone replay of the robot’s actions. To inspect actual model inputs and action arrays too, use the optional policy recorder in the GR00T Python environment, with GR00T_MODEL_PATH set to the checkpoint and GR00T_POLICY_OUTPUT set to a new output directory.
Check the result
The three 24-second trials recorded two lifts and zero placements. The two 70-second trials placed the cube at 34.73 and 38.27 simulated seconds. The starting layouts also differed, so these runs cannot isolate the effect of the time limit.
November 19 rerun: lifting did not complete the task
A fresh run on the RTX 5090 used the real GR00T N1.6-DROID policy, Arena c7edf3a, GR00T e29d8fc, and Isaac Sim 6.1.0.0. The checkpoint revision was ae3ebe8d288971ac53aa30c756ea5cba0f52611b. Both 70-second trials reached the lift predicate, but neither passed the destination-placement predicate. The process completed normally; the task result was 0 successful placements out of 2 trials.
| Trial | Lift time, simulated | Placement | Total control steps |
|---|---|---|---|
| 1 | 55.87 s | Not reached | 1,050 |
| 2 | 28.47 s | Not reached | 1,050 |
Watch the new attempt: a 20-second viewport crop from trial 2, played at 2× speed. It shows a lift followed by a transfer past the bowl. The complete episode records retain both failed outcomes separately from the earlier successful run. Starting layouts differed between runs, so these few trials do not establish a general success rate or identify the cause of failure.
The policy recorder received two RGB camera inputs shaped [1, 1, 180, 320, 3]: batch, time, height, width, and RGB channels. Each camera supplied one 320×180 frame. It produced joint_position shaped [1, 32, 7] plus gripper_position shaped [1, 32, 1]. The action axes are batch, action horizon, and controlled joints; 32 is the predicted action sequence length, not the episode length.
python3 labs/gr00t-placement/check_result.py labs/gr00t-placement/recorded-2026-11-19.jsonl
Replay the checks without a GPU
Run both commands from the repository root. Python reads the bundled 24-second recording and 70-second recording.
python3 labs/gr00t-placement/check_result.py labs/gr00t-placement/recorded-24s.jsonl
python3 labs/gr00t-placement/check_result.py labs/gr00t-placement/recorded-70s.jsonl
Example run and recording
A lift is not yet a placement
November 19 live GR00T run, shown at 2× speed. Both new trials lifted the cube but neither placed it in the bowl.
Source code
droid-70s.yaml
View files on GitHub# Copyright (c) 2026, The Isaac Lab Arena Project Developers (https://github.com/isaac-sim/IsaacLab-Arena/blob/main/CONTRIBUTORS.md).
# All rights reserved.
#
# SPDX-License-Identifier: Apache-2.0
# Adapted from the Arena DROID example to record two 70-second trials.
runs:
droid_pnp_gr00t:
environment:
type: pick_and_place_maple_table
enable_cameras: true
episode_length_s: 70.0
embodiment: droid_abs_joint_pos
pick_up_object: rubiks_cube_hot3d_robolab
destination_location: bowl_ycb_robolab
hdr: home_office_robolab
environment_builder:
language_instruction: Pick up the Rubik's cube and place it in the bowl.
policy:
type: isaaclab_arena_gr00t.policy.gr00t_remote_closedloop_policy.Gr00tRemoteClosedloopPolicy
policy_config_yaml_path: isaaclab_arena_gr00t/policy/config/droid_manip_gr00t_closedloop_config.yaml
policy_device: cuda:0
remote_host: 127.0.0.1
remote_port: 5555
rollout_limit:
num_episodes: 2
droid-24s.yaml
View files on GitHub# Copyright (c) 2026, The Isaac Lab Arena Project Developers (https://github.com/isaac-sim/IsaacLab-Arena/blob/main/CONTRIBUTORS.md).
# All rights reserved.
#
# SPDX-License-Identifier: Apache-2.0
# Adapted from the Arena DROID example with a 24-second episode limit.
runs:
droid_pnp_gr00t:
environment:
type: pick_and_place_maple_table
enable_cameras: true
episode_length_s: 24.0
embodiment: droid_abs_joint_pos
pick_up_object: rubiks_cube_hot3d_robolab
destination_location: bowl_ycb_robolab
hdr: home_office_robolab
environment_builder:
language_instruction: Pick up the Rubik's cube and place it in the bowl.
policy:
type: isaaclab_arena_gr00t.policy.gr00t_remote_closedloop_policy.Gr00tRemoteClosedloopPolicy
policy_config_yaml_path: isaaclab_arena_gr00t/policy/config/droid_manip_gr00t_closedloop_config.yaml
policy_device: cuda:0
remote_host: 127.0.0.1
remote_port: 5555
rollout_limit:
num_episodes: 3
run_arena.sh
View files on GitHub#!/usr/bin/env bash
set -euo pipefail
if [[ $# -lt 3 || ! "$2" =~ ^(24|70)$ ]]; then
printf 'Usage: bash %s ARENA_CHECKOUT 24|70 OUTPUT_DIRECTORY [ARENA_OVERRIDES...]\n' "$0" >&2
exit 2
fi
here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)"
config="$here/droid-${2}s.yaml"
mkdir -p "$3"
output="$(cd "$3" && pwd -P)"
cd "$1"
shift 3
# Start the official loopback GR00T server before running this script.
python -m isaaclab_arena.evaluation.experiment_runner --viz kit --record_camera_video \
--experiment_config "$config" --experiment_output_directory "$output" "$@"
check_result.py
View files on GitHub"""Read Arena's episode JSONL and distinguish lifting from stable placement."""
import argparse
import json
from pathlib import Path
def read_episodes(path):
episodes = []
for line in path.read_text().splitlines():
if not line.strip():
continue
row = json.loads(line)
milestones = {event["predicate_name"]: event["step"] for event in row["progress"]["events"]}
lift = next((step for name, step in milestones.items() if "object_is_above_height" in name), None)
placed = next((step for name, step in milestones.items() if "object_on_destination" in name), None)
if bool(row["success"]) != (placed is not None):
raise ValueError(f"Arena success and placement event disagree in episode {len(episodes) + 1}")
episodes.append({"steps": row["episode_length"], "lift_step": lift, "placement_step": placed})
if not episodes:
raise ValueError("No episodes in the Arena report")
return episodes
if __name__ == "__main__":
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("episode_report", type=Path)
parser.add_argument("--rate", type=float, default=15, help="Arena environment control steps per second")
args = parser.parse_args()
if args.rate <= 0:
parser.error("--rate must be positive")
for index, episode in enumerate(read_episodes(args.episode_report), start=1):
lifted = episode["lift_step"]
placed = episode["placement_step"]
print(f"Trial {index}: {episode['steps']} steps,"
f" lift {format(lifted / args.rate, '.2f') + ' s' if lifted is not None else 'no'},"
f" placed {format(placed / args.rate, '.2f') + ' s' if placed is not None else 'no'}")
arena-production-assets.patch
View files on GitHubdiff --git a/isaaclab_arena/assets/nucleus.py b/isaaclab_arena/assets/nucleus.py
index a7da76a..7d8ac2c 100644
--- a/isaaclab_arena/assets/nucleus.py
+++ b/isaaclab_arena/assets/nucleus.py
@@ -14 +14 @@ ARENA_STAGING_NUCLEUS_DIR: str = "omniverse://isaac-dev.ov.nvidia.com/Isaac/Isaa
-ARENA_NUCLEUS_DIR: str = ISAACLAB_NUCLEUS_DIR.replace("omniverse-content-production", "omniverse-content-staging")
+ARENA_NUCLEUS_DIR: str = ISAACLAB_NUCLEUS_DIR
record_gui.py
View files on GitHub"""Record the native Isaac Sim window during a real GR00T/Arena rollout.
Use run_recorded.sh after starting the official GR00T policy server. The USD
snapshot preserves the scene setup; the MP4 records the live policy rollout,
not an animation authored from a known successful trajectory.
"""
from datetime import datetime, timezone
from importlib.metadata import version
import json
import os
from pathlib import Path
import signal
import subprocess
import time
from isaaclab_arena.evaluation import experiment_runner
output = Path(os.environ["GR00T_GUI_OUTPUT"]).resolve()
output.mkdir(parents=True, exist_ok=True)
original_enter = experiment_runner.SimulationAppContext.__enter__
def window_id():
listing = subprocess.check_output(["xwininfo", "-root", "-tree"], text=True)
for line in listing.splitlines():
if '"Isaac Lab ' in line and "mutter-x11-frames" not in line:
return str(int(line.split()[0], 16))
raise RuntimeError("The native Isaac Sim GUI window was not found")
class ObservedEnvironment:
def __init__(self, original):
self.original = original
self.camera = None
self.started = None
self.frame_number = 0
self.window = None
self.wall_log = (output / "viewport-timing.jsonl").open("w", buffering=1)
self.ffmpeg_log = (output / "viewport-ffmpeg.log").open("w")
def __getattr__(self, name):
return getattr(self.original, name)
def reset(self, *args, **kwargs):
observation = self.original.reset(*args, **kwargs)
if self.camera is not None:
return observation
from omni.usd import get_context
stage = get_context().get_stage()
assert stage and stage.GetRootLayer().Export(str(output / "gr00t-arena.usda"))
self.window = window_id()
self.camera = subprocess.Popen([
os.environ["GR00T_GUI_FFMPEG"], "-hide_banner", "-loglevel", "warning", "-y",
"-f", "x11grab", "-framerate", "30", "-draw_mouse", "0",
"-window_id", self.window, "-i", f"{os.environ['DISPLAY']}.0", "-an",
"-c:v", "libx264", "-preset", "ultrafast", "-crf", "18", "-pix_fmt", "yuv420p",
"-movflags", "+faststart", str(output / "gr00t-gui.mp4"),
], stdin=subprocess.DEVNULL, stdout=self.ffmpeg_log, stderr=subprocess.STDOUT)
self.started = time.monotonic()
self.started_utc = datetime.now(timezone.utc).isoformat()
print("GR00T_GUI_START: scene exported; recording the live policy rollout", flush=True)
self.sample()
return observation
def sample(self):
env = self.original.unwrapped
self.wall_log.write(json.dumps({
"wall_s": time.monotonic() - self.started,
"global_step": self.frame_number,
"episodes_completed": int(env.completed_episode_count),
"positions_m": {name: asset.data.root_pos_w[0].tolist()
for name, asset in env.scene.rigid_objects.items()},
}) + "\n")
def step(self, actions):
result = self.original.step(actions)
self.frame_number += 1
if self.frame_number % 5 == 0:
self.sample()
return result
def finish(self):
try:
if self.camera is not None:
self.sample()
if self.camera.poll() is None:
self.camera.send_signal(signal.SIGINT)
self.camera.wait(timeout=20)
video = output / "gr00t-gui.mp4"
if not video.exists() or video.stat().st_size < 50_000:
raise RuntimeError("The GUI recording is missing or unexpectedly small")
(output / "capture.json").write_text(json.dumps({
"application": "Isaac Sim GUI launched through Isaac Lab-Arena",
"isaacsim_version": version("isaacsim"),
"recording_mode": "live_policy_native_x11_window",
"video": video.name,
"scene": "gr00t-arena.usda",
"window_id": self.window,
"started_utc": self.started_utc,
"ended_utc": datetime.now(timezone.utc).isoformat(),
"wall_seconds": round(time.monotonic() - self.started, 3),
"control_steps": self.frame_number,
"ffmpeg_exit_code": self.camera.returncode,
"bytes": video.stat().st_size,
}, indent=2) + "\n")
finally:
self.ffmpeg_log.close()
self.wall_log.close()
def enter(context):
value = original_enter(context)
from isaaclab_arena.evaluation import run_execution
rollout = run_execution.rollout_policy
def observed_rollout(env, policy, *args, **kwargs):
instrumented = ObservedEnvironment(env)
try:
return rollout(instrumented, policy, *args, **kwargs)
finally:
instrumented.finish()
run_execution.rollout_policy = observed_rollout
return value
if __name__ == "__main__":
experiment_runner.SimulationAppContext.__enter__ = enter
try:
experiment_runner.main()
except BaseException:
# Some Kit shutdown paths mask a Python failure with exit code zero.
(output / "capture-failed").touch()
raise
record_policy.py
View files on GitHub"""Run the official GR00T policy and save one real observation/action per reset.
Set GR00T_MODEL_PATH to a prepared checkpoint and GR00T_POLICY_OUTPUT to the
output directory. No robot actions are synthesized by this recorder.
"""
import json
import os
from pathlib import Path
import time
import numpy as np
from PIL import Image
import torch
from gr00t.data.embodiment_tags import EmbodimentTag
from gr00t.policy.gr00t_policy import Gr00tPolicy
from gr00t.policy.server_client import PolicyServer
output = Path(os.environ["GR00T_POLICY_OUTPUT"]).resolve()
output.mkdir(parents=True, exist_ok=True)
policy = Gr00tPolicy(
embodiment_tag=EmbodimentTag.OXE_DROID,
model_path=os.environ["GR00T_MODEL_PATH"],
device="cuda",
strict=True,
)
print(f"GR00T N1.6-DROID | torch {torch.__version__} | {torch.cuda.get_device_name(0)}", flush=True)
get_action = policy.get_action
reset = policy.reset
episode = 0
count = 0
saved = set()
def record_reset(*args, **kwargs):
global episode
response = reset(*args, **kwargs)
episode += 1
return response
def record_action(observation, options=None):
global count
started = time.perf_counter()
action, info = get_action(observation, options)
elapsed = 1000 * (time.perf_counter() - started)
count += 1
arrays = {key: np.asarray(values) for key, values in action.items()}
if not all(np.isfinite(a).all() and a.size > 0 for a in arrays.values()):
raise RuntimeError("GR00T returned an invalid action")
if count <= 3 or count % 25 == 0:
print(f"ACTION {count} | reset {episode} | {elapsed:.1f} ms | {list(action)}", flush=True)
if episode not in saved:
save_dir = output / f"request-{count:04d}"
save_dir.mkdir(parents=True, exist_ok=True)
cameras = {}
for index, (name, values) in enumerate(observation.get("video", {}).items()):
array = np.asarray(values)
if array.ndim != 5 or array.shape[-1] != 3:
continue
frame = array[0, -1]
if frame.dtype != np.uint8:
frame = (frame * 255 if frame.max() <= 1.0 else frame).clip(0, 255).astype(np.uint8)
filename = f"camera-{index + 1}.png"
Image.fromarray(frame).save(save_dir / filename)
cameras[name] = {"file": filename, "shape": list(array.shape)}
record = {
"model": "GR00T N1.6-DROID",
"observation_origin": "Isaac Lab-Arena DROID simulator",
"request": count,
"server_reset_count": episode,
"instruction": observation.get("language", {}),
"cameras": cameras,
"observed_state": {k: np.asarray(v).tolist() for k, v in observation.get("state", {}).items()},
"actions": {k: v.tolist() for k, v in arrays.items()},
"action_shapes": {k: list(v.shape) for k, v in arrays.items()},
"server_inference_ms": round(elapsed, 2),
}
(save_dir / "observation-action.json").write_text(json.dumps(
record, indent=2, default=lambda value: np.asarray(value).tolist()) + "\n")
saved.add(episode)
return action, info
policy.get_action = record_action
policy.reset = record_reset
PolicyServer(policy, host="127.0.0.1", port=int(os.environ.get("GR00T_POLICY_PORT", "5555"))).run()
run_recorded.sh
View files on GitHub#!/usr/bin/env bash
set -euo pipefail
if [[ $# -lt 3 || ! "$2" =~ ^(24|70)$ ]]; then
printf 'Usage: bash %s ARENA_CHECKOUT 24|70 OUTPUT_DIRECTORY [ARENA_OVERRIDES...]\n' "$0" >&2
exit 2
fi
here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)"
config="$here/droid-${2}s.yaml"
mkdir -p "$3"
export GR00T_GUI_OUTPUT="$(cd "$3" && pwd -P)"
export GR00T_GUI_FFMPEG="${GR00T_GUI_FFMPEG:-$(command -v ffmpeg)}"
if [[ ! -x "$GR00T_GUI_FFMPEG" ]]; then
printf 'Set GR00T_GUI_FFMPEG to an executable ffmpeg binary.\n' >&2
exit 2
fi
command -v xwininfo >/dev/null
: "${DISPLAY:?Run this from a Linux desktop session with DISPLAY set.}"
cd "$1"
shift 3
# Start the official loopback GR00T server separately before running this script.
"${GR00T_ARENA_PYTHON:-python}" -u "$here/record_gui.py" --viz kit --record_camera_video \
--experiment_config "$config" --experiment_output_directory "$GR00T_GUI_OUTPUT/evaluation" "$@"
if [[ -e "$GR00T_GUI_OUTPUT/capture-failed" || ! -s "$GR00T_GUI_OUTPUT/capture.json" ]]; then
printf 'The GUI run did not finish cleanly; keep the logs and inspect the failure.\n' >&2
exit 1
fi

