Use GR00T to pick up a cube and place it in a bowl

Watch the gripper hold, carry, open, and leave the cube resting in the bowl.

  • GR00T N1.6-DROID
  • Isaac Lab-Arena
  • Isaac Sim
GR00T puts the cube in the bowl
Recorded runExample run and recording

Browser exercise · No setup

Did the robot complete the task?

Read the first three Arena trials. A lift is an intermediate milestone. The task succeeds only when the cube is left in the bowl.

Recorded trials with a 24-second limit
TrialLiftedPlaced in bowl
1YesNo
2YesNo
3NoNo
Compare with the other recording

Placement succeeded in 0 of 3 trials with the 24-second settings and 2 of 2 with the 70-second settings. The starting layouts also differed, so these runs do not isolate the effect of time.

Next lab · Check camera data without a GPU

How it works

How the data moves
  1. InputWhat the robot seescamera views + joint state + instruction

    Arena packages the simulated observations for the GR00T DROID checkpoint.

  2. ProcessWhat the policy predictssequence of joint and gripper targets

    The runner executes a portion of the prediction before requesting a new one.

  3. OutputWhat the simulator recordsepisode_results_*.jsonl

    Isaac Sim applies the targets. Arena writes one trial per line, with separate lift and placement events.

The robot moves, the cameras see the new scene, and the policy gets a new observation.

How does a camera image become a robot action?

Isaac Sim produces the virtual camera views and robot state. Arena packages them in the format expected by the GR00T N1.6-DROID checkpoint and adds the instruction to put the cube in the bowl.

Data passed along What it tells the next stage Consumer
Camera views Where the hand, cube, and bowl appear relative to each other. GR00T vision input
Robot state and instruction How the arm is posed and what task it should complete. DROID state and language inputs
Predicted action sequence Arm-joint and gripper targets for several future control steps. Isaac Sim controller through the Arena adapter

What happens inside GR00T?

The lab uses the DROID checkpoint, a version of GR00T N1.6-3B fine-tuned on DROID. A vision-language model first turns the camera views and instruction into numeric representations. Conditioned on those representations and the robot state, the action head starts with sampled noise and updates it iteratively to produce a sequence of continuous commands. The training method for learning that transformation is called flow matching. Those internal updates produce one action prediction. The steps in the log below count actions applied in the simulated control loop. NVIDIA N1.6 architecture · DROID checkpoint

The policy proposes a sequence of actions. The runner applies a configured portion, reads a fresh observation, and queries the policy again. The camera views and joint state change after the arm moves, so subsequent actions can respond to that change. Match the arm’s joint order and normalization to the checkpoint and adapter. Arena GR00T walkthrough · GR00T modality configuration

Read “lift” and “placement” in a saved episode

Here is a projection of a few fields from the first 70-second trial. JSONL stores one complete trial per line. Within the trial, progress.events records which checks passed and at which control steps.

JSON
{
  "success": true,
  "episode_length": 521,
  "progress": { "events": [{ "step": 1 }, { "step": 442 }, { "step": 521 }] }
}

Step 1 marks the objects settling, step 442 marks the cube reaching the lift height, and step 521 marks placement on the destination. Convert steps to time using the control rate. The checker uses 15 Hz, so 442 ÷ 15 ≈ 29.47 seconds. In the shorter 24-second runs, a successful lift could still end without a placement. Check for object_on_destination and the final outcome when you replay your own episode. Result checker

First time? Get the lab files

Open a terminal on the Isaac Sim machine for GPU labs, or on your own computer for CPU-only steps. Press Enter after each line. You can copy a multi-line command block as a whole.

Shell
cd "$HOME"
git clone https://github.com/james-joobs/awesome-nvidia-physical-ai.git
cd awesome-nvidia-physical-ai
pwd

This example clones into your home folder. If you already cloned the repo, skip git clone and enter your project folder. In each new terminal, check that pwd ends in awesome-nvidia-physical-ai so commands can find the labs/ files.

Run it

Before you start · Use an RTX Linux machine with a desktop session and uv. Follow the Isaac Sim setup and prepare the native uv runtime in the official Arena GR00T guide. This example uses Arena c7edf3a and GR00T submodule e29d8fc. The commands assume both repos are in your home folder. If you cloned them elsewhere, change the cd paths and ARENA_DIR.

1. Start GR00T in terminal A

Move into Arena, fetch its GR00T submodule, and start the model. You can paste the whole block. The first run downloads the model files. Keep this terminal open once the model is ready.

Shell
cd "$HOME/IsaacLab-Arena"
git submodule update --init submodules/Isaac-GR00T
cd submodules/Isaac-GR00T
uv run python gr00t/eval/run_gr00t_server.py --model-path nvidia/GR00T-N1.6-DROID \
  --embodiment-tag OXE_DROID --device cuda --host 127.0.0.1 --port 5555

2. Run the robot in terminal B on the same machine

Activate Arena’s Python environment, then enter this repository to run its script. The 70 selects the 70-second trial configuration. The Isaac Sim window opens and the arm starts moving. Arena asks the model in terminal A for actions at 127.0.0.1:5555.

Shell
ARENA_DIR="$HOME/IsaacLab-Arena"
source "$ARENA_DIR/.venv/bin/activate"
cd "$HOME/awesome-nvidia-physical-ai"
bash labs/gr00t-placement/run_arena.sh "$ARENA_DIR" 70 ./runs/gr00t

If Arena gets a 404 from its staging asset URL, apply the asset URL patch in the Arena checkout with git apply --unidiff-zero, then retry step 2.

3. Read your run log in terminal B

When the window closes and the terminal is ready for input again, run the commands below. If the filename differs, use the path printed by find in the second line. lift is when the robot picked the cube up. placed is when it left the cube in the bowl. no means the trial did not reach that milestone.

Shell
find ./runs/gr00t -name 'episode_results_*.jsonl'
python labs/gr00t-placement/check_result.py ./runs/gr00t/droid_pnp_gr00t/episode_results_rebuild0.jsonl

Record your own live GUI run

Keep the policy server in terminal A running. In terminal B, use the command below instead of step 2 to save the native Isaac Sim window, an initial USD scene, and object-position samples. This recorder needs an X11 desktop, ffmpeg, and xwininfo; it does not replay a previously successful trajectory.

Shell
export GR00T_GUI_FFMPEG="$(command -v ffmpeg)"
bash labs/gr00t-placement/run_recorded.sh "$ARENA_DIR" 70 ./runs/gr00t-gui
python labs/gr00t-placement/check_result.py \
  ./runs/gr00t-gui/evaluation/droid_pnp_gr00t/episode_results_rebuild0.jsonl

The GUI recorder writes gr00t-gui.mp4, gr00t-arena.usda, viewport-timing.jsonl, and capture.json. The USD captures the initial scene and can reference official external assets; it does not include the policy weights or a standalone replay of the robot’s actions. To inspect actual model inputs and action arrays too, use the optional policy recorder in the GR00T Python environment, with GR00T_MODEL_PATH set to the checkpoint and GR00T_POLICY_OUTPUT set to a new output directory.

Check the result

The three 24-second trials recorded two lifts and zero placements. The two 70-second trials placed the cube at 34.73 and 38.27 simulated seconds. The starting layouts also differed, so these runs cannot isolate the effect of the time limit.

November 19 rerun: lifting did not complete the task

A fresh run on the RTX 5090 used the real GR00T N1.6-DROID policy, Arena c7edf3a, GR00T e29d8fc, and Isaac Sim 6.1.0.0. The checkpoint revision was ae3ebe8d288971ac53aa30c756ea5cba0f52611b. Both 70-second trials reached the lift predicate, but neither passed the destination-placement predicate. The process completed normally; the task result was 0 successful placements out of 2 trials.

Trial Lift time, simulated Placement Total control steps
1 55.87 s Not reached 1,050
2 28.47 s Not reached 1,050

Watch the new attempt: a 20-second viewport crop from trial 2, played at 2× speed. It shows a lift followed by a transfer past the bowl. The complete episode records retain both failed outcomes separately from the earlier successful run. Starting layouts differed between runs, so these few trials do not establish a general success rate or identify the cause of failure.

The policy recorder received two RGB camera inputs shaped [1, 1, 180, 320, 3]: batch, time, height, width, and RGB channels. Each camera supplied one 320×180 frame. It produced joint_position shaped [1, 32, 7] plus gripper_position shaped [1, 32, 1]. The action axes are batch, action horizon, and controlled joints; 32 is the predicted action sequence length, not the episode length.

Shell
python3 labs/gr00t-placement/check_result.py labs/gr00t-placement/recorded-2026-11-19.jsonl

Replay the checks without a GPU

Run both commands from the repository root. Python reads the bundled 24-second recording and 70-second recording.

Shell
python3 labs/gr00t-placement/check_result.py labs/gr00t-placement/recorded-24s.jsonl
python3 labs/gr00t-placement/check_result.py labs/gr00t-placement/recorded-70s.jsonl

Example run and recording

A lift is not yet a placement

November 19 live GR00T run, shown at 2× speed. Both new trials lifted the cube but neither placed it in the bowl.

Source code

droid-70s.yamlView files on GitHub
YAML
# Copyright (c) 2026, The Isaac Lab Arena Project Developers (https://github.com/isaac-sim/IsaacLab-Arena/blob/main/CONTRIBUTORS.md).
# All rights reserved.
#
# SPDX-License-Identifier: Apache-2.0
# Adapted from the Arena DROID example to record two 70-second trials.

runs:
  droid_pnp_gr00t:
    environment:
      type: pick_and_place_maple_table
      enable_cameras: true
      episode_length_s: 70.0
      embodiment: droid_abs_joint_pos
      pick_up_object: rubiks_cube_hot3d_robolab
      destination_location: bowl_ycb_robolab
      hdr: home_office_robolab
    environment_builder:
      language_instruction: Pick up the Rubik's cube and place it in the bowl.
    policy:
      type: isaaclab_arena_gr00t.policy.gr00t_remote_closedloop_policy.Gr00tRemoteClosedloopPolicy
      policy_config_yaml_path: isaaclab_arena_gr00t/policy/config/droid_manip_gr00t_closedloop_config.yaml
      policy_device: cuda:0
      remote_host: 127.0.0.1
      remote_port: 5555
    rollout_limit:
      num_episodes: 2
droid-24s.yamlView files on GitHub
YAML
# Copyright (c) 2026, The Isaac Lab Arena Project Developers (https://github.com/isaac-sim/IsaacLab-Arena/blob/main/CONTRIBUTORS.md).
# All rights reserved.
#
# SPDX-License-Identifier: Apache-2.0
# Adapted from the Arena DROID example with a 24-second episode limit.

runs:
  droid_pnp_gr00t:
    environment:
      type: pick_and_place_maple_table
      enable_cameras: true
      episode_length_s: 24.0
      embodiment: droid_abs_joint_pos
      pick_up_object: rubiks_cube_hot3d_robolab
      destination_location: bowl_ycb_robolab
      hdr: home_office_robolab
    environment_builder:
      language_instruction: Pick up the Rubik's cube and place it in the bowl.
    policy:
      type: isaaclab_arena_gr00t.policy.gr00t_remote_closedloop_policy.Gr00tRemoteClosedloopPolicy
      policy_config_yaml_path: isaaclab_arena_gr00t/policy/config/droid_manip_gr00t_closedloop_config.yaml
      policy_device: cuda:0
      remote_host: 127.0.0.1
      remote_port: 5555
    rollout_limit:
      num_episodes: 3
run_arena.shView files on GitHub
Shell
#!/usr/bin/env bash
set -euo pipefail
if [[ $# -lt 3 || ! "$2" =~ ^(24|70)$ ]]; then
  printf 'Usage: bash %s ARENA_CHECKOUT 24|70 OUTPUT_DIRECTORY [ARENA_OVERRIDES...]\n' "$0" >&2
  exit 2
fi
here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)"
config="$here/droid-${2}s.yaml"
mkdir -p "$3"
output="$(cd "$3" && pwd -P)"
cd "$1"
shift 3
# Start the official loopback GR00T server before running this script.
python -m isaaclab_arena.evaluation.experiment_runner --viz kit --record_camera_video \
  --experiment_config "$config" --experiment_output_directory "$output" "$@"
check_result.pyView files on GitHub
Python
"""Read Arena's episode JSONL and distinguish lifting from stable placement."""

import argparse
import json
from pathlib import Path


def read_episodes(path):
    episodes = []
    for line in path.read_text().splitlines():
        if not line.strip():
            continue
        row = json.loads(line)
        milestones = {event["predicate_name"]: event["step"] for event in row["progress"]["events"]}
        lift = next((step for name, step in milestones.items() if "object_is_above_height" in name), None)
        placed = next((step for name, step in milestones.items() if "object_on_destination" in name), None)
        if bool(row["success"]) != (placed is not None):
            raise ValueError(f"Arena success and placement event disagree in episode {len(episodes) + 1}")
        episodes.append({"steps": row["episode_length"], "lift_step": lift, "placement_step": placed})
    if not episodes:
        raise ValueError("No episodes in the Arena report")
    return episodes


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("episode_report", type=Path)
    parser.add_argument("--rate", type=float, default=15, help="Arena environment control steps per second")
    args = parser.parse_args()
    if args.rate <= 0:
        parser.error("--rate must be positive")
    for index, episode in enumerate(read_episodes(args.episode_report), start=1):
        lifted = episode["lift_step"]
        placed = episode["placement_step"]
        print(f"Trial {index}: {episode['steps']} steps,"
              f" lift {format(lifted / args.rate, '.2f') + ' s' if lifted is not None else 'no'},"
              f" placed {format(placed / args.rate, '.2f') + ' s' if placed is not None else 'no'}")
arena-production-assets.patchView files on GitHub
Text
diff --git a/isaaclab_arena/assets/nucleus.py b/isaaclab_arena/assets/nucleus.py
index a7da76a..7d8ac2c 100644
--- a/isaaclab_arena/assets/nucleus.py
+++ b/isaaclab_arena/assets/nucleus.py
@@ -14 +14 @@ ARENA_STAGING_NUCLEUS_DIR: str = "omniverse://isaac-dev.ov.nvidia.com/Isaac/Isaa
-ARENA_NUCLEUS_DIR: str = ISAACLAB_NUCLEUS_DIR.replace("omniverse-content-production", "omniverse-content-staging")
+ARENA_NUCLEUS_DIR: str = ISAACLAB_NUCLEUS_DIR
record_gui.pyView files on GitHub
Python
"""Record the native Isaac Sim window during a real GR00T/Arena rollout.

Use run_recorded.sh after starting the official GR00T policy server. The USD
snapshot preserves the scene setup; the MP4 records the live policy rollout,
not an animation authored from a known successful trajectory.
"""

from datetime import datetime, timezone
from importlib.metadata import version
import json
import os
from pathlib import Path
import signal
import subprocess
import time

from isaaclab_arena.evaluation import experiment_runner


output = Path(os.environ["GR00T_GUI_OUTPUT"]).resolve()
output.mkdir(parents=True, exist_ok=True)
original_enter = experiment_runner.SimulationAppContext.__enter__


def window_id():
    listing = subprocess.check_output(["xwininfo", "-root", "-tree"], text=True)
    for line in listing.splitlines():
        if '"Isaac Lab ' in line and "mutter-x11-frames" not in line:
            return str(int(line.split()[0], 16))
    raise RuntimeError("The native Isaac Sim GUI window was not found")


class ObservedEnvironment:
    def __init__(self, original):
        self.original = original
        self.camera = None
        self.started = None
        self.frame_number = 0
        self.window = None
        self.wall_log = (output / "viewport-timing.jsonl").open("w", buffering=1)
        self.ffmpeg_log = (output / "viewport-ffmpeg.log").open("w")

    def __getattr__(self, name):
        return getattr(self.original, name)

    def reset(self, *args, **kwargs):
        observation = self.original.reset(*args, **kwargs)
        if self.camera is not None:
            return observation
        from omni.usd import get_context

        stage = get_context().get_stage()
        assert stage and stage.GetRootLayer().Export(str(output / "gr00t-arena.usda"))
        self.window = window_id()
        self.camera = subprocess.Popen([
            os.environ["GR00T_GUI_FFMPEG"], "-hide_banner", "-loglevel", "warning", "-y",
            "-f", "x11grab", "-framerate", "30", "-draw_mouse", "0",
            "-window_id", self.window, "-i", f"{os.environ['DISPLAY']}.0", "-an",
            "-c:v", "libx264", "-preset", "ultrafast", "-crf", "18", "-pix_fmt", "yuv420p",
            "-movflags", "+faststart", str(output / "gr00t-gui.mp4"),
        ], stdin=subprocess.DEVNULL, stdout=self.ffmpeg_log, stderr=subprocess.STDOUT)
        self.started = time.monotonic()
        self.started_utc = datetime.now(timezone.utc).isoformat()
        print("GR00T_GUI_START: scene exported; recording the live policy rollout", flush=True)
        self.sample()
        return observation

    def sample(self):
        env = self.original.unwrapped
        self.wall_log.write(json.dumps({
            "wall_s": time.monotonic() - self.started,
            "global_step": self.frame_number,
            "episodes_completed": int(env.completed_episode_count),
            "positions_m": {name: asset.data.root_pos_w[0].tolist()
                            for name, asset in env.scene.rigid_objects.items()},
        }) + "\n")

    def step(self, actions):
        result = self.original.step(actions)
        self.frame_number += 1
        if self.frame_number % 5 == 0:
            self.sample()
        return result

    def finish(self):
        try:
            if self.camera is not None:
                self.sample()
                if self.camera.poll() is None:
                    self.camera.send_signal(signal.SIGINT)
                self.camera.wait(timeout=20)
                video = output / "gr00t-gui.mp4"
                if not video.exists() or video.stat().st_size < 50_000:
                    raise RuntimeError("The GUI recording is missing or unexpectedly small")
                (output / "capture.json").write_text(json.dumps({
                    "application": "Isaac Sim GUI launched through Isaac Lab-Arena",
                    "isaacsim_version": version("isaacsim"),
                    "recording_mode": "live_policy_native_x11_window",
                    "video": video.name,
                    "scene": "gr00t-arena.usda",
                    "window_id": self.window,
                    "started_utc": self.started_utc,
                    "ended_utc": datetime.now(timezone.utc).isoformat(),
                    "wall_seconds": round(time.monotonic() - self.started, 3),
                    "control_steps": self.frame_number,
                    "ffmpeg_exit_code": self.camera.returncode,
                    "bytes": video.stat().st_size,
                }, indent=2) + "\n")
        finally:
            self.ffmpeg_log.close()
            self.wall_log.close()


def enter(context):
    value = original_enter(context)
    from isaaclab_arena.evaluation import run_execution

    rollout = run_execution.rollout_policy

    def observed_rollout(env, policy, *args, **kwargs):
        instrumented = ObservedEnvironment(env)
        try:
            return rollout(instrumented, policy, *args, **kwargs)
        finally:
            instrumented.finish()

    run_execution.rollout_policy = observed_rollout
    return value


if __name__ == "__main__":
    experiment_runner.SimulationAppContext.__enter__ = enter
    try:
        experiment_runner.main()
    except BaseException:
        # Some Kit shutdown paths mask a Python failure with exit code zero.
        (output / "capture-failed").touch()
        raise
record_policy.pyView files on GitHub
Python
"""Run the official GR00T policy and save one real observation/action per reset.

Set GR00T_MODEL_PATH to a prepared checkpoint and GR00T_POLICY_OUTPUT to the
output directory. No robot actions are synthesized by this recorder.
"""

import json
import os
from pathlib import Path
import time

import numpy as np
from PIL import Image
import torch

from gr00t.data.embodiment_tags import EmbodimentTag
from gr00t.policy.gr00t_policy import Gr00tPolicy
from gr00t.policy.server_client import PolicyServer


output = Path(os.environ["GR00T_POLICY_OUTPUT"]).resolve()
output.mkdir(parents=True, exist_ok=True)
policy = Gr00tPolicy(
    embodiment_tag=EmbodimentTag.OXE_DROID,
    model_path=os.environ["GR00T_MODEL_PATH"],
    device="cuda",
    strict=True,
)
print(f"GR00T N1.6-DROID | torch {torch.__version__} | {torch.cuda.get_device_name(0)}", flush=True)
get_action = policy.get_action
reset = policy.reset
episode = 0
count = 0
saved = set()


def record_reset(*args, **kwargs):
    global episode
    response = reset(*args, **kwargs)
    episode += 1
    return response


def record_action(observation, options=None):
    global count
    started = time.perf_counter()
    action, info = get_action(observation, options)
    elapsed = 1000 * (time.perf_counter() - started)
    count += 1
    arrays = {key: np.asarray(values) for key, values in action.items()}
    if not all(np.isfinite(a).all() and a.size > 0 for a in arrays.values()):
        raise RuntimeError("GR00T returned an invalid action")
    if count <= 3 or count % 25 == 0:
        print(f"ACTION {count} | reset {episode} | {elapsed:.1f} ms | {list(action)}", flush=True)
    if episode not in saved:
        save_dir = output / f"request-{count:04d}"
        save_dir.mkdir(parents=True, exist_ok=True)
        cameras = {}
        for index, (name, values) in enumerate(observation.get("video", {}).items()):
            array = np.asarray(values)
            if array.ndim != 5 or array.shape[-1] != 3:
                continue
            frame = array[0, -1]
            if frame.dtype != np.uint8:
                frame = (frame * 255 if frame.max() <= 1.0 else frame).clip(0, 255).astype(np.uint8)
            filename = f"camera-{index + 1}.png"
            Image.fromarray(frame).save(save_dir / filename)
            cameras[name] = {"file": filename, "shape": list(array.shape)}
        record = {
            "model": "GR00T N1.6-DROID",
            "observation_origin": "Isaac Lab-Arena DROID simulator",
            "request": count,
            "server_reset_count": episode,
            "instruction": observation.get("language", {}),
            "cameras": cameras,
            "observed_state": {k: np.asarray(v).tolist() for k, v in observation.get("state", {}).items()},
            "actions": {k: v.tolist() for k, v in arrays.items()},
            "action_shapes": {k: list(v.shape) for k, v in arrays.items()},
            "server_inference_ms": round(elapsed, 2),
        }
        (save_dir / "observation-action.json").write_text(json.dumps(
            record, indent=2, default=lambda value: np.asarray(value).tolist()) + "\n")
        saved.add(episode)
    return action, info


policy.get_action = record_action
policy.reset = record_reset
PolicyServer(policy, host="127.0.0.1", port=int(os.environ.get("GR00T_POLICY_PORT", "5555"))).run()
run_recorded.shView files on GitHub
Shell
#!/usr/bin/env bash
set -euo pipefail
if [[ $# -lt 3 || ! "$2" =~ ^(24|70)$ ]]; then
  printf 'Usage: bash %s ARENA_CHECKOUT 24|70 OUTPUT_DIRECTORY [ARENA_OVERRIDES...]\n' "$0" >&2
  exit 2
fi
here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)"
config="$here/droid-${2}s.yaml"
mkdir -p "$3"
export GR00T_GUI_OUTPUT="$(cd "$3" && pwd -P)"
export GR00T_GUI_FFMPEG="${GR00T_GUI_FFMPEG:-$(command -v ffmpeg)}"
if [[ ! -x "$GR00T_GUI_FFMPEG" ]]; then
  printf 'Set GR00T_GUI_FFMPEG to an executable ffmpeg binary.\n' >&2
  exit 2
fi
command -v xwininfo >/dev/null
: "${DISPLAY:?Run this from a Linux desktop session with DISPLAY set.}"
cd "$1"
shift 3
# Start the official loopback GR00T server separately before running this script.
"${GR00T_ARENA_PYTHON:-python}" -u "$here/record_gui.py" --viz kit --record_camera_video \
  --experiment_config "$config" --experiment_output_directory "$GR00T_GUI_OUTPUT/evaluation" "$@"
if [[ -e "$GR00T_GUI_OUTPUT/capture-failed" || ! -s "$GR00T_GUI_OUTPUT/capture.json" ]]; then
  printf 'The GUI run did not finish cleanly; keep the logs and inspect the failure.\n' >&2
  exit 1
fi