Advanced Replicator lab · RGB, depth, and masks in 12 frames

Inspect RGB, camera-plane depth, and semantic masks from 12 captured frames.

  • Isaac Sim
  • Replicator
  • OpenUSD
Three outputs from one camera
Recorded runExample run and recording

Optional advanced lab. Use Isaac Sim 6.1 and Isaac Lab with this example’s own USD and checker to generate 960 × 540 data. If you are new to camera capture, start with the Isaac Sim 6.2 workbench capstone.

How it works

How the data moves
  1. InputOne scene and camerareplicator-live.usda · /World/SensorCamera

    USD describes the cube, lights, and the camera used for capture.

  2. ProcessRead one render three waysRGB · camera-plane depth · class ID

    Each annotator reads the same pixel location for a different kind of information.

  3. OutputOne folder per framergb.png · depth_m.npy · semantic_ids.npy

    The PNG stores colors. The arrays store meters or class numbers. The JSON file gives those numbers names.

Read color, depth, and labels from the same view

A Replicator render product pairs a USD camera with its capture resolution. The script attaches three annotators to read color, camera-plane depth, and semantic labels. The lab varies cube position, lighting, camera settings, and occlusion, then reads all three outputs from the same render. Semantic IDs come from class labels authored on the USD prims. Isaac Sim capture examples · Lab capture code

Look up the same pixel in the frame directory

File Shape and type in this lab Meaning
rgb.png [540, 960, 3] when decoded as uint8 Red, green, and blue in the range 0–255.
depth_m.npy [540, 960], float32 Depth in meters along the camera’s forward axis. A pixel’s straight-line distance to the camera can differ.
semantic_ids.npy [540, 960], uint32 A class ID for each pixel.
semantic_labels.json ID → label Look up the IDs named target to select the visible cube pixels.

Image coordinates (u, v) increase right and down from the top-left corner. NumPy indexes the row first, so read the same point with rgb[v, u], depth[v, u], and ids[v, u]. Multiple objects can share one semantic class. To distinguish them as individual objects, use instance segmentation.

How does a mask turn into a box?

Select all pixels labeled target and take their left, top, right, and bottom extremes. In the recorded frame 0, the box is [264, 159, 401, 292] with 15121 target pixels. The extrema are inclusive, so the rectangle is 138 × 134 pixels. It also contains background pixels, so use the mask count for visible area.

The occluded example at frame 3 has 4102 target pixels. The cube also moved between these frames, so that difference alone does not isolate occlusion. To test occlusion independently, fix the camera and cube and change only the blocker.

Recorded frame 3 showing RGB, camera-plane depth, and semantic labels. The yellow box encloses the visible sliver of the cube to the left of the blocker.

The box encloses the visible cube pixels in frame 3. The rest of the cube is behind the blocker. Preview script

First time? Get the lab files

Open a terminal on the Isaac Sim machine for GPU labs, or on your own computer for CPU-only steps. Press Enter after each line. You can copy a multi-line command block as a whole.

Shell
cd "$HOME"
git clone https://github.com/james-joobs/awesome-nvidia-physical-ai.git
cd awesome-nvidia-physical-ai
pwd

This example clones into your home folder. If you already cloned the repo, skip git clone and enter your project folder. In each new terminal, check that pwd ends in awesome-nvidia-physical-ai so commands can find the labs/ files.

Run it

Before you start · Activate the Isaac Lab Python environment paired with Isaac Sim 6.1 on your RTX machine. See the Isaac Sim setup guide for the available installation types. This script uses Isaac Lab’s app launcher, which a generic Python environment with NumPy does not provide.

1. Open the stage and capture the frames

From this repository in a terminal on the simulator machine, run the block below. The first path is the USD scene to open. The second is where the script saves the capture. --viz kit opens the window. Use a fresh output directory on a second run.

Shell
python labs/replicator-outputs/capture.py \
  labs/replicator-outputs/replicator-live.usda ./runs/replicator --viz kit

2. Read the first frame’s files

After the run, use the same terminal for the commands below. Open rgb.png in an image viewer and semantic_labels.json in a text editor. The two .npy files contain numeric arrays, which the checker can read. It checks that the three channels share the same dimensions and that the cube’s depth pixels are valid.

Shell
ls ./runs/replicator/frames/00
python labs/replicator-outputs/check_capture.py ./runs/replicator

With FFmpeg installed, run python labs/replicator-outputs/preview_dataset.py ./runs/replicator for a side-by-side preview. NVIDIA Replicator tutorial

Check the result

Look for 12 distinct 960 × 540 triplets. The checker reads the arrays, compares the RGB hashes and semantic pixel counts to the manifest, and checks that cube pixels have finite, positive depth. Occluded frames 03, 07 and 11 contain fewer visible cube pixels. Recorded manifest

Example run and recording

Source code

capture.pyView files on GitHub
Python
"""Generate and record actual Replicator outputs from the Isaac Sim GUI."""

import argparse
import hashlib
import json
import os
from pathlib import Path
import signal
import subprocess
import time
import traceback

from isaaclab.app import AppLauncher

parser = argparse.ArgumentParser()
parser.add_argument("scene", type=Path)
parser.add_argument("output", type=Path)
parser.add_argument("--record-window", type=Path, metavar="FFMPEG",
                    help="Optional Linux X11 recording using this FFmpeg executable.")
AppLauncher.add_app_launcher_args(parser)
args = parser.parse_args()
args.enable_cameras = True
args.output.mkdir(parents=True, exist_ok=True)
app = AppLauncher(args).app

import carb
import numpy as np
import omni.replicator.core as rep
import omni.timeline
import omni.usd
from omni.kit.viewport.utility import get_active_viewport
from PIL import Image
from pxr import Gf, UsdGeom, UsdLux, UsdSemantics

WIDTH, HEIGHT = 960, 540


def update_for(seconds):
    deadline = time.monotonic() + seconds
    while time.monotonic() < deadline:
        app.update()


def screen_recorder(log):
    for line in subprocess.check_output(["xwininfo", "-root", "-tree"], text=True).splitlines():
        if '"Isaac Lab ' in line and "mutter-x11-frames" not in line:
            identifier = str(int(line.split()[0], 16))
            break
    else:
        raise RuntimeError("Isaac Sim window not found")
    return subprocess.Popen([
        str(args.record_window), "-hide_banner", "-loglevel", "warning", "-y",
        "-f", "x11grab", "-draw_mouse", "0", "-framerate", "30",
        "-window_id", identifier, "-i", f"{os.environ['DISPLAY']}.0",
        "-an", "-c:v", "libx264", "-preset", "ultrafast", "-crf", "18",
        "-pix_fmt", "yuv420p", "-movflags", "+faststart", str(args.output / "replicator-gui.mp4"),
    ], stdin=subprocess.DEVNULL, stdout=log, stderr=subprocess.STDOUT)


def capture_dataset(stage, directory):
    annotator_names = ("rgb", "distance_to_image_plane", "semantic_segmentation")
    sensor = "/World/SensorCamera"
    viewport = get_active_viewport()
    viewport.camera_path = "/World/FilmCamera"
    stage.SetEndTimeCode(900)
    for path, category in (("/World/Target", "target"), ("/World/Distractor", "distractor"),
                           ("/World/Occluder", "occluder"), ("/World/Workbench/Top", "workbench")):
        UsdSemantics.LabelsAPI.Apply(stage.GetPrimAtPath(path), "class").CreateLabelsAttr([category])
    assert stage.GetRootLayer().Export(str(args.output / "replicator-workbench.usda"))
    target = UsdGeom.Xformable(stage.GetPrimAtPath("/World/Target")).GetOrderedXformOps()[0]
    shield = UsdGeom.Xformable(stage.GetPrimAtPath("/World/Occluder")).GetOrderedXformOps()[0]
    lens = UsdGeom.Xformable(stage.GetPrimAtPath(sensor)).GetOrderedXformOps()[0]
    key = UsdLux.RectLight.Get(stage, "/World/Lighting/Key").GetIntensityAttr()
    product = rep.create.render_product(sensor, (WIDTH, HEIGHT))
    options = {"semantic_segmentation": {"colorize": False}}
    annotators = {name: rep.AnnotatorRegistry.get_annotator(name, init_params=options[name])
                  if name in options else rep.AnnotatorRegistry.get_annotator(name) for name in annotator_names}
    for annotator in annotators.values():
        annotator.attach([product])
    timeline = omni.timeline.get_timeline_interface()
    timeline.play()
    timeline.commit()
    for _ in range(12):
        update_for(.50)
        live_rgb = np.asarray(annotators["rgb"].get_data())
        if live_rgb.shape[:2] == (HEIGHT, WIDTH) and live_rgb[:, :, :3].mean() > 5:
            break
    else:
        raise RuntimeError("Replicator RGB render product did not initialize")

    shots = []
    hashes = set()
    with (args.output / "recorder.log").open("w") as log:
        recorder = screen_recorder(log) if args.record_window else None
        begin = time.monotonic()
        try:
            update_for(1.2)
            for i in range(12):
                x = (-.24, -.09, .08, .25)[i % 4]
                energy = (950, 1550, 2300)[i // 4]
                occluded = i % 4 == 3
                target.Set(Gf.Vec3d(x, -.025, .87))
                shield.Set(Gf.Vec3d(x+.10 if occluded else .43, -.113 if occluded else .12, .935))
                key.Set(energy)
                eye = Gf.Vec3d(1.10 + (i//4)*.13, -1.67, 1.60 + (i//4)*.07)
                lens.Set(Gf.Matrix4d(1).SetLookAt(eye, Gf.Vec3d(0,0,.84), Gf.Vec3d(0,0,1)).GetInverse())
                if i == 6:
                    viewport.camera_path = sensor
                update_for(.72)
                rgb = np.asarray(annotators["rgb"].get_data(), dtype=np.uint8)
                depth = np.asarray(annotators["distance_to_image_plane"].get_data(), dtype=np.float32).squeeze()
                segmentation = annotators["semantic_segmentation"].get_data()
                ids = np.asarray(segmentation["data"], dtype=np.uint32).squeeze()
                labels = segmentation["info"]["idToLabels"]
                if isinstance(labels, str):
                    labels = json.loads(labels)
                target_ids = [int(v) for v,label in labels.items() if (
                    label.get("class") if isinstance(label, dict) else label
                ) == "target"]
                assert rgb.shape[:2] == depth.shape == ids.shape == (HEIGHT, WIDTH)
                digest = hashlib.sha256(rgb[:, :, :3].tobytes()).hexdigest()
                assert digest not in hashes, (i, "duplicate RGB buffer")
                hashes.add(digest)
                mask = np.isin(ids, target_ids)
                if not occluded:
                    assert mask.sum() >= 100, (i, labels)
                assert not np.isnan(depth[mask]).any() and np.isfinite(depth[mask]).all()
                frame = directory / f"{i:02d}"
                frame.mkdir()
                Image.fromarray(rgb[:, :, :3]).save(frame / "rgb.png")
                np.save(frame / "depth_m.npy", depth)
                np.save(frame / "semantic_ids.npy", ids)
                (frame / "semantic_labels.json").write_text(json.dumps(labels, indent=2) + "\n")
                yy, xx = np.nonzero(mask)
                bbox = None if len(xx) == 0 else [int(xx.min()), int(yy.min()), int(xx.max()), int(yy.max())]
                info = {"frame": i, "target_x_m": x, "candidate_occluded": occluded, "key_light_intensity": energy,
                        "target_pixels": int(mask.sum()), "target_bbox_xyxy": bbox,
                        "target_median_depth_m": float(np.median(depth[mask])) if mask.any() else None,
                        "rgb_sha256": digest, "gui_wall_s": time.monotonic() - begin}
                shots.append(info)
                print(f"REPLICATOR_SAMPLE {i:02d} | RGB/DEPTH/IDS | {info['target_pixels']} target pixels", flush=True)
                update_for(.37)
            update_for(2)
        finally:
            if recorder:
                recorder.send_signal(signal.SIGINT)
                recorder.wait(timeout=14)
            for annotator in annotators.values():
                annotator.detach([product])
            timeline.stop()
    assert len(shots) == 12
    (args.output / "result.json").write_text(json.dumps({
        "resolution_px": [WIDTH, HEIGHT], "channels": list(annotators), "frames": shots,
    }, indent=2) + "\n")


try:
    settings = carb.settings.get_settings()
    settings.set_bool("/omni/replicator/captureOnPlay", True)
    settings.set_bool("/rtx/hydra/readTransformsFromFabricInRenderDelegate", False)
    settings.set_bool("/persistent/app/viewport/Viewport/Viewport0/scene/cameras/visible", False)
    context = omni.usd.get_context()
    assert context.open_stage(str(args.scene.resolve()))
    update_for(1.5)
    stage = context.get_stage()
    assert stage and stage.GetPrimAtPath("/World/SensorCamera")
    frames = args.output / "frames"
    frames.mkdir()
    capture_dataset(stage, frames)
except Exception:
    (args.output / "capture-failed").touch()
    traceback.print_exc()
    raise
finally:
    app.close()
preview_dataset.pyView files on GitHub
Python
"""Preview the real RGB, depth, and semantic ID arrays captured by Replicator."""

import argparse
import json
from pathlib import Path
import subprocess
from tempfile import TemporaryDirectory
from zipfile import ZIP_DEFLATED, ZipFile

import numpy as np
from PIL import Image, ImageDraw, ImageFont

PALETTE = {"target": (125, 222, 133), "distractor": (74, 133, 204),
           "occluder": (234, 133, 97), "workbench": (179, 160, 112)}


def font(size):
    location = Path("/System/Library/Fonts/Avenir Next.ttc")
    return ImageFont.truetype(str(location) if location.exists() else "DejaVuSans.ttf", size)


def panels(frame, row):
    rgb = Image.open(frame / "rgb.png").convert("RGB")
    depth = np.load(frame / "depth_m.npy", allow_pickle=False)
    sid = np.load(frame / "semantic_ids.npy", allow_pickle=False)
    labels = json.loads((frame / "semantic_labels.json").read_text())
    assert depth.shape == sid.shape == (540, 960)
    shade = np.zeros((*depth.shape, 3), np.uint8)
    finite = np.isfinite(depth)
    ranges = [1.2, 1.9, 2.6, 3.4]
    gradient = np.array([(247, 224, 132), (98, 188, 161), (39, 102, 141), (13, 27, 48)])
    shade[finite] = np.stack([np.interp(depth[finite], ranges, gradient[:, k]) for k in range(3)], axis=1).astype(np.uint8)
    semantic = np.full((*sid.shape, 3), (19, 28, 35), np.uint8)
    for serial, label in labels.items():
        category = label.get("class") if isinstance(label, dict) else label
        semantic[sid == int(serial)] = PALETTE.get(category, (41, 52, 58))
    if row["target_bbox_xyxy"] is not None:
        drawing = ImageDraw.Draw(rgb)
        drawing.rectangle(tuple(row["target_bbox_xyxy"]), outline=(138, 241, 151), width=3)
    return rgb, Image.fromarray(shade), Image.fromarray(semantic)


def preview(frame, row, path):
    directions = ["RGB + box from semantic pixels", "Camera-plane depth [1.2–3.4 m]", "Semantic mask"]
    values = panels(frame, row)
    canvas = Image.new("RGB", (1440, 372), (18, 23, 27))
    draw = ImageDraw.Draw(canvas)
    for i, (image, text) in enumerate(zip(values, directions, strict=True)):
        x = 12 + i*480
        draw.text((x, 11), text, font=font(19), fill=(229, 239, 236))
        canvas.paste(image.resize((456, 256), Image.Resampling.LANCZOS), (x, 43))
    explanation = f"FRAME {row['frame']:02d}     •     CUBE PIXELS {row['target_pixels']:,}"
    draw.text((12, 321), explanation, font=font(19), fill=(229, 239, 236))
    draw.text((974, 322), "Target   •   Distractor   •   Occluder", font=font(17), fill=(153, 205, 167))
    canvas.save(path)


def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("dataset", type=Path)
    args = parser.parse_args()
    data = json.loads((args.dataset / "result.json").read_text())
    assert len(data["frames"]) == 12
    with TemporaryDirectory(prefix="replicator-preview-") as temporary:
        root = Path(temporary)
        for row in data["frames"]:
            i = row["frame"]
            preview(args.dataset / "frames" / f"{i:02d}", row, root / f"{i:02d}.png")
        (args.dataset / "replicator-sample.png").write_bytes((root / "03.png").read_bytes())
        subprocess.run([
            "ffmpeg", "-hide_banner", "-loglevel", "error", "-y", "-framerate", "5/3",
            "-i", str(root / "%02d.png"), "-vf", "fps=30", "-c:v", "libx264",
            "-pix_fmt", "yuv420p", "-movflags", "+faststart", str(args.dataset / "replicator-samples.mp4"),
        ], check=True)
    with ZipFile(args.dataset / "replicator-dataset.zip", "w", compression=ZIP_DEFLATED) as archive:
        archive.write(args.dataset / "result.json", "result.json")
        for file in sorted((args.dataset / "frames").rglob("*")):
            if file.is_file():
                archive.write(file, str(file.relative_to(args.dataset)))


if __name__ == "__main__":
    main()
check_capture.pyView files on GitHub
Python
"""Check the recorded RGB, depth and semantic masks against Replicator's manifest."""

import argparse
import hashlib
import json
from pathlib import Path

import numpy as np
from PIL import Image


def check(folder):
    report = json.loads((folder / "result.json").read_text())
    width, height = report["resolution_px"]
    frames = report["frames"]
    assert len(frames) == 12
    seen = set()
    for i, row in enumerate(frames):
        assert row["frame"] == i
        sample = folder / "frames" / f"{i:02d}"
        with Image.open(sample / "rgb.png") as png:
            rgb = np.asarray(png.convert("RGB"))
        depth = np.load(sample / "depth_m.npy", allow_pickle=False)
        ids = np.load(sample / "semantic_ids.npy", allow_pickle=False)
        labels = json.loads((sample / "semantic_labels.json").read_text())
        assert rgb.shape == (height, width, 3) and ids.shape == depth.shape == (height, width)
        digest = hashlib.sha256(rgb.tobytes()).hexdigest()
        assert digest == row["rgb_sha256"] and digest not in seen
        seen.add(digest)
        target_ids = [int(key) for key, value in labels.items()
                      if (value.get("class") if isinstance(value, dict) else value) == "target"]
        mask = np.isin(ids, target_ids)
        assert np.count_nonzero(mask) == row["target_pixels"] > 0
        assert np.isfinite(depth[mask]).all() and (depth[mask] > 0).all()
    print(f"Verified {len(frames)} RGB/depth/semantic triplets at {width}×{height}")


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("capture", type=Path)
    check(parser.parse_args().capture)