Optional advanced lab. Use Isaac Sim 6.1 and Isaac Lab with this example’s own USD and checker to generate 960 × 540 data. If you are new to camera capture, start with the Isaac Sim 6.2 workbench capstone.
How it works
- InputOne scene and camera
replicator-live.usda · /World/SensorCameraUSD describes the cube, lights, and the camera used for capture.
- ProcessRead one render three ways
RGB · camera-plane depth · class IDEach annotator reads the same pixel location for a different kind of information.
- OutputOne folder per frame
rgb.png · depth_m.npy · semantic_ids.npyThe PNG stores colors. The arrays store meters or class numbers. The JSON file gives those numbers names.
Read color, depth, and labels from the same view
A Replicator render product pairs a USD camera with its capture resolution. The script attaches three annotators to read color, camera-plane depth, and semantic labels. The lab varies cube position, lighting, camera settings, and occlusion, then reads all three outputs from the same render. Semantic IDs come from class labels authored on the USD prims. Isaac Sim capture examples · Lab capture code
Look up the same pixel in the frame directory
| File | Shape and type in this lab | Meaning |
|---|---|---|
rgb.png |
[540, 960, 3] when decoded as uint8 |
Red, green, and blue in the range 0–255. |
depth_m.npy |
[540, 960], float32 |
Depth in meters along the camera’s forward axis. A pixel’s straight-line distance to the camera can differ. |
semantic_ids.npy |
[540, 960], uint32 |
A class ID for each pixel. |
semantic_labels.json |
ID → label | Look up the IDs named target to select the visible cube pixels. |
Image coordinates (u, v) increase right and down from the top-left corner. NumPy indexes the row first, so read the same point with rgb[v, u], depth[v, u], and ids[v, u]. Multiple objects can share one semantic class. To distinguish them as individual objects, use instance segmentation.
How does a mask turn into a box?
Select all pixels labeled target and take their left, top, right, and bottom extremes. In the recorded frame 0, the box is [264, 159, 401, 292] with 15121 target pixels. The extrema are inclusive, so the rectangle is 138 × 134 pixels. It also contains background pixels, so use the mask count for visible area.
The occluded example at frame 3 has 4102 target pixels. The cube also moved between these frames, so that difference alone does not isolate occlusion. To test occlusion independently, fix the camera and cube and change only the blocker.

The box encloses the visible cube pixels in frame 3. The rest of the cube is behind the blocker. Preview script
First time? Get the lab files
Open a terminal on the Isaac Sim machine for GPU labs, or on your own computer for CPU-only steps. Press Enter after each line. You can copy a multi-line command block as a whole.
cd "$HOME"
git clone https://github.com/james-joobs/awesome-nvidia-physical-ai.git
cd awesome-nvidia-physical-ai
pwd
This example clones into your home folder. If you already cloned the repo, skip git clone and enter your project folder. In each new terminal, check that pwd ends in awesome-nvidia-physical-ai so commands can find the labs/ files.
Run it
Before you start · Activate the Isaac Lab Python environment paired with Isaac Sim 6.1 on your RTX machine. See the Isaac Sim setup guide for the available installation types. This script uses Isaac Lab’s app launcher, which a generic Python environment with NumPy does not provide.
1. Open the stage and capture the frames
From this repository in a terminal on the simulator machine, run the block below. The first path is the USD scene to open. The second is where the script saves the capture. --viz kit opens the window. Use a fresh output directory on a second run.
python labs/replicator-outputs/capture.py \
labs/replicator-outputs/replicator-live.usda ./runs/replicator --viz kit
2. Read the first frame’s files
After the run, use the same terminal for the commands below. Open rgb.png in an image viewer and semantic_labels.json in a text editor. The two .npy files contain numeric arrays, which the checker can read. It checks that the three channels share the same dimensions and that the cube’s depth pixels are valid.
ls ./runs/replicator/frames/00
python labs/replicator-outputs/check_capture.py ./runs/replicator
With FFmpeg installed, run python labs/replicator-outputs/preview_dataset.py ./runs/replicator for a side-by-side preview. NVIDIA Replicator tutorial
Check the result
Look for 12 distinct 960 × 540 triplets. The checker reads the arrays, compares the RGB hashes and semantic pixel counts to the manifest, and checks that cube pixels have finite, positive depth. Occluded frames 03, 07 and 11 contain fewer visible cube pixels. Recorded manifest
Example run and recording
Source code
capture.py
View files on GitHub"""Generate and record actual Replicator outputs from the Isaac Sim GUI."""
import argparse
import hashlib
import json
import os
from pathlib import Path
import signal
import subprocess
import time
import traceback
from isaaclab.app import AppLauncher
parser = argparse.ArgumentParser()
parser.add_argument("scene", type=Path)
parser.add_argument("output", type=Path)
parser.add_argument("--record-window", type=Path, metavar="FFMPEG",
help="Optional Linux X11 recording using this FFmpeg executable.")
AppLauncher.add_app_launcher_args(parser)
args = parser.parse_args()
args.enable_cameras = True
args.output.mkdir(parents=True, exist_ok=True)
app = AppLauncher(args).app
import carb
import numpy as np
import omni.replicator.core as rep
import omni.timeline
import omni.usd
from omni.kit.viewport.utility import get_active_viewport
from PIL import Image
from pxr import Gf, UsdGeom, UsdLux, UsdSemantics
WIDTH, HEIGHT = 960, 540
def update_for(seconds):
deadline = time.monotonic() + seconds
while time.monotonic() < deadline:
app.update()
def screen_recorder(log):
for line in subprocess.check_output(["xwininfo", "-root", "-tree"], text=True).splitlines():
if '"Isaac Lab ' in line and "mutter-x11-frames" not in line:
identifier = str(int(line.split()[0], 16))
break
else:
raise RuntimeError("Isaac Sim window not found")
return subprocess.Popen([
str(args.record_window), "-hide_banner", "-loglevel", "warning", "-y",
"-f", "x11grab", "-draw_mouse", "0", "-framerate", "30",
"-window_id", identifier, "-i", f"{os.environ['DISPLAY']}.0",
"-an", "-c:v", "libx264", "-preset", "ultrafast", "-crf", "18",
"-pix_fmt", "yuv420p", "-movflags", "+faststart", str(args.output / "replicator-gui.mp4"),
], stdin=subprocess.DEVNULL, stdout=log, stderr=subprocess.STDOUT)
def capture_dataset(stage, directory):
annotator_names = ("rgb", "distance_to_image_plane", "semantic_segmentation")
sensor = "/World/SensorCamera"
viewport = get_active_viewport()
viewport.camera_path = "/World/FilmCamera"
stage.SetEndTimeCode(900)
for path, category in (("/World/Target", "target"), ("/World/Distractor", "distractor"),
("/World/Occluder", "occluder"), ("/World/Workbench/Top", "workbench")):
UsdSemantics.LabelsAPI.Apply(stage.GetPrimAtPath(path), "class").CreateLabelsAttr([category])
assert stage.GetRootLayer().Export(str(args.output / "replicator-workbench.usda"))
target = UsdGeom.Xformable(stage.GetPrimAtPath("/World/Target")).GetOrderedXformOps()[0]
shield = UsdGeom.Xformable(stage.GetPrimAtPath("/World/Occluder")).GetOrderedXformOps()[0]
lens = UsdGeom.Xformable(stage.GetPrimAtPath(sensor)).GetOrderedXformOps()[0]
key = UsdLux.RectLight.Get(stage, "/World/Lighting/Key").GetIntensityAttr()
product = rep.create.render_product(sensor, (WIDTH, HEIGHT))
options = {"semantic_segmentation": {"colorize": False}}
annotators = {name: rep.AnnotatorRegistry.get_annotator(name, init_params=options[name])
if name in options else rep.AnnotatorRegistry.get_annotator(name) for name in annotator_names}
for annotator in annotators.values():
annotator.attach([product])
timeline = omni.timeline.get_timeline_interface()
timeline.play()
timeline.commit()
for _ in range(12):
update_for(.50)
live_rgb = np.asarray(annotators["rgb"].get_data())
if live_rgb.shape[:2] == (HEIGHT, WIDTH) and live_rgb[:, :, :3].mean() > 5:
break
else:
raise RuntimeError("Replicator RGB render product did not initialize")
shots = []
hashes = set()
with (args.output / "recorder.log").open("w") as log:
recorder = screen_recorder(log) if args.record_window else None
begin = time.monotonic()
try:
update_for(1.2)
for i in range(12):
x = (-.24, -.09, .08, .25)[i % 4]
energy = (950, 1550, 2300)[i // 4]
occluded = i % 4 == 3
target.Set(Gf.Vec3d(x, -.025, .87))
shield.Set(Gf.Vec3d(x+.10 if occluded else .43, -.113 if occluded else .12, .935))
key.Set(energy)
eye = Gf.Vec3d(1.10 + (i//4)*.13, -1.67, 1.60 + (i//4)*.07)
lens.Set(Gf.Matrix4d(1).SetLookAt(eye, Gf.Vec3d(0,0,.84), Gf.Vec3d(0,0,1)).GetInverse())
if i == 6:
viewport.camera_path = sensor
update_for(.72)
rgb = np.asarray(annotators["rgb"].get_data(), dtype=np.uint8)
depth = np.asarray(annotators["distance_to_image_plane"].get_data(), dtype=np.float32).squeeze()
segmentation = annotators["semantic_segmentation"].get_data()
ids = np.asarray(segmentation["data"], dtype=np.uint32).squeeze()
labels = segmentation["info"]["idToLabels"]
if isinstance(labels, str):
labels = json.loads(labels)
target_ids = [int(v) for v,label in labels.items() if (
label.get("class") if isinstance(label, dict) else label
) == "target"]
assert rgb.shape[:2] == depth.shape == ids.shape == (HEIGHT, WIDTH)
digest = hashlib.sha256(rgb[:, :, :3].tobytes()).hexdigest()
assert digest not in hashes, (i, "duplicate RGB buffer")
hashes.add(digest)
mask = np.isin(ids, target_ids)
if not occluded:
assert mask.sum() >= 100, (i, labels)
assert not np.isnan(depth[mask]).any() and np.isfinite(depth[mask]).all()
frame = directory / f"{i:02d}"
frame.mkdir()
Image.fromarray(rgb[:, :, :3]).save(frame / "rgb.png")
np.save(frame / "depth_m.npy", depth)
np.save(frame / "semantic_ids.npy", ids)
(frame / "semantic_labels.json").write_text(json.dumps(labels, indent=2) + "\n")
yy, xx = np.nonzero(mask)
bbox = None if len(xx) == 0 else [int(xx.min()), int(yy.min()), int(xx.max()), int(yy.max())]
info = {"frame": i, "target_x_m": x, "candidate_occluded": occluded, "key_light_intensity": energy,
"target_pixels": int(mask.sum()), "target_bbox_xyxy": bbox,
"target_median_depth_m": float(np.median(depth[mask])) if mask.any() else None,
"rgb_sha256": digest, "gui_wall_s": time.monotonic() - begin}
shots.append(info)
print(f"REPLICATOR_SAMPLE {i:02d} | RGB/DEPTH/IDS | {info['target_pixels']} target pixels", flush=True)
update_for(.37)
update_for(2)
finally:
if recorder:
recorder.send_signal(signal.SIGINT)
recorder.wait(timeout=14)
for annotator in annotators.values():
annotator.detach([product])
timeline.stop()
assert len(shots) == 12
(args.output / "result.json").write_text(json.dumps({
"resolution_px": [WIDTH, HEIGHT], "channels": list(annotators), "frames": shots,
}, indent=2) + "\n")
try:
settings = carb.settings.get_settings()
settings.set_bool("/omni/replicator/captureOnPlay", True)
settings.set_bool("/rtx/hydra/readTransformsFromFabricInRenderDelegate", False)
settings.set_bool("/persistent/app/viewport/Viewport/Viewport0/scene/cameras/visible", False)
context = omni.usd.get_context()
assert context.open_stage(str(args.scene.resolve()))
update_for(1.5)
stage = context.get_stage()
assert stage and stage.GetPrimAtPath("/World/SensorCamera")
frames = args.output / "frames"
frames.mkdir()
capture_dataset(stage, frames)
except Exception:
(args.output / "capture-failed").touch()
traceback.print_exc()
raise
finally:
app.close()
preview_dataset.py
View files on GitHub"""Preview the real RGB, depth, and semantic ID arrays captured by Replicator."""
import argparse
import json
from pathlib import Path
import subprocess
from tempfile import TemporaryDirectory
from zipfile import ZIP_DEFLATED, ZipFile
import numpy as np
from PIL import Image, ImageDraw, ImageFont
PALETTE = {"target": (125, 222, 133), "distractor": (74, 133, 204),
"occluder": (234, 133, 97), "workbench": (179, 160, 112)}
def font(size):
location = Path("/System/Library/Fonts/Avenir Next.ttc")
return ImageFont.truetype(str(location) if location.exists() else "DejaVuSans.ttf", size)
def panels(frame, row):
rgb = Image.open(frame / "rgb.png").convert("RGB")
depth = np.load(frame / "depth_m.npy", allow_pickle=False)
sid = np.load(frame / "semantic_ids.npy", allow_pickle=False)
labels = json.loads((frame / "semantic_labels.json").read_text())
assert depth.shape == sid.shape == (540, 960)
shade = np.zeros((*depth.shape, 3), np.uint8)
finite = np.isfinite(depth)
ranges = [1.2, 1.9, 2.6, 3.4]
gradient = np.array([(247, 224, 132), (98, 188, 161), (39, 102, 141), (13, 27, 48)])
shade[finite] = np.stack([np.interp(depth[finite], ranges, gradient[:, k]) for k in range(3)], axis=1).astype(np.uint8)
semantic = np.full((*sid.shape, 3), (19, 28, 35), np.uint8)
for serial, label in labels.items():
category = label.get("class") if isinstance(label, dict) else label
semantic[sid == int(serial)] = PALETTE.get(category, (41, 52, 58))
if row["target_bbox_xyxy"] is not None:
drawing = ImageDraw.Draw(rgb)
drawing.rectangle(tuple(row["target_bbox_xyxy"]), outline=(138, 241, 151), width=3)
return rgb, Image.fromarray(shade), Image.fromarray(semantic)
def preview(frame, row, path):
directions = ["RGB + box from semantic pixels", "Camera-plane depth [1.2–3.4 m]", "Semantic mask"]
values = panels(frame, row)
canvas = Image.new("RGB", (1440, 372), (18, 23, 27))
draw = ImageDraw.Draw(canvas)
for i, (image, text) in enumerate(zip(values, directions, strict=True)):
x = 12 + i*480
draw.text((x, 11), text, font=font(19), fill=(229, 239, 236))
canvas.paste(image.resize((456, 256), Image.Resampling.LANCZOS), (x, 43))
explanation = f"FRAME {row['frame']:02d} • CUBE PIXELS {row['target_pixels']:,}"
draw.text((12, 321), explanation, font=font(19), fill=(229, 239, 236))
draw.text((974, 322), "Target • Distractor • Occluder", font=font(17), fill=(153, 205, 167))
canvas.save(path)
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("dataset", type=Path)
args = parser.parse_args()
data = json.loads((args.dataset / "result.json").read_text())
assert len(data["frames"]) == 12
with TemporaryDirectory(prefix="replicator-preview-") as temporary:
root = Path(temporary)
for row in data["frames"]:
i = row["frame"]
preview(args.dataset / "frames" / f"{i:02d}", row, root / f"{i:02d}.png")
(args.dataset / "replicator-sample.png").write_bytes((root / "03.png").read_bytes())
subprocess.run([
"ffmpeg", "-hide_banner", "-loglevel", "error", "-y", "-framerate", "5/3",
"-i", str(root / "%02d.png"), "-vf", "fps=30", "-c:v", "libx264",
"-pix_fmt", "yuv420p", "-movflags", "+faststart", str(args.dataset / "replicator-samples.mp4"),
], check=True)
with ZipFile(args.dataset / "replicator-dataset.zip", "w", compression=ZIP_DEFLATED) as archive:
archive.write(args.dataset / "result.json", "result.json")
for file in sorted((args.dataset / "frames").rglob("*")):
if file.is_file():
archive.write(file, str(file.relative_to(args.dataset)))
if __name__ == "__main__":
main()
check_capture.py
View files on GitHub"""Check the recorded RGB, depth and semantic masks against Replicator's manifest."""
import argparse
import hashlib
import json
from pathlib import Path
import numpy as np
from PIL import Image
def check(folder):
report = json.loads((folder / "result.json").read_text())
width, height = report["resolution_px"]
frames = report["frames"]
assert len(frames) == 12
seen = set()
for i, row in enumerate(frames):
assert row["frame"] == i
sample = folder / "frames" / f"{i:02d}"
with Image.open(sample / "rgb.png") as png:
rgb = np.asarray(png.convert("RGB"))
depth = np.load(sample / "depth_m.npy", allow_pickle=False)
ids = np.load(sample / "semantic_ids.npy", allow_pickle=False)
labels = json.loads((sample / "semantic_labels.json").read_text())
assert rgb.shape == (height, width, 3) and ids.shape == depth.shape == (height, width)
digest = hashlib.sha256(rgb.tobytes()).hexdigest()
assert digest == row["rgb_sha256"] and digest not in seen
seen.add(digest)
target_ids = [int(key) for key, value in labels.items()
if (value.get("class") if isinstance(value, dict) else value) == "target"]
mask = np.isin(ids, target_ids)
assert np.count_nonzero(mask) == row["target_pixels"] > 0
assert np.isfinite(depth[mask]).all() and (depth[mask] > 0).all()
print(f"Verified {len(frames)} RGB/depth/semantic triplets at {width}×{height}")
if __name__ == "__main__":
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("capture", type=Path)
check(parser.parse_args().capture)