From a CAD workbench to sensor data

Open a CAD workbench, vary the camera scene in Replicator, and inspect RGB, depth, labels, and a reconstructed point cloud.

  • CadQuery
  • OpenUSD
  • Isaac Sim
  • PhysX
  • Replicator
From a scene to sensor data
Recorded runExample run and recording

How it works

How the data moves
  1. InputCamera dataUSD camera + depth.npy + semantic mask

    The mask tells us which pixels belong to the cube. The USD camera gives us the lens and position.

  2. ProcessPlace the pixels in 3Dpixel (u, v) + depth → point (x, y, z)

    Use the lens, depth, and camera position to place each visible cube pixel in the scene.

  3. OutputCompare the cube’s movementsurface-00.npy · result.json

    Compare the median of the visible points with frame 0 to estimate movement.

Move a pixel into the 3D scene

Find the ID labeled part in the JSON lookup, select those pixels, and keep their finite positive depths. Each pixel identifies a direction from the camera. The USD focal length and sensor width determine that direction, and the depth locates the visible surface along it. The script assumes a centered camera and square pixels.

Text
Select part in the mask → read column u, row v, and depth
                                     ↓
                          point in camera coordinates
                                     ↓ USD camera position and rotation
                                world (x, y, z)

At pixel (u=613, v=231) in the first saved frame, the depth is about 3.164 m and the semantic ID is 2. That frame’s JSON maps 2 to part, so the pixel contributes to the cube point cloud. A background or workbench pixel is excluded even when it has valid depth. Backprojection script · Source RGB-D frames

Why subtract the first frame?

The cloud contains the visible surface. The apparent center of those visible points can be offset from the cube’s physical center. The checker subtracts frame 0’s median X coordinate to estimate relative motion. In the saved frame 1 result, estimated_relative_x_m ≈ -0.300 and authored_relative_x_m = -0.305. The minus sign means world X decreased, and the two displacements differ by about 5.27 mm.

Read RGB, depth, and a label at one pixel

This is a 3 × 3 crop across the cube and tabletop in the first saved workbench frame. Select an RGB cell to compare the same coordinates.

RGB

Depth (m)

Semantic ID

u=613, v=231 · RGB [155, 63, 14] · 3.164 m · 2 (part)

Brighter depth cells are nearer within this crop. The IDs come from this frame’s label JSON.Source data

First time? Get the lab files

Open a terminal on the Isaac Sim machine for GPU labs, or on your own computer for CPU-only steps. Press Enter after each line. You can copy a multi-line command block as a whole.

Shell
cd "$HOME"
git clone https://github.com/james-joobs/awesome-nvidia-physical-ai.git
cd awesome-nvidia-physical-ai
pwd

This example clones into your home folder. If you already cloned the repo, skip git clone and enter your project folder. In each new terminal, check that pwd ends in awesome-nvidia-physical-ai so commands can find the labs/ files.

Run it

Before you start · Open a terminal on any computer with Python 3.12. This part uses saved captures, so it runs without a GPU.

1. Prepare Python for the point cloud

Enter this repository’s folder. A separate virtual environment keeps these libraries apart from other Python projects.

Shell
python3.12 -m venv .venv-workbench
source .venv-workbench/bin/activate
python -m pip install -r labs/scene-to-data/requirements-check.txt

2. Read four saved camera frames

In the same terminal, run the command below. --recorded uses the bundled USD and camera captures. The checker saves the files and results under runs/workbench-replay.

Shell
python labs/scene-to-data/check_workbench.py runs/workbench-replay --recorded

Open check/frame-00.png in that folder to see the photo, depth, and mask side by side. In check/point-cloud/result.json, estimated_relative_x_m shows the X movement in meters. To make new captures, continue to the workbench capstone. Workbench USD · Point-cloud replay · 12-frame Replicator lab

Check the result

The STEP round trip checks a 1,200 × 600 × 40 mm tabletop. In our run the cube center stopped at 0.8615 m. The four recorded point clouds recovered relative X displacement with a maximum error of about 5.3 mm. It compares the visible surface to the first frame, so the measurement describes movement of the surface rather than absolute cube-center position. Recorded measurements

Example run and recording

Source code

make_step.pyView files on GitHub
Python
"""Create the pilot's STEP file and tessellate the re-imported solids."""
import argparse
import json
from pathlib import Path

import cadquery as cq
from cadquery import exporters, importers

parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("output", type=Path)
folder = parser.parse_args().output
folder.mkdir(parents=True, exist_ok=True)
def box(w, d, h, x, y, z, bevel=0):
    part = cq.Workplane("XY").box(w, d, h)
    if bevel:
        part = part.edges("|Z").fillet(bevel)
    return part.val().translate((x, y, z))

parts = [
    box(1200, 600, 40, 0, 0, 780, 9),
    box(1000, 510, 22, 0, 0, 124, 6),
]
parts.extend(box(48, 48, 758, x, y, 379, 4) for x in (-516,516) for y in (-224,224))
parts.extend(box(1032, 29, 41, 0, y, 721, 4) for y in (-224,224))
parts.extend(box(29, 448, 41, x, 0, 721, 4) for x in (-516,516))
step = folder / "workbench.step"
exporters.export(cq.Compound.makeCompound(parts), str(step))

solids = importers.importStep(str(step)).solids().vals()
assert len(solids) == len(parts)
top = max(solids, key=lambda solid: solid.Center().z)
size = top.BoundingBox()
assert abs(size.xlen-1200) < 0.01 and abs(size.ylen-600) < 0.01 and abs(size.zlen-40) < 0.01
result = []
for number, solid in enumerate(solids):
    name = "Tabletop" if solid is top else "Shelf" if abs(solid.Center().z-124)<1 else f"Frame_{number:02d}"
    vertices, triangles = solid.tessellate(0.7)
    result.append({"name":name, "points":[[u/1000 for u in v.toTuple()] for v in vertices],
                   "indices":[index for face in triangles for index in face]})
(folder/"cad_meshes.json").write_text(json.dumps(result))
print(f"STEP round trip: {len(solids)} solids")
print(f"Measured tabletop (mm): {size.xlen:.2f} x {size.ylen:.2f} x {size.zlen:.2f}")
print(f"STEP: {step.name} | tessellated points: {sum(len(m['points']) for m in result)}")
workbench_final_take.pyView files on GitHub
Python
"""Render the actual CAD-derived scene and Replicator outputs on RTX."""
import argparse
import json
import time
from pathlib import Path

from isaacsim import SimulationApp

parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("mesh", type=Path)
parser.add_argument("output", type=Path)
parser.add_argument("--headless", action="store_true", help="Capture without opening a viewport")
args = parser.parse_args()
cap = args.output.resolve()
cap.mkdir(parents=True, exist_ok=True)
app = SimulationApp({
    "headless": args.headless, "width": 1440, "height": 900, "window_width": 1440, "window_height": 900,
    "renderer": "RaytracedLighting", "anti_aliasing": 2, "fast_shutdown": True,
})

import carb
import numpy as np
from PIL import Image
import omni.physx
import omni.replicator.core as rep
import omni.timeline
import omni.usd
from isaacsim.core.utils.semantics import add_labels
from omni.kit.viewport.utility import get_active_viewport
from pxr import Gf, Sdf, Usd, UsdGeom, UsdLux, UsdPhysics, UsdShade

settings = carb.settings.get_settings()
settings.set_bool("/physics/updateToUsd", True)
settings.set_bool("/omni/replicator/captureOnPlay", False)
omni.usd.get_context().new_stage()
stage = omni.usd.get_context().get_stage()
UsdGeom.SetStageUpAxis(stage, UsdGeom.Tokens.z)
UsdGeom.SetStageMetersPerUnit(stage, 1.0)
world = UsdGeom.Xform.Define(stage,"/World")
stage.SetDefaultPrim(world.GetPrim())
scene = UsdPhysics.Scene.Define(stage,"/World/Physics")
scene.CreateGravityDirectionAttr(Gf.Vec3f(0,0,-1))
scene.CreateGravityMagnitudeAttr(9.81)

def material(name, rgb, roughness, metallic=0.0):
    target = UsdShade.Material.Define(stage, "/World/Looks/"+name)
    shader = UsdShade.Shader.Define(stage,target.GetPath().AppendChild("Shader"))
    shader.CreateIdAttr("UsdPreviewSurface")
    shader.CreateInput("diffuseColor",Sdf.ValueTypeNames.Color3f).Set(Gf.Vec3f(*rgb))
    shader.CreateInput("roughness",Sdf.ValueTypeNames.Float).Set(roughness)
    shader.CreateInput("metallic",Sdf.ValueTypeNames.Float).Set(metallic)
    target.CreateSurfaceOutput().ConnectToSource(shader.ConnectableAPI(),"surface")
    return target

def bind(prim, look):
    UsdShade.MaterialBindingAPI.Apply(prim).Bind(look)

def block(name, location, dimensions, look):
    obj=UsdGeom.Cube.Define(stage, "/World/"+name)
    obj.CreateSizeAttr(1.0)
    obj.AddTranslateOp().Set(Gf.Vec3d(*location))
    obj.AddScaleOp().Set(Gf.Vec3d(*dimensions))
    bind(obj.GetPrim(),look)
    return obj

top_look=material("Top",(.48,.30,.15),.34)
metal=material("Metal",(.039,.16,.19),.31,.54)
shelf=material("Shelf",(.075,.19,.20),.43,.35)
floor=material("Floor",(.072,.099,.12),.56)
orange=material("Orange",(.90,.22,.031),.33,.05)
mint=material("Mark",(.06,.42,.38),.45)
light_label=material("White",(.72,.78,.79),.55)
root=UsdGeom.Xform.Define(stage,"/World/Workbench")
for part in json.loads(args.mesh.read_text()):
    m=UsdGeom.Mesh.Define(stage,root.GetPath().AppendChild(part["name"]))
    m.CreatePointsAttr([Gf.Vec3f(*p) for p in part["points"]])
    m.CreateFaceVertexCountsAttr([3]*(len(part["indices"])//3))
    m.CreateFaceVertexIndicesAttr(part["indices"])
    m.CreateSubdivisionSchemeAttr("none")
    m.CreateDoubleSidedAttr(True)
    bind(m.GetPrim(),top_look if part["name"]=="Tabletop" else shelf if part["name"]=="Shelf" else metal)
    if part["name"]=="Tabletop":
        UsdPhysics.CollisionAPI.Apply(m.GetPrim())
        UsdPhysics.MeshCollisionAPI.Apply(m.GetPrim()).CreateApproximationAttr("convexHull")
add_labels(root.GetPrim(), ["workbench"], instance_name="class")
ground=block("Floor",(0,0,-.037),(40,40,.07),floor)
UsdPhysics.CollisionAPI.Apply(ground.GetPrim())
for i,x in enumerate((-0.83,0.83)):
    block(f"FloorMark{i}",(x,0,.001),(.021,1.65,.002),mint)
for i,y in enumerate((-.82,.82)):
    block(f"CrossMark{i}",(0,y,.001),(1.66,.021,.002),mint)
for i,x in enumerate((-.30,.30)):
    block(f"TopMark{i}",(x,.07,.801),(.007,.27,.002),light_label)
body = UsdGeom.Xform.Define(stage, "/World/TestPart")
cube_location = body.AddTranslateOp()
cube_location.Set(Gf.Vec3d(.125, -.075, 1.23))
cube = UsdGeom.Cube.Define(stage, body.GetPath().AppendChild("Geometry"))
cube.CreateSizeAttr(.123)
bind(cube.GetPrim(), orange)
UsdPhysics.CollisionAPI.Apply(cube.GetPrim())
UsdPhysics.RigidBodyAPI.Apply(body.GetPrim()).CreateRigidBodyEnabledAttr(True)
UsdPhysics.MassAPI.Apply(body.GetPrim()).CreateMassAttr(.21)
add_labels(cube.GetPrim(), ["part"], instance_name="class")

def aim(prim, pos, target):
    xf=Gf.Matrix4d(1).SetLookAt(Gf.Vec3d(*pos),Gf.Vec3d(*target),Gf.Vec3d(0,0,1)).GetInverse()
    UsdGeom.Xformable(prim).AddTransformOp().Set(xf)

camera=UsdGeom.Camera.Define(stage,"/World/InspectionCamera")
aim(camera.GetPrim(), (2.04,-2.42,1.83), (0,0,.61))
camera.CreateFocalLengthAttr(36)
camera.CreateClippingRangeAttr(Gf.Vec2f(.05,35))
ambient=UsdLux.DomeLight.Define(stage,"/World/Ambient")
ambient.CreateIntensityAttr(380)
key=UsdLux.RectLight.Define(stage,"/World/Key")
aim(key.GetPrim(),(-1.35,-1.19,2.82),(0,0,.5))
key.CreateWidthAttr(2.15)
key.CreateHeightAttr(1.65)
key.CreateIntensityAttr(1250)
rim=UsdLux.RectLight.Define(stage,"/World/Rim")
aim(rim.GetPrim(),(1.8,1.4,2.1),(0,0,.67))
rim.CreateWidthAttr(1.9)
rim.CreateHeightAttr(1.6)
rim.CreateIntensityAttr(710)
rim.CreateColorAttr(Gf.Vec3f(.55,.81,1.0))

cap.mkdir(exist_ok=True)
stage.Export(str(cap/"workbench_scene.usda"))
for _ in range(12):
    app.update()
viewport = None if args.headless else get_active_viewport()
if viewport:
    viewport.camera_path = camera.GetPath()
for _ in range(10):
    app.update()
rp=rep.create.render_product(camera.GetPath(),(1280,720))
timeline = omni.timeline.get_timeline_interface()
annotators = {
    "rgb": rep.AnnotatorRegistry.get_annotator("rgb"),
    "distance_to_image_plane": rep.AnnotatorRegistry.get_annotator("distance_to_image_plane"),
    "semantic_segmentation": rep.AnnotatorRegistry.get_annotator("semantic_segmentation",
                                                                 init_params={"colorize": False}),
}
for annotator in annotators.values():
    annotator.attach([rp])
for _ in range(12):
    rep.orchestrator.step(rt_subframes=8, pause_timeline=True)
    sample = np.asarray(annotators["rgb"].get_data())
    if sample.ndim == 3 and sample[:,:,:3].mean() > 5:
        break
else:
    raise RuntimeError("RTX camera did not produce an initialized RGB frame")
print("STAGE_READY: CAD-derived OpenUSD scene, Replicator camera 1280x720", flush=True)
if viewport:
    (cap/"viewport-ready").touch()
    until = time.monotonic() + 4
    while time.monotonic() < until:
        app.update()

def cube_position():
    value = omni.physx.get_physx_interface().get_rigidbody_transformation(str(body.GetPath()))
    return tuple(float(x) for x in value["position"]) if value.get("ret_val",False) else None

timeline.play()
positions = []
for index in range(48):
    rep.orchestrator.step(delta_time=1/30, rt_subframes=2, pause_timeline=False)
    positions.append(cube_position())
    if index in (0, 20, 47):
        print(f"PHYSICS_FRAME {index:02d} | part {positions[-1]}", flush=True)
timeline.stop()
for _ in range(3):
    app.update()

out = cap/"replicator"
out.mkdir(exist_ok=True)
focus_values = []
for frame, x in enumerate((.125, -.18, .23, .03)):
    cube_location.Set(Gf.Vec3d(x, -.075, .8615))
    key.GetIntensityAttr().Set(1100 + 170 * frame)
    for _ in range(3):
        app.update()
    rep.orchestrator.step(delta_time=0, rt_subframes=10, pause_timeline=True)
    rgb = np.asarray(annotators["rgb"].get_data(), np.uint8)
    depth = np.asarray(annotators["distance_to_image_plane"].get_data(), np.float32)
    value = annotators["semantic_segmentation"].get_data()
    ids = np.asarray(value["data"], np.uint32)
    labels = value["info"]["idToLabels"]
    part_labels = [int(i) for i,label in labels.items() if
                   (label.get("class") if isinstance(label,dict) else label) == "part"]
    part = np.isin(ids, part_labels)
    assert rgb.shape[0:2] == depth.shape == ids.shape == (720, 1280)
    assert part.any() and np.isfinite(depth[part]).all(), (frame, labels)
    Image.fromarray(rgb[:,:,:3]).save(out/f"rgb_{frame:04d}.png")
    np.save(out/f"distance_to_image_plane_{frame:04d}.npy",depth)
    np.save(out/f"semantic_segmentation_{frame:04d}.npy",ids)
    (out/f"semantic_segmentation_labels_{frame:04d}.json").write_text(json.dumps(labels))
    approx_part_depth = float(np.median(depth[part]))
    focus_values.append({"frame":frame, "part_pixels":int(part.sum()), "part_depth_m":approx_part_depth})
    print(f"CAPTURE_FRAME {frame:02d} | part pixels {part.sum()} | median depth {approx_part_depth:.3f} m",flush=True)
    if viewport:
        until=time.monotonic()+1.1
        while time.monotonic()<until:
            app.update()

valid = [point[2] for point in positions if point is not None]
expected = .8 + .123/2
summary = {"physics_frames":len(positions),"tabletop_top_m":.8,"expected_rest_center_m":expected,
           "first_cube_z_m":valid[0] if valid else None,"final_cube_z_m":valid[-1] if valid else None,
           "rest_within_2cm":bool(valid and abs(valid[-1]-expected)<.02),
           "replicator_frames":focus_values}
(cap/"workbench-result.json").write_text(json.dumps(summary,indent=2))
print("RESULT",json.dumps(summary),flush=True)
if viewport:
    (cap/"viewport-result").touch()
    until = time.monotonic() + 10
    while time.monotonic() < until:
        app.update()
app.close()
backproject.pyView files on GitHub
Python
"""Lift Isaac Sim camera depth into world coordinates and test relative displacement."""

import argparse
import json
from pathlib import Path

import numpy as np
from pxr import Usd, UsdGeom

TRUE_CUBE_X = [.125, -.18, .23, .03]


def reconstruct(capture, output):
    output.mkdir(parents=True, exist_ok=True)
    stage = Usd.Stage.Open(str(capture / "workbench_scene.usda"))
    camera = UsdGeom.Camera.Get(stage, "/World/InspectionCamera")
    assert camera
    view = np.array(UsdGeom.XformCache().GetLocalToWorldTransform(camera.GetPrim()), dtype=np.float64)
    focal = camera.GetFocalLengthAttr().Get()
    aperture = camera.GetHorizontalApertureAttr().Get()
    assert focal > 0 and aperture > 0
    rows = []
    depths = capture / "replicator"
    for i, x_ref in enumerate(TRUE_CUBE_X):
        d = np.load(depths / f"distance_to_image_plane_{i:04}.npy", allow_pickle=False)
        seg = np.load(depths / f"semantic_segmentation_{i:04}.npy", allow_pickle=False)
        labels = json.loads((depths / f"semantic_segmentation_labels_{i:04}.json").read_text())
        part_ids = [int(key) for key, value in labels.items()
                    if (value.get("class") if isinstance(value, dict) else value) == "part"]
        y, x = np.nonzero(np.isin(seg, part_ids) & np.isfinite(d) & (d > 0))
        height, width = d.shape
        fx = focal / aperture * width
        z = d[y, x]
        local = np.column_stack(((x + .5 - width / 2) * z / fx, (height / 2 - y - .5) * z / fx, -z, np.ones_like(z)))
        world = (local @ view)[:, :3]
        assert world.shape[0] >= 1000 and np.isfinite(world).all()
        midpoint = np.median(world, axis=0)
        np.save(output / f"surface-{i:02}.npy", world[::8].astype(np.float32))
        rows.append({"frame": i, "authored_center_x_m": x_ref, "visible_surface_median_xyz_m": midpoint.tolist(),
                     "valid_surface_pixels": len(world)})
    initial = rows[0]["visible_surface_median_xyz_m"][0]
    for row in rows:
        estimate = row["visible_surface_median_xyz_m"][0] - initial
        reference = row["authored_center_x_m"] - TRUE_CUBE_X[0]
        row.update({"estimated_relative_x_m": estimate, "authored_relative_x_m": reference,
                    "absolute_displacement_error_mm": abs(estimate-reference)*1000})
        print(f"frame {row['frame']} | measured dx {estimate:+.3f} m | authored {reference:+.3f} m", flush=True)
    result = {
        "camera": "/World/InspectionCamera", "depth_channel": "distance_to_image_plane",
        "projection_assumption": "centered camera with horizontal aperture preserved and square pixels",
        "baseline": "frame 0 surface median; used only for displacement, not absolute center estimation",
        "authored_cube_x_source": "workbench_final_take.py",
        "frames": rows,
    }
    (output / "result.json").write_text(json.dumps(result, indent=2) + "\n")
    return result


def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("capture", type=Path)
    parser.add_argument("output", type=Path)
    args = parser.parse_args()
    reconstruct(args.capture, args.output)


if __name__ == "__main__":
    main()
check_workbench.pyView files on GitHub
Python
"""Check the CAD workbench capture on a CPU, from an archive or a fresh run."""

import argparse
import json
import shutil
from math import isfinite
from pathlib import Path
from zipfile import ZipFile

import numpy as np
from PIL import Image
from pxr import Usd, UsdGeom, UsdPhysics

from backproject import TRUE_CUBE_X, reconstruct

LAB = Path(__file__).resolve().parent
SIZE = (1280, 720)
CHANNELS = ("rgb_{:04}.png", "distance_to_image_plane_{:04}.npy",
            "semantic_segmentation_{:04}.npy", "semantic_segmentation_labels_{:04}.json")
FRAME_FILES = [tuple(pattern.format(i) for pattern in CHANNELS) for i in range(len(TRUE_CUBE_X))]


def require(condition, message):
    if not condition:
        raise ValueError(message)


def unpack_recording(capture):
    target = capture / "replicator"
    target.mkdir(parents=True, exist_ok=True)
    shutil.copyfile(LAB / "workbench_scene.usda", capture / "workbench_scene.usda")
    with ZipFile(LAB / "replicator-data.zip") as archive:
        for names in FRAME_FILES:
            for name in names:
                (target / name).write_bytes(archive.read(f"replicator/{name}"))


def check_scene(capture):
    stage = Usd.Stage.Open(str(capture / "workbench_scene.usda"))
    require(UsdGeom.GetStageMetersPerUnit(stage) == 1, "Expected USD scene units in meters")
    require(UsdGeom.GetStageUpAxis(stage) == UsdGeom.Tokens.z, "Expected a Z-up USD stage")
    tabletop = stage.GetPrimAtPath("/World/Workbench/Tabletop")
    require(tabletop and tabletop.HasAPI(UsdPhysics.CollisionAPI), "Missing tabletop collision geometry")
    bounds = UsdGeom.BBoxCache(Usd.TimeCode.Default(), [UsdGeom.Tokens.default_])
    box = bounds.ComputeWorldBound(tabletop).ComputeAlignedBox()
    dimensions = np.array(box.GetSize()) * 1000
    require(np.allclose(dimensions, (1200, 600, 40), rtol=0, atol=.5),
            f"Unexpected tabletop dimensions (mm): {dimensions}")
    cube = UsdGeom.Cube.Get(stage, "/World/TestPart/Geometry")
    require(cube and cube.GetSizeAttr().HasAuthoredValue(), "Missing test cube size")
    expected_z = float(box.GetMax()[2] + cube.GetSizeAttr().Get() / 2)
    return dimensions.tolist(), expected_z


def check_frames(capture, output):
    folder = capture / "replicator"
    expected_rgb = {files[0] for files in FRAME_FILES}
    require({file.name for file in folder.glob("rgb_*.png")} == expected_rgb,
            f"Expected {len(expected_rgb)} numbered RGB frames in {folder}")
    stats = []
    for index, (rgb_file, depth_file, ids_file, labels_file) in enumerate(FRAME_FILES):
        with Image.open(folder / rgb_file) as png:
            rgb = png.convert("RGB")
        depth = np.load(folder / depth_file, allow_pickle=False)
        ids = np.load(folder / ids_file, allow_pickle=False)
        labels = json.loads((folder / labels_file).read_text())
        require(rgb.size == SIZE and depth.shape == ids.shape == SIZE[::-1], f"Frame {index}: mismatched shapes")
        require(depth.dtype.kind == "f" and ids.dtype.kind in "iu", f"Frame {index}: wrong depth or mask dtype")
        part_ids = [int(key) for key, label in labels.items()
                    if (label.get("class") if isinstance(label, dict) else label) == "part"]
        mask = np.isin(ids, part_ids)
        part_depth = depth[mask]
        require(part_depth.size >= 1000, f"Frame {index}: fewer than 1,000 labeled cube pixels")
        require(np.isfinite(part_depth).all() and (part_depth > 0).all(),
                f"Frame {index}: cube depth must be finite and positive")
        stats.append({"frame": index, "part_pixels": int(part_depth.size),
                      "median_camera_depth_m": float(np.median(part_depth))})
        if index == 0:
            valid = np.isfinite(depth) & (depth > 0)
            depth_gray = np.zeros(depth.shape, dtype=np.uint8)
            depth_gray[valid] = (255 * (1 - np.clip(depth[valid] / 6, 0, 1))).astype(np.uint8)
            images = (rgb, Image.fromarray(depth_gray), Image.fromarray(mask.astype(np.uint8) * 255))
            preview = Image.new("RGB", (1280, 240))
            for panel, image in enumerate(images):
                preview.paste(image.resize((426, 240)), (panel * 427, 0))
            preview.save(output / "frame-00.png")
    return stats


def check(capture, recorded):
    if recorded:
        require(not (capture / "workbench-result.json").exists(),
                "Use a separate directory for archived frames to preserve the fresh capture")
    output = capture / "check"
    output.mkdir(parents=True, exist_ok=True)
    report_file = output / "report.json"
    report_file.unlink(missing_ok=True)
    if recorded:
        unpack_recording(capture)

    dimensions, expected_z = check_scene(capture)
    final_z = None
    if not recorded:
        summary = json.loads((capture / "workbench-result.json").read_text())
        initial_z, final_z = summary["first_cube_z_m"], summary["final_cube_z_m"]
        require(initial_z is not None and final_z is not None and isfinite(initial_z) and isfinite(final_z)
                and initial_z > expected_z + .05 and abs(final_z - expected_z) <= .02,
                f"Cube did not settle on the tabletop at {expected_z:.4f} m (final: {final_z})")

    frames = check_frames(capture, output)
    points = reconstruct(capture, output / "point-cloud")
    max_error = max(frame["absolute_displacement_error_mm"] for frame in points["frames"])
    require(isfinite(max_error) and max_error <= 10, f"X displacement error {max_error:.2f} mm exceeds 10 mm")
    report = {
        "input": "recorded" if recorded else "fresh",
        "tabletop_mm": dimensions,
        "resolution_px": SIZE,
        "frames": frames,
        "max_displacement_error_mm": max_error,
        "expected_rest_center_m": expected_z,
        "checked_physics_log": not recorded,
        "final_cube_center_m": final_z,
    }
    report_file.write_text(json.dumps(report, indent=2) + "\n")
    print(f"PASS: {len(frames)} RGB/depth/mask sets at {SIZE[0]}x{SIZE[1]}")
    print(f"PASS: maximum X displacement error {max_error:.2f} mm (limit: 10 mm)")
    if not recorded:
        print(f"PASS: captured cube center {final_z:.4f} m (expected {expected_z:.4f} m)")
    print(f"Report: {report_file}")


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("capture", type=Path, help="Run directory for inputs and generated results")
    parser.add_argument("--recorded", action="store_true", help="Unpack and check the bundled camera frames")
    args = parser.parse_args()
    try:
        check(args.capture, args.recorded)
    except (OSError, ValueError, KeyError) as error:
        parser.exit(1, f"Check failed: {error}\n")