Control a snack drawer with a virtual camera

An alarm rolls away, a snack drawer responds to a card, a cleanup rule puts a mouse in the bin, and a cart tests aisle widths.

  • Isaac Sim
  • PhysX
  • OpenUSD
  • OpenCV
Experiments around a desk
Recorded runExample run and recording

How it works

How the data moves
  1. InputA camera view of the drawerBGR image of card, hand, and slot

    The lab draws a synthetic camera frame to test the control rule.

  2. ProcessMake a decision from the imagecard seen? · slot occupied?

    OpenCV checks color in specific image regions. Those checks decide whether the drawer should move.

  3. OutputMove one joint and keep a traceslide position · result.json

    The simulation applies motion to the drawer’s sliding joint. The trace tells you when it moved.

Turn a camera frame into three checks

The script draws the hand, card, and obstruction into a color array that acts as the camera input. OpenCV checks three cropped regions of interest for those features. Within each crop it produces a mask of pixels inside a given BGR (blue, green, red) range. If more than 160 pixels in the card crop match the selected green range, the script sets badge_seen to true. Camera input and control code · OpenCV color range masks

Connect the decision to motion

Detecting a hand activates the control rule. A visible green card or an obstruction at the slot keeps the target closure at 0. With neither condition, the target becomes 1, and the drawer moves toward it a little on each frame.

Trace field How to interpret it
badge_seen, slot_occupied Card and obstruction decisions from the image. The same green card yields the same decision for different people.
drawer_closed_fraction Closure from 0 (open) to 1 (closed).
drawer_final_m Final slide distance in the saved scene results. The run with no card and a clear slot moves about 0.305 m.

The same pattern appears elsewhere in the collection. The cart uses passage width as input, and the desk cleanup uses a rule for what to keep or discard. Change one relevant input and observe which decision changes.

First time? Get the lab files

Open a terminal on the Isaac Sim machine for GPU labs, or on your own computer for CPU-only steps. Press Enter after each line. You can copy a multi-line command block as a whole.

Shell
cd "$HOME"
git clone https://github.com/james-joobs/awesome-nvidia-physical-ai.git
cd awesome-nvidia-physical-ai
pwd

This example clones into your home folder. If you already cloned the repo, skip git clone and enter your project folder. In each new terminal, check that pwd ends in awesome-nvidia-physical-ai so commands can find the labs/ files.

Run it

1. Watch the drawer in Isaac Sim

Open the snack drawer USD via File → Open. Switch to FilmCamera and press Play. Watch how the drawer moves when a card is present or an object blocks the slot.

2. Generate a decision log on your computer

From this repository in a terminal, prepare Python and FFmpeg. Check that command -v ffmpeg prints a path, then run the block. This step runs on a CPU.

Shell
python3 -m venv .venv-desk
source .venv-desk/bin/activate
python -m pip install numpy opencv-python-headless
python labs/everyday-robots/run_story_demos.py ./runs/stories "$(command -v ffmpeg)"

Open runs/stories/28-snack-guard/result/result.json in a text editor. badge_seen means the card was visible, slot_occupied means the slot was blocked, and drawer_closed_fraction tells you how far it closed.

3. Try another scene

Open the alarm, cleanup, or cart the same way. Keep the cart’s workbench-asset.usda beside its stage. To regenerate stages, use Isaac Sim Python for drawer physics and cleanup, or Python with usd-core for the aisle check.

Check the result

The reversing alarm crossed the edge; the edge-aware mode stayed on top. Only the no-card case closed the drawer by about 30.5 cm. The selective cleanup put the snack in the bin and left the mouse and cup on the desk. The 55 cm and 75 cm aisles fail the 80 cm clearance rule, while the 95 cm aisle passes. Recorded results

Example run and recording

Source code

run_story_demos.pyView files on GitHub
Python
"""Run the mobile-alarm and color-badge drawer simulations and render their traces."""

import argparse
import json
import math
import subprocess
from pathlib import Path

import cv2
import numpy as np

FPS = 30
DARK = (21, 23, 25)
TEXT = (237, 241, 240)
GREEN = (114, 221, 158)
AMBER = (108, 185, 247)


def label(frame, message, x, y, size=.55, color=TEXT):
    cv2.putText(frame, message, (x, y), cv2.FONT_HERSHEY_SIMPLEX, size, color, 1, cv2.LINE_AA)


def encode(frames, destination, width, height, ffmpeg):
    with subprocess.Popen([
        str(ffmpeg), "-nostdin", "-hide_banner", "-loglevel", "error", "-y", "-f", "rawvideo",
        "-pix_fmt", "bgr24", "-s", f"{width}x{height}", "-r", str(FPS), "-i", "pipe:0",
        "-an", "-c:v", "libx264", "-preset", "fast", "-crf", "18", "-pix_fmt", "yuv420p",
        "-movflags", "+faststart", str(destination),
    ], stdin=subprocess.PIPE) as encoder:
        assert encoder.stdin is not None
        for frame in frames:
            assert frame.shape == (height, width, 3)
            encoder.stdin.write(frame.tobytes())
        encoder.stdin.close()
        assert encoder.wait() == 0


def wrap(angle):
    return math.atan2(math.sin(angle), math.cos(angle))


def alarm_trace(mode):
    wheel_radius, base = .03, .1125
    x, y, yaw, z = .53, .30, math.pi, .0
    waypoints = [(.93, .30), (.93, .47), (.70, .47)]
    index = 0
    triggered = False
    fall_at = press_at = None
    min_clear = math.inf
    trace = []
    for frame in range(180):
        t = frame / FPS
        hand = (.10 + min(t * .30, .97), .30)
        separation = math.hypot(x - hand[0], y - hand[1])
        if fall_at is None and press_at is None and separation <= .075:
            press_at = t
        triggered = triggered or separation <= .25
        v = turn = 0.0
        if fall_at is None and press_at is None and triggered:
            if mode == "reverse_only":
                v = -.40
            elif mode == "edge_aware" and index < len(waypoints):
                tx, ty = waypoints[index]
                if math.hypot(tx - x, ty - y) < .015:
                    index += 1
                else:
                    err = wrap(math.atan2(ty - y, tx - x) + math.pi - yaw)
                    turn = float(np.clip(5 * err, -3.5, 3.5))
                    if abs(err) < .10:
                        v = -min(.4, 5 * math.hypot(tx - x, ty - y))
        left, right = (v - turn * base / 2) / wheel_radius, (v + turn * base / 2) / wheel_radius
        if fall_at is None:
            speed, omega = wheel_radius * (left + right) / 2, wheel_radius * (right - left) / base
            x += speed * math.cos(yaw + omega / (2 * FPS)) / FPS
            y += speed * math.sin(yaw + omega / (2 * FPS)) / FPS
            yaw = wrap(yaw + omega / FPS)
            min_clear = min(min_clear, x - .075, 1.20 - x - .075, y - .075, .60 - y - .075)
            if not (0 <= x <= 1.20 and 0 <= y <= .60):
                fall_at = t
        else:
            z = -min(.8, 9.81 * (t - fall_at) ** 2 / 2)
        trace.append({"t": t, "pose": [x, y, yaw, z], "hand": hand, "wheel_rad_s": [left, right]})
    return {"mode": mode, "fall_s": fall_at, "pressed_s": press_at, "minimum_footprint_clearance_m": min_clear, "trace": trace}


def alarm_frames(rows):
    names = {"stationary": "STATIONARY", "reverse_only": "REVERSE ONLY", "edge_aware": "EDGE-AWARE ROUTE"}
    for frame_idx in range(len(rows[0]["trace"])):
        frame = np.full((414, 1440, 3), DARK, dtype=np.uint8)
        label(frame, "RUNAWAY ALARM  /  SAME HAND TRAJECTORY", 20, 32, .72)
        for panel, row in enumerate(rows):
            x0, y0, scale = 38 + panel * 480, 124, 324
            cv2.rectangle(frame, (x0, y0), (x0 + 389, y0 + 194), (100, 113, 128), -1)
            cv2.rectangle(frame, (x0+24, y0+24), (x0+365, y0+170), (135, 149, 164), 1)
            label(frame, names[row["mode"]], x0, 89, .67)
            now = row["trace"][frame_idx]
            t = now["t"]
            x, y, yaw, z = now["pose"]
            xy = (int(x0 + x * scale), int(y0 + (.6 - y) * scale))
            line = [(int(x0 + p["pose"][0] * scale), int(y0 + (.6-p["pose"][1]) * scale)) for p in row["trace"][:frame_idx+1]]
            if len(line) > 1:
                cv2.polylines(frame, [np.array(line, np.int32)], False, GREEN, 2)
            hx, hy = now["hand"]
            cv2.circle(frame, (round(x0 + hx*scale), round(y0 + (.60-hy)*scale)), 13, (201, 202, 215), -1, cv2.LINE_AA)
            if z > -.7:
                cv2.circle(frame, xy, max(9, round(22 - abs(z)*10)), (52, 74, 76), -1, cv2.LINE_AA)
                direction = (round(xy[0] + math.cos(yaw)*27), round(xy[1] - math.sin(yaw)*27))
                cv2.arrowedLine(frame, xy, direction, AMBER, 4, cv2.LINE_AA)
            pressed = row["pressed_s"] is not None and t >= row["pressed_s"]
            fell = row["fall_s"] is not None and t >= row["fall_s"]
            state = "OVER EDGE" if fell else "BUTTON PRESSED" if pressed else "RUNNING"
            label(frame, state, x0, 358, .58, AMBER if fell else GREEN)
            label(frame, f"{t:.2f} s  |  wheel {now['wheel_rad_s'][0]:+.1f} / {now['wheel_rad_s'][1]:+.1f} rad/s", x0, 386, .48)
        yield frame


def camera_frame(t, badge, obstruction):
    image = np.full((244, 324, 3), (42, 46, 48), np.uint8)
    cv2.rectangle(image, (92, 53), (232, 205), (87, 101, 113), -1)
    cv2.rectangle(image, (115, 71), (210, 134), (29, 31, 35), -1)
    hand_y = max(88, int(235 - t * 92))
    cv2.circle(image, (163, hand_y), 22, (30, 108, 219), -1, cv2.LINE_AA)
    if badge:
        cv2.rectangle(image, (139, hand_y+15), (166, hand_y+41), (45, 195, 73), -1)
    if obstruction:
        cv2.rectangle(image, (168, 90), (193, 116), (219, 99, 28), -1)
    return image


def drawer_control(name, badge, obstruction):
    close, active = 0.0, False
    any_block_time = any_grant_time = None
    frames = []
    for index in range(108):
        t = index / FPS
        camera = camera_frame(t, badge, obstruction)
        zone = camera[78:156, 104:218]
        hand = int(cv2.countNonZero(cv2.inRange(zone, (10, 75, 180), (70, 145, 245)))) > 120
        # Card and slot are camera regions. Actuation does not read the scenario's input flags.
        card_roi = camera[80:205, 110:209]
        access = int(cv2.countNonZero(cv2.inRange(card_roi, (25, 155, 45), (65, 235, 115)))) > 160
        slot = camera[74:132, 115:208]
        occupied = int(cv2.countNonZero(cv2.inRange(slot, (170, 65, 5), (245, 136, 65)))) > 120
        active = active or hand
        if active:
            if access and any_grant_time is None:
                any_grant_time = t
            if not access and occupied and any_block_time is None:
                any_block_time = t
            goal = 0.0 if access or occupied else 1.0
            close += float(np.clip(goal - close, -.034, .034))
        frames.append({"t": t, "hand_present": hand, "badge_seen": access, "slot_occupied": occupied, "drawer_closed_fraction": close})
    return {"scenario": name, "input_badge": badge, "input_obstruction": obstruction,
            "grant_s": any_grant_time, "obstruction_interlock_s": any_block_time,
            "max_closed_fraction": max(item["drawer_closed_fraction"] for item in frames),
            "final_closed_fraction": frames[-1]["drawer_closed_fraction"], "trace": frames}


def guard_frames(rows):
    for i in range(len(rows[0]["trace"])):
        frame = np.full((370, 1440, 3), DARK, np.uint8)
        label(frame, "SNACK GUARD  /  WHAT THE RGB CAMERA SEES", 20, 32, .72)
        for col, row in enumerate(rows):
            x = 18 + 360*col
            item = row["trace"][i]
            image = camera_frame(item["t"], row["input_badge"], row["input_obstruction"])
            cv2.rectangle(image, (104, 78), (218, 156), (178, 183, 188), 1)
            frame[75:319, x:x+324] = image
            label(frame, row["scenario"], x, 63, .51)
            fraction = item["drawer_closed_fraction"]
            cv2.rectangle(frame, (x, 326), (x + round(fraction*310), 336), AMBER, -1)
            label(frame, f"DRAWER CLOSED {fraction:.0%}", x, 356, .49, AMBER if fraction > .5 else GREEN)
        yield frame


def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("root", type=Path)
    parser.add_argument("ffmpeg", type=Path)
    args = parser.parse_args()
    alarms = [alarm_trace(name) for name in ("stationary", "reverse_only", "edge_aware")]
    assert alarms[0]["pressed_s"] is not None and alarms[1]["fall_s"] is not None
    assert alarms[2]["pressed_s"] is None and alarms[2]["fall_s"] is None and alarms[2]["minimum_footprint_clearance_m"] > 0
    guards = [drawer_control("OWNER / NO CARD", False, False), drawer_control("OWNER / GREEN CARD", True, False),
              drawer_control("GUEST / SAME GREEN CARD", True, False), drawer_control("OBJECT IN SLOT", False, True)]
    assert guards[0]["final_closed_fraction"] == 1
    assert guards[1]["grant_s"] is not None and guards[2]["grant_s"] is not None
    assert guards[3]["obstruction_interlock_s"] is not None and guards[3]["max_closed_fraction"] == 0
    for name, rows, media, dimensions in (("27-runaway-alarm", alarms, alarm_frames, (1440, 414)),
                                           ("28-snack-guard", guards, guard_frames, (1440, 370))):
        folder = args.root / name / "result"
        folder.mkdir(parents=True, exist_ok=True)
        (folder / "result.json").write_text(json.dumps({"fps": FPS, "cases": rows}, indent=2) + "\n")
        encode(media(rows), folder / "comparison.mp4", *dimensions, args.ffmpeg)
        for row in rows:
            print(name, {key: value for key, value in row.items() if key != "trace"}, flush=True)


if __name__ == "__main__":
    main()
physics.pyView files on GitHub
Python
"""Compute the falling alarm and the drawer drive response with Isaac Sim PhysX."""

import json
import math
import sys
import traceback
from pathlib import Path

from isaacsim import SimulationApp

app = SimulationApp({"headless": True, "fast_shutdown": True})

from omni.physx import get_physx_interface, get_physx_simulation_interface
from pxr import Gf, Usd, UsdGeom, UsdPhysics, UsdUtils

DT = 1 / 120


def make_stage():
    stage = Usd.Stage.CreateInMemory()
    UsdGeom.SetStageUpAxis(stage, UsdGeom.Tokens.z)
    UsdGeom.SetStageMetersPerUnit(stage, 1)
    world = UsdGeom.Xform.Define(stage, "/World").GetPrim()
    stage.SetDefaultPrim(world)
    scene = UsdPhysics.Scene.Define(stage, "/World/Physics")
    scene.CreateGravityDirectionAttr(Gf.Vec3f(0, 0, -1))
    scene.CreateGravityMagnitudeAttr(9.81)
    return stage


def box(stage, name, center, size):
    shape = UsdGeom.Cube.Define(stage, name)
    shape.CreateSizeAttr(1.0)
    shape.AddTranslateOp().Set(Gf.Vec3d(*center))
    shape.AddScaleOp().Set(Gf.Vec3f(*size))
    UsdPhysics.CollisionAPI.Apply(shape.GetPrim())
    return shape


def body(stage, name, center, size, mass):
    prim = UsdGeom.Xform.Define(stage, name)
    prim.AddTranslateOp().Set(Gf.Vec3d(*center))
    box(stage, name + "/Collision", (0, 0, 0), size)
    UsdPhysics.MassAPI.Apply(prim.GetPrim()).CreateMassAttr(mass)
    return UsdPhysics.RigidBodyAPI.Apply(prim.GetPrim())


def simulate(stage, count, read, before=None):
    cache = UsdUtils.StageCache.Get()
    handle = cache.Insert(stage)
    driver = get_physx_simulation_interface()
    info = get_physx_interface()
    driver.attach_stage(handle.ToLongInt())
    try:
        for i in range(count):
            if before:
                before(i)
            driver.simulate(DT, i * DT)
            driver.fetch_results()
            if i % 4 == 0:
                read(info)
    finally:
        driver.detach_stage()
        cache.Erase(stage)


def alarm(output):
    stage = make_stage()
    z = .844
    box(stage, "/World/Floor", (0, 0, -.06), (20, 20, .12))
    # The drive trace ends 1 cm beyond the remaining tabletop edge.
    box(stage, "/World/Table", (-.61, 0, .78), (1.2, .60, .04))
    buzz = body(stage, "/World/Alarm", (0, 0, z), (.137, .14, .083), .35)
    buzz.CreateVelocityAttr(Gf.Vec3f(.40, 0, 0))
    samples = [[0, z, 0]]

    def read(info):
        pose = info.get_rigidbody_transformation("/World/Alarm")
        assert pose["ret_val"]
        x, _, height = pose["position"]
        qx, qy, qz, qw = pose["rotation"]
        pitch = math.degrees(math.asin(max(-1, min(1, 2 * (qw*qy-qz*qx)))))
        samples.append([float(x), float(height), float(pitch)])

    simulate(stage, 272, read)
    assert min(row[1] for row in samples) < .12
    (output / "alarm-physx.json").write_text(json.dumps({
        "engine": "Isaac Sim PhysX", "samples_fps": 30, "start_height_m": z,
        "resting_center_z_m": samples[-1][1], "samples": samples,
    }, indent=2) + "\n")
    print(f"FALL_OK: starts at {z:.3f} m, settles at {samples[-1][1]:.3f} m", flush=True)


def drawers(input_trace, output):
    inputs = json.loads(input_trace.read_text())["cases"]
    stage = make_stage()
    drives, samples = [], [[] for _ in inputs]
    for index, _ in enumerate(inputs):
        x = index * .82
        name = f"/World/Drawer_{index}"
        body(stage, name, (x, 0, .85), (.55, .35, .059), .8)
        joint = UsdPhysics.PrismaticJoint.Define(stage, f"/World/Slide_{index}")
        joint.CreateBody1Rel().SetTargets([name])
        joint.CreateAxisAttr("Y")
        joint.CreateLocalPos0Attr(Gf.Vec3f(x, 0, .85))
        joint.CreateLocalPos1Attr(Gf.Vec3f(0))
        joint.CreateLowerLimitAttr(0.0)
        joint.CreateUpperLimitAttr(.309)
        drive = UsdPhysics.DriveAPI.Apply(joint.GetPrim(), "linear")
        drive.CreateTypeAttr("force")
        drive.CreateStiffnessAttr(650)
        drive.CreateDampingAttr(38)
        drive.CreateMaxForceAttr(95)
        drives.append(drive.CreateTargetPositionAttr(0))

    def before(step):
        if step % 4 == 0:
            for drive, case in zip(drives, inputs, strict=True):
                drive.Set(.305 * case["trace"][step // 4]["drawer_closed_fraction"])

    def read(info):
        for index, row in enumerate(samples):
            pose = info.get_rigidbody_transformation(f"/World/Drawer_{index}")
            assert pose["ret_val"]
            row.append(max(0.0, float(pose["position"][1])))

    count = len(inputs[0]["trace"])
    simulate(stage, count * 4, read, before)
    assert all(len(row) == count for row in samples)
    assert samples[0][-1] > .285 and all(max(row) < .012 for row in samples[1:])
    cases = [{"scenario": case["scenario"], "final_position_m": row[-1], "position_m": row}
             for case, row in zip(inputs, samples, strict=True)]
    (output / "drawer-physx.json").write_text(json.dumps({
        "engine": "Isaac Sim PhysX prismatic drive", "sample_rate_hz": 30, "cases": cases,
    }, indent=2) + "\n")
    print(f"DRAWERS_OK: final positions {[(row['scenario'], round(row['final_position_m'], 4)) for row in cases]}", flush=True)


try:
    working = Path(sys.argv[1])
    working.mkdir(parents=True, exist_ok=True)
    alarm(working)
    drawers(Path(sys.argv[2]), working)
except Exception:
    traceback.print_exc()
    raise
finally:
    app.close()
desk_cleanup.pyView files on GitHub
Python
"""Compare two cleanup objectives with Isaac Sim and record the PhysX poses."""

import argparse
import json
from pathlib import Path

from isaacsim import SimulationApp

parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("output", type=Path)
output = parser.parse_args().output.resolve()
output.mkdir(parents=True, exist_ok=True)
app = SimulationApp({"headless": True, "fast_shutdown": True})

import carb
import numpy as np
from omni.physx import get_physx_interface, get_physx_simulation_interface
from pxr import Gf, Sdf, Usd, UsdGeom, UsdLux, UsdPhysics, UsdShade, UsdUtils


def material(stage, name, color, friction=0.24):
    result = UsdShade.Material.Define(stage, f"/World/Looks/{name}")
    shader = UsdShade.Shader.Define(stage, result.GetPath().AppendChild("Shader"))
    shader.CreateIdAttr("UsdPreviewSurface")
    shader.CreateInput("diffuseColor", Sdf.ValueTypeNames.Color3f).Set(Gf.Vec3f(*color))
    shader.CreateInput("roughness", Sdf.ValueTypeNames.Float).Set(.45)
    result.CreateSurfaceOutput().ConnectToSource(shader.ConnectableAPI(), "surface")
    physics = UsdPhysics.MaterialAPI.Apply(result.GetPrim())
    physics.CreateStaticFrictionAttr(friction)
    physics.CreateDynamicFrictionAttr(friction)
    physics.CreateRestitutionAttr(0)
    return result


def cube(stage, path, location, size, material, collision=False):
    shape = UsdGeom.Cube.Define(stage, path)
    shape.CreateSizeAttr(1)
    shape.AddTranslateOp().Set(Gf.Vec3d(*location))
    shape.AddScaleOp().Set(Gf.Vec3d(*size))
    binding = UsdShade.MaterialBindingAPI.Apply(shape.GetPrim())
    binding.Bind(material)
    if collision:
        UsdPhysics.CollisionAPI.Apply(shape.GetPrim())
        binding.Bind(material, materialPurpose="physics")
    return shape


def main():
    carb.settings.get_settings().set_bool("/physics/updateToUsd", True)
    stage = Usd.Stage.CreateInMemory()
    UsdGeom.SetStageUpAxis(stage, UsdGeom.Tokens.z)
    UsdGeom.SetStageMetersPerUnit(stage, 1)
    stage.SetDefaultPrim(UsdGeom.Xform.Define(stage, "/World").GetPrim())
    physics = UsdPhysics.Scene.Define(stage, "/World/Physics")
    physics.CreateGravityDirectionAttr(Gf.Vec3f(0, 0, -1))
    physics.CreateGravityMagnitudeAttr(9.81)
    table_contact = material(stage, "DeskContact", (.48, .31, .19))
    colors = {"snack": material(stage, "Snack", (.93, .22, .045)), "mouse": material(stage, "Mouse", (.08, .17, .30)),
              "cup": material(stage, "Cup", (.66, .91, .92)), "bin": material(stage, "Bin", (.20, .52, .26), .45),
              "floor": material(stage, "Floor", (.1, .12, .14)), "white": material(stage, "White", (.94, .92, .80))}
    cube(stage, "/World/Floor", (0, 0, -.04), (16, 16, .08), colors["floor"], collision=True)
    items = {"snack": (-.36, (.11, .14, .04)), "cup": (0, (.09, .09, .13)), "mouse": (.36, (.10, .16, .05))}
    scenarios = {}
    for name, center in (("ClearEverything", -.82), ("SortTrashOnly", .82)):
        prefix = f"/World/{name}"
        bench = UsdGeom.Xform.Define(stage, prefix + "/Desk")
        bench.GetPrim().GetReferences().AddReference(str(Path(__file__).with_name("workbench-asset.usda").resolve()))
        bench.AddTranslateOp().Set(Gf.Vec3d(center, 0, 0))
        tabletop = stage.GetPrimAtPath(prefix + "/Desk/Workbench/Tabletop")
        assert tabletop.IsValid()
        UsdShade.MaterialBindingAPI.Apply(tabletop).Bind(table_contact, materialPurpose="physics")
        cube(stage, prefix + "/BinFloor", (center, .79, .42), (1.22, .74, .06), colors["bin"], collision=True)
        for index, x in enumerate((center - .62, center + .62)):
            cube(stage, prefix + f"/BinSide{index}", (x, .79, .52), (.025, .74, .22), colors["bin"], collision=True)
        for index, y in enumerate((.42, 1.16)):
            cube(stage, prefix + f"/BinEnd{index}", (center, y, .52), (1.22, .025, .22), colors["bin"], collision=True)
        body_paths = {}
        for item, (x, size) in items.items():
            path = prefix + "/" + item.capitalize()
            body = UsdGeom.Xform.Define(stage, path)
            body.AddTranslateOp().Set(Gf.Vec3d(center + x, -.065, .803 + size[2] / 2))
            cube(stage, path + "/Shape", (0, 0, 0), size, colors[item], collision=True)
            accent_z = size[2] / 2 + .001
            cube(stage, path + "/Accent", (0, 0, accent_z), (size[0] * .46, size[1] * .34, .003), colors["white"])
            rigid = UsdPhysics.RigidBodyAPI.Apply(body.GetPrim())
            rigid.CreateVelocityAttr(Gf.Vec3f(0, 2.30 if name == "ClearEverything" or item == "snack" else 0, 0))
            UsdPhysics.MassAPI.Apply(body.GetPrim()).CreateMassAttr(.15)
            body_paths[item] = path
        scenarios[name] = (center, body_paths)

    camera = UsdGeom.Camera.Define(stage, "/World/Camera")
    pose = Gf.Matrix4d(1).SetLookAt(Gf.Vec3d(0, 4.25, 3.05), Gf.Vec3d(0, .24, .62), Gf.Vec3d(0, 0, 1)).GetInverse()
    camera.AddTransformOp().Set(pose)
    camera.CreateFocalLengthAttr(24)
    ambient = UsdLux.DomeLight.Define(stage, "/World/Ambient")
    ambient.CreateIntensityAttr(420)
    light = UsdLux.DistantLight.Define(stage, "/World/Key")
    light.CreateIntensityAttr(1350)
    light.CreateAngleAttr(14)
    light.AddRotateXYZOp().Set(Gf.Vec3f(34, -17, 0))
    print("DESK_READY | Two CAD-defined workbenches | Same initial objects", flush=True)
    cache = UsdUtils.StageCache.Get()
    stage_id = cache.Insert(stage).ToLongInt()
    driver = get_physx_simulation_interface()
    query = get_physx_interface()
    body_order = [path for _, body_paths in scenarios.values() for path in body_paths.values()]
    poses = []
    driver.attach_stage(stage_id)
    try:
        for frame in range(150):
            driver.simulate(1 / 60, frame / 60)
            driver.fetch_results()
            positions = []
            for path in body_order:
                pose = query.get_rigidbody_transformation(path)
                assert pose["ret_val"], path
                positions.append(list(pose["position"]) + list(pose["rotation"]))
            poses.append(positions)
            if frame in (16, 33, 59, 149):
                print(f"PHYSX_STEP {frame + 1}/150", flush=True)
        results = {}
        for name, (center, body_paths) in scenarios.items():
            objects = {}
            for item, path in body_paths.items():
                pose = query.get_rigidbody_transformation(path)
                assert pose["ret_val"], path
                x, y, z = list(pose["position"])
                on_desk = abs(x - center) < .60 and abs(y) < .30 and z >= .80
                in_bin = abs(x - center) < .60 and .42 < y < 1.16 and .44 < z < .66
                objects[item] = {"xyz_m": [x, y, z], "on_desk": on_desk, "in_bin": in_bin}
            results[name] = {"objects": objects, "remaining_on_desk": sum(entry["on_desk"] for entry in objects.values()),
                             "task_passed": objects["snack"]["in_bin"] and objects["mouse"]["on_desk"] and objects["cup"]["on_desk"]}
        record = {"physics": "Isaac Sim 6.2.0 / PhysX", "simulated_seconds": 2.5,
                  "actuation": "initial linear velocities on selected objects", "scenarios": results,
                  "sample_rate_hz": 60, "body_paths": body_order,
                  "geometry_sizes_m": {kind: list(size) for kind, (_, size) in items.items()}}
        assert np.asarray(poses).shape == (150, 6, 7)
        assert np.isfinite(poses).all()
        np.save(output / "poses.npy", np.asarray(poses, dtype=np.float32))
        (output / "result.json").write_text(json.dumps(record, indent=2) + "\n")
        print(f"RESULT {json.dumps(record)}", flush=True)
    finally:
        driver.detach_stage()
        cache.Erase(stage)


try:
    main()
finally:
    app.close()
aisle.pyView files on GitHub
Python
"""Compare three OpenUSD layouts using the measured bounds of the CAD workbench."""

import argparse
import hashlib
import json
from pathlib import Path

from pxr import Gf, Sdf, Usd, UsdGeom


def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("source", type=Path)
    parser.add_argument("output", type=Path)
    args = parser.parse_args()
    root = args.output
    root.mkdir(parents=True, exist_ok=True)
    digest = hashlib.sha256(args.source.read_bytes()).hexdigest()
    source = Usd.Stage.Open(str(args.source))
    asset_layer = Sdf.Layer.CreateAnonymous()
    for path in ("/World", "/World/Looks", "/World/Workbench"):
        Sdf.CreatePrimInLayer(asset_layer, path)
    for path in ("/World/Looks", "/World/Workbench"):
        Sdf.CopySpec(source.GetRootLayer(), path, asset_layer, path)
    asset = Usd.Stage.Open(asset_layer)
    asset.GetPrimAtPath("/World").SetSpecifier(Sdf.SpecifierDef)
    asset.SetDefaultPrim(asset.GetPrimAtPath("/World"))
    UsdGeom.SetStageUpAxis(asset, UsdGeom.Tokens.z)
    UsdGeom.SetStageMetersPerUnit(asset, 1.0)
    asset.Export(str(root / "workbench-asset.usda"))

    stage = Usd.Stage.CreateInMemory()
    scene = UsdGeom.Xform.Define(stage, "/Cell").GetPrim()
    stage.SetDefaultPrim(scene)
    UsdGeom.SetStageUpAxis(stage, UsdGeom.Tokens.z)
    UsdGeom.SetStageMetersPerUnit(stage, 1)
    fixed = UsdGeom.Xform.Define(stage, "/Cell/Fixed")
    moved = UsdGeom.Xform.Define(stage, "/Cell/Moved")
    # Resolve both references from a file-backed version of the stage once authored.
    for bench in (fixed, moved):
        bench.GetPrim().GetReferences().AddReference(str((root / "workbench-asset.usda").resolve()))
    variants = scene.GetVariantSets().AddVariantSet("aisle")
    for name, separation in (("narrow", 1.15), ("borderline", 1.35), ("wide", 1.55)):
        variants.AddVariant(name)
        variants.SetVariantSelection(name)
        with variants.GetVariantEditContext():
            UsdGeom.Xformable(moved).AddTranslateOp().Set(Gf.Vec3d(0, separation, 0))
    variants.SetVariantSelection("wide")
    stage.GetRootLayer().Export(str(root / "aisle-layouts.usda"))
    path = root / "aisle-layouts.usda"
    path.write_text(path.read_text().replace(str((root / "workbench-asset.usda").resolve()), "./workbench-asset.usda"))

    measured = Usd.Stage.Open(str(path))
    variants = measured.GetDefaultPrim().GetVariantSets().GetVariantSet("aisle")
    measurements = []
    required = 0.8  # 0.60 m demo cart plus 0.10 m clearance per side.
    for name in ("narrow", "borderline", "wide"):
        variants.SetVariantSelection(name)
        bounds = UsdGeom.BBoxCache(Usd.TimeCode.Default(), [UsdGeom.Tokens.default_])
        boxes = []
        for prim_path in ("/Cell/Fixed/Workbench", "/Cell/Moved/Workbench"):
            prim = measured.GetPrimAtPath(prim_path)
            assert prim.IsValid(), prim_path
            world = bounds.ComputeWorldBound(prim).ComputeAlignedRange()
            boxes.append([list(world.GetMin()), list(world.GetMax())])
        gap = boxes[1][0][1] - boxes[0][1][1]
        row = {"variant": name, "clear_m": round(gap, 5), "fits_demo_cart": gap >= required, "bounds_m": boxes}
        print(f"{name:11s} | clear {gap:.3f} m | {'PASS' if row['fits_demo_cart'] else 'FAIL'}", flush=True)
        measurements.append(row)
    assert [item["fits_demo_cart"] for item in measurements] == [False, False, True]
    assert digest == hashlib.sha256(args.source.read_bytes()).hexdigest()
    (root / "result.json").write_text(json.dumps({
        "asset_sha256": digest, "cart_width_m": 0.60, "clearance_per_side_m": 0.10,
        "required_clearance_m": required, "layouts": measurements,
    }, indent=2) + "\n")


if __name__ == "__main__":
    main()