How it works
- InputA camera view of the drawer
BGR image of card, hand, and slotThe lab draws a synthetic camera frame to test the control rule.
- ProcessMake a decision from the image
card seen? · slot occupied?OpenCV checks color in specific image regions. Those checks decide whether the drawer should move.
- OutputMove one joint and keep a trace
slide position · result.jsonThe simulation applies motion to the drawer’s sliding joint. The trace tells you when it moved.
Turn a camera frame into three checks
The script draws the hand, card, and obstruction into a color array that acts as the camera input. OpenCV checks three cropped regions of interest for those features. Within each crop it produces a mask of pixels inside a given BGR (blue, green, red) range. If more than 160 pixels in the card crop match the selected green range, the script sets badge_seen to true. Camera input and control code · OpenCV color range masks
Connect the decision to motion
Detecting a hand activates the control rule. A visible green card or an obstruction at the slot keeps the target closure at 0. With neither condition, the target becomes 1, and the drawer moves toward it a little on each frame.
| Trace field | How to interpret it |
|---|---|
badge_seen, slot_occupied |
Card and obstruction decisions from the image. The same green card yields the same decision for different people. |
drawer_closed_fraction |
Closure from 0 (open) to 1 (closed). |
drawer_final_m |
Final slide distance in the saved scene results. The run with no card and a clear slot moves about 0.305 m. |
The same pattern appears elsewhere in the collection. The cart uses passage width as input, and the desk cleanup uses a rule for what to keep or discard. Change one relevant input and observe which decision changes.
First time? Get the lab files
Open a terminal on the Isaac Sim machine for GPU labs, or on your own computer for CPU-only steps. Press Enter after each line. You can copy a multi-line command block as a whole.
cd "$HOME"
git clone https://github.com/james-joobs/awesome-nvidia-physical-ai.git
cd awesome-nvidia-physical-ai
pwd
This example clones into your home folder. If you already cloned the repo, skip git clone and enter your project folder. In each new terminal, check that pwd ends in awesome-nvidia-physical-ai so commands can find the labs/ files.
Run it
1. Watch the drawer in Isaac Sim
Open the snack drawer USD via File → Open. Switch to FilmCamera and press Play. Watch how the drawer moves when a card is present or an object blocks the slot.
2. Generate a decision log on your computer
From this repository in a terminal, prepare Python and FFmpeg. Check that command -v ffmpeg prints a path, then run the block. This step runs on a CPU.
python3 -m venv .venv-desk
source .venv-desk/bin/activate
python -m pip install numpy opencv-python-headless
python labs/everyday-robots/run_story_demos.py ./runs/stories "$(command -v ffmpeg)"
Open runs/stories/28-snack-guard/result/result.json in a text editor. badge_seen means the card was visible, slot_occupied means the slot was blocked, and drawer_closed_fraction tells you how far it closed.
3. Try another scene
Open the alarm, cleanup, or cart the same way. Keep the cart’s workbench-asset.usda beside its stage. To regenerate stages, use Isaac Sim Python for drawer physics and cleanup, or Python with usd-core for the aisle check.
Check the result
The reversing alarm crossed the edge; the edge-aware mode stayed on top. Only the no-card case closed the drawer by about 30.5 cm. The selective cleanup put the snack in the bin and left the mouse and cup on the desk. The 55 cm and 75 cm aisles fail the 80 cm clearance rule, while the 95 cm aisle passes. Recorded results
Example run and recording
Source code
run_story_demos.py
View files on GitHub"""Run the mobile-alarm and color-badge drawer simulations and render their traces."""
import argparse
import json
import math
import subprocess
from pathlib import Path
import cv2
import numpy as np
FPS = 30
DARK = (21, 23, 25)
TEXT = (237, 241, 240)
GREEN = (114, 221, 158)
AMBER = (108, 185, 247)
def label(frame, message, x, y, size=.55, color=TEXT):
cv2.putText(frame, message, (x, y), cv2.FONT_HERSHEY_SIMPLEX, size, color, 1, cv2.LINE_AA)
def encode(frames, destination, width, height, ffmpeg):
with subprocess.Popen([
str(ffmpeg), "-nostdin", "-hide_banner", "-loglevel", "error", "-y", "-f", "rawvideo",
"-pix_fmt", "bgr24", "-s", f"{width}x{height}", "-r", str(FPS), "-i", "pipe:0",
"-an", "-c:v", "libx264", "-preset", "fast", "-crf", "18", "-pix_fmt", "yuv420p",
"-movflags", "+faststart", str(destination),
], stdin=subprocess.PIPE) as encoder:
assert encoder.stdin is not None
for frame in frames:
assert frame.shape == (height, width, 3)
encoder.stdin.write(frame.tobytes())
encoder.stdin.close()
assert encoder.wait() == 0
def wrap(angle):
return math.atan2(math.sin(angle), math.cos(angle))
def alarm_trace(mode):
wheel_radius, base = .03, .1125
x, y, yaw, z = .53, .30, math.pi, .0
waypoints = [(.93, .30), (.93, .47), (.70, .47)]
index = 0
triggered = False
fall_at = press_at = None
min_clear = math.inf
trace = []
for frame in range(180):
t = frame / FPS
hand = (.10 + min(t * .30, .97), .30)
separation = math.hypot(x - hand[0], y - hand[1])
if fall_at is None and press_at is None and separation <= .075:
press_at = t
triggered = triggered or separation <= .25
v = turn = 0.0
if fall_at is None and press_at is None and triggered:
if mode == "reverse_only":
v = -.40
elif mode == "edge_aware" and index < len(waypoints):
tx, ty = waypoints[index]
if math.hypot(tx - x, ty - y) < .015:
index += 1
else:
err = wrap(math.atan2(ty - y, tx - x) + math.pi - yaw)
turn = float(np.clip(5 * err, -3.5, 3.5))
if abs(err) < .10:
v = -min(.4, 5 * math.hypot(tx - x, ty - y))
left, right = (v - turn * base / 2) / wheel_radius, (v + turn * base / 2) / wheel_radius
if fall_at is None:
speed, omega = wheel_radius * (left + right) / 2, wheel_radius * (right - left) / base
x += speed * math.cos(yaw + omega / (2 * FPS)) / FPS
y += speed * math.sin(yaw + omega / (2 * FPS)) / FPS
yaw = wrap(yaw + omega / FPS)
min_clear = min(min_clear, x - .075, 1.20 - x - .075, y - .075, .60 - y - .075)
if not (0 <= x <= 1.20 and 0 <= y <= .60):
fall_at = t
else:
z = -min(.8, 9.81 * (t - fall_at) ** 2 / 2)
trace.append({"t": t, "pose": [x, y, yaw, z], "hand": hand, "wheel_rad_s": [left, right]})
return {"mode": mode, "fall_s": fall_at, "pressed_s": press_at, "minimum_footprint_clearance_m": min_clear, "trace": trace}
def alarm_frames(rows):
names = {"stationary": "STATIONARY", "reverse_only": "REVERSE ONLY", "edge_aware": "EDGE-AWARE ROUTE"}
for frame_idx in range(len(rows[0]["trace"])):
frame = np.full((414, 1440, 3), DARK, dtype=np.uint8)
label(frame, "RUNAWAY ALARM / SAME HAND TRAJECTORY", 20, 32, .72)
for panel, row in enumerate(rows):
x0, y0, scale = 38 + panel * 480, 124, 324
cv2.rectangle(frame, (x0, y0), (x0 + 389, y0 + 194), (100, 113, 128), -1)
cv2.rectangle(frame, (x0+24, y0+24), (x0+365, y0+170), (135, 149, 164), 1)
label(frame, names[row["mode"]], x0, 89, .67)
now = row["trace"][frame_idx]
t = now["t"]
x, y, yaw, z = now["pose"]
xy = (int(x0 + x * scale), int(y0 + (.6 - y) * scale))
line = [(int(x0 + p["pose"][0] * scale), int(y0 + (.6-p["pose"][1]) * scale)) for p in row["trace"][:frame_idx+1]]
if len(line) > 1:
cv2.polylines(frame, [np.array(line, np.int32)], False, GREEN, 2)
hx, hy = now["hand"]
cv2.circle(frame, (round(x0 + hx*scale), round(y0 + (.60-hy)*scale)), 13, (201, 202, 215), -1, cv2.LINE_AA)
if z > -.7:
cv2.circle(frame, xy, max(9, round(22 - abs(z)*10)), (52, 74, 76), -1, cv2.LINE_AA)
direction = (round(xy[0] + math.cos(yaw)*27), round(xy[1] - math.sin(yaw)*27))
cv2.arrowedLine(frame, xy, direction, AMBER, 4, cv2.LINE_AA)
pressed = row["pressed_s"] is not None and t >= row["pressed_s"]
fell = row["fall_s"] is not None and t >= row["fall_s"]
state = "OVER EDGE" if fell else "BUTTON PRESSED" if pressed else "RUNNING"
label(frame, state, x0, 358, .58, AMBER if fell else GREEN)
label(frame, f"{t:.2f} s | wheel {now['wheel_rad_s'][0]:+.1f} / {now['wheel_rad_s'][1]:+.1f} rad/s", x0, 386, .48)
yield frame
def camera_frame(t, badge, obstruction):
image = np.full((244, 324, 3), (42, 46, 48), np.uint8)
cv2.rectangle(image, (92, 53), (232, 205), (87, 101, 113), -1)
cv2.rectangle(image, (115, 71), (210, 134), (29, 31, 35), -1)
hand_y = max(88, int(235 - t * 92))
cv2.circle(image, (163, hand_y), 22, (30, 108, 219), -1, cv2.LINE_AA)
if badge:
cv2.rectangle(image, (139, hand_y+15), (166, hand_y+41), (45, 195, 73), -1)
if obstruction:
cv2.rectangle(image, (168, 90), (193, 116), (219, 99, 28), -1)
return image
def drawer_control(name, badge, obstruction):
close, active = 0.0, False
any_block_time = any_grant_time = None
frames = []
for index in range(108):
t = index / FPS
camera = camera_frame(t, badge, obstruction)
zone = camera[78:156, 104:218]
hand = int(cv2.countNonZero(cv2.inRange(zone, (10, 75, 180), (70, 145, 245)))) > 120
# Card and slot are camera regions. Actuation does not read the scenario's input flags.
card_roi = camera[80:205, 110:209]
access = int(cv2.countNonZero(cv2.inRange(card_roi, (25, 155, 45), (65, 235, 115)))) > 160
slot = camera[74:132, 115:208]
occupied = int(cv2.countNonZero(cv2.inRange(slot, (170, 65, 5), (245, 136, 65)))) > 120
active = active or hand
if active:
if access and any_grant_time is None:
any_grant_time = t
if not access and occupied and any_block_time is None:
any_block_time = t
goal = 0.0 if access or occupied else 1.0
close += float(np.clip(goal - close, -.034, .034))
frames.append({"t": t, "hand_present": hand, "badge_seen": access, "slot_occupied": occupied, "drawer_closed_fraction": close})
return {"scenario": name, "input_badge": badge, "input_obstruction": obstruction,
"grant_s": any_grant_time, "obstruction_interlock_s": any_block_time,
"max_closed_fraction": max(item["drawer_closed_fraction"] for item in frames),
"final_closed_fraction": frames[-1]["drawer_closed_fraction"], "trace": frames}
def guard_frames(rows):
for i in range(len(rows[0]["trace"])):
frame = np.full((370, 1440, 3), DARK, np.uint8)
label(frame, "SNACK GUARD / WHAT THE RGB CAMERA SEES", 20, 32, .72)
for col, row in enumerate(rows):
x = 18 + 360*col
item = row["trace"][i]
image = camera_frame(item["t"], row["input_badge"], row["input_obstruction"])
cv2.rectangle(image, (104, 78), (218, 156), (178, 183, 188), 1)
frame[75:319, x:x+324] = image
label(frame, row["scenario"], x, 63, .51)
fraction = item["drawer_closed_fraction"]
cv2.rectangle(frame, (x, 326), (x + round(fraction*310), 336), AMBER, -1)
label(frame, f"DRAWER CLOSED {fraction:.0%}", x, 356, .49, AMBER if fraction > .5 else GREEN)
yield frame
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("root", type=Path)
parser.add_argument("ffmpeg", type=Path)
args = parser.parse_args()
alarms = [alarm_trace(name) for name in ("stationary", "reverse_only", "edge_aware")]
assert alarms[0]["pressed_s"] is not None and alarms[1]["fall_s"] is not None
assert alarms[2]["pressed_s"] is None and alarms[2]["fall_s"] is None and alarms[2]["minimum_footprint_clearance_m"] > 0
guards = [drawer_control("OWNER / NO CARD", False, False), drawer_control("OWNER / GREEN CARD", True, False),
drawer_control("GUEST / SAME GREEN CARD", True, False), drawer_control("OBJECT IN SLOT", False, True)]
assert guards[0]["final_closed_fraction"] == 1
assert guards[1]["grant_s"] is not None and guards[2]["grant_s"] is not None
assert guards[3]["obstruction_interlock_s"] is not None and guards[3]["max_closed_fraction"] == 0
for name, rows, media, dimensions in (("27-runaway-alarm", alarms, alarm_frames, (1440, 414)),
("28-snack-guard", guards, guard_frames, (1440, 370))):
folder = args.root / name / "result"
folder.mkdir(parents=True, exist_ok=True)
(folder / "result.json").write_text(json.dumps({"fps": FPS, "cases": rows}, indent=2) + "\n")
encode(media(rows), folder / "comparison.mp4", *dimensions, args.ffmpeg)
for row in rows:
print(name, {key: value for key, value in row.items() if key != "trace"}, flush=True)
if __name__ == "__main__":
main()
physics.py
View files on GitHub"""Compute the falling alarm and the drawer drive response with Isaac Sim PhysX."""
import json
import math
import sys
import traceback
from pathlib import Path
from isaacsim import SimulationApp
app = SimulationApp({"headless": True, "fast_shutdown": True})
from omni.physx import get_physx_interface, get_physx_simulation_interface
from pxr import Gf, Usd, UsdGeom, UsdPhysics, UsdUtils
DT = 1 / 120
def make_stage():
stage = Usd.Stage.CreateInMemory()
UsdGeom.SetStageUpAxis(stage, UsdGeom.Tokens.z)
UsdGeom.SetStageMetersPerUnit(stage, 1)
world = UsdGeom.Xform.Define(stage, "/World").GetPrim()
stage.SetDefaultPrim(world)
scene = UsdPhysics.Scene.Define(stage, "/World/Physics")
scene.CreateGravityDirectionAttr(Gf.Vec3f(0, 0, -1))
scene.CreateGravityMagnitudeAttr(9.81)
return stage
def box(stage, name, center, size):
shape = UsdGeom.Cube.Define(stage, name)
shape.CreateSizeAttr(1.0)
shape.AddTranslateOp().Set(Gf.Vec3d(*center))
shape.AddScaleOp().Set(Gf.Vec3f(*size))
UsdPhysics.CollisionAPI.Apply(shape.GetPrim())
return shape
def body(stage, name, center, size, mass):
prim = UsdGeom.Xform.Define(stage, name)
prim.AddTranslateOp().Set(Gf.Vec3d(*center))
box(stage, name + "/Collision", (0, 0, 0), size)
UsdPhysics.MassAPI.Apply(prim.GetPrim()).CreateMassAttr(mass)
return UsdPhysics.RigidBodyAPI.Apply(prim.GetPrim())
def simulate(stage, count, read, before=None):
cache = UsdUtils.StageCache.Get()
handle = cache.Insert(stage)
driver = get_physx_simulation_interface()
info = get_physx_interface()
driver.attach_stage(handle.ToLongInt())
try:
for i in range(count):
if before:
before(i)
driver.simulate(DT, i * DT)
driver.fetch_results()
if i % 4 == 0:
read(info)
finally:
driver.detach_stage()
cache.Erase(stage)
def alarm(output):
stage = make_stage()
z = .844
box(stage, "/World/Floor", (0, 0, -.06), (20, 20, .12))
# The drive trace ends 1 cm beyond the remaining tabletop edge.
box(stage, "/World/Table", (-.61, 0, .78), (1.2, .60, .04))
buzz = body(stage, "/World/Alarm", (0, 0, z), (.137, .14, .083), .35)
buzz.CreateVelocityAttr(Gf.Vec3f(.40, 0, 0))
samples = [[0, z, 0]]
def read(info):
pose = info.get_rigidbody_transformation("/World/Alarm")
assert pose["ret_val"]
x, _, height = pose["position"]
qx, qy, qz, qw = pose["rotation"]
pitch = math.degrees(math.asin(max(-1, min(1, 2 * (qw*qy-qz*qx)))))
samples.append([float(x), float(height), float(pitch)])
simulate(stage, 272, read)
assert min(row[1] for row in samples) < .12
(output / "alarm-physx.json").write_text(json.dumps({
"engine": "Isaac Sim PhysX", "samples_fps": 30, "start_height_m": z,
"resting_center_z_m": samples[-1][1], "samples": samples,
}, indent=2) + "\n")
print(f"FALL_OK: starts at {z:.3f} m, settles at {samples[-1][1]:.3f} m", flush=True)
def drawers(input_trace, output):
inputs = json.loads(input_trace.read_text())["cases"]
stage = make_stage()
drives, samples = [], [[] for _ in inputs]
for index, _ in enumerate(inputs):
x = index * .82
name = f"/World/Drawer_{index}"
body(stage, name, (x, 0, .85), (.55, .35, .059), .8)
joint = UsdPhysics.PrismaticJoint.Define(stage, f"/World/Slide_{index}")
joint.CreateBody1Rel().SetTargets([name])
joint.CreateAxisAttr("Y")
joint.CreateLocalPos0Attr(Gf.Vec3f(x, 0, .85))
joint.CreateLocalPos1Attr(Gf.Vec3f(0))
joint.CreateLowerLimitAttr(0.0)
joint.CreateUpperLimitAttr(.309)
drive = UsdPhysics.DriveAPI.Apply(joint.GetPrim(), "linear")
drive.CreateTypeAttr("force")
drive.CreateStiffnessAttr(650)
drive.CreateDampingAttr(38)
drive.CreateMaxForceAttr(95)
drives.append(drive.CreateTargetPositionAttr(0))
def before(step):
if step % 4 == 0:
for drive, case in zip(drives, inputs, strict=True):
drive.Set(.305 * case["trace"][step // 4]["drawer_closed_fraction"])
def read(info):
for index, row in enumerate(samples):
pose = info.get_rigidbody_transformation(f"/World/Drawer_{index}")
assert pose["ret_val"]
row.append(max(0.0, float(pose["position"][1])))
count = len(inputs[0]["trace"])
simulate(stage, count * 4, read, before)
assert all(len(row) == count for row in samples)
assert samples[0][-1] > .285 and all(max(row) < .012 for row in samples[1:])
cases = [{"scenario": case["scenario"], "final_position_m": row[-1], "position_m": row}
for case, row in zip(inputs, samples, strict=True)]
(output / "drawer-physx.json").write_text(json.dumps({
"engine": "Isaac Sim PhysX prismatic drive", "sample_rate_hz": 30, "cases": cases,
}, indent=2) + "\n")
print(f"DRAWERS_OK: final positions {[(row['scenario'], round(row['final_position_m'], 4)) for row in cases]}", flush=True)
try:
working = Path(sys.argv[1])
working.mkdir(parents=True, exist_ok=True)
alarm(working)
drawers(Path(sys.argv[2]), working)
except Exception:
traceback.print_exc()
raise
finally:
app.close()
desk_cleanup.py
View files on GitHub"""Compare two cleanup objectives with Isaac Sim and record the PhysX poses."""
import argparse
import json
from pathlib import Path
from isaacsim import SimulationApp
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("output", type=Path)
output = parser.parse_args().output.resolve()
output.mkdir(parents=True, exist_ok=True)
app = SimulationApp({"headless": True, "fast_shutdown": True})
import carb
import numpy as np
from omni.physx import get_physx_interface, get_physx_simulation_interface
from pxr import Gf, Sdf, Usd, UsdGeom, UsdLux, UsdPhysics, UsdShade, UsdUtils
def material(stage, name, color, friction=0.24):
result = UsdShade.Material.Define(stage, f"/World/Looks/{name}")
shader = UsdShade.Shader.Define(stage, result.GetPath().AppendChild("Shader"))
shader.CreateIdAttr("UsdPreviewSurface")
shader.CreateInput("diffuseColor", Sdf.ValueTypeNames.Color3f).Set(Gf.Vec3f(*color))
shader.CreateInput("roughness", Sdf.ValueTypeNames.Float).Set(.45)
result.CreateSurfaceOutput().ConnectToSource(shader.ConnectableAPI(), "surface")
physics = UsdPhysics.MaterialAPI.Apply(result.GetPrim())
physics.CreateStaticFrictionAttr(friction)
physics.CreateDynamicFrictionAttr(friction)
physics.CreateRestitutionAttr(0)
return result
def cube(stage, path, location, size, material, collision=False):
shape = UsdGeom.Cube.Define(stage, path)
shape.CreateSizeAttr(1)
shape.AddTranslateOp().Set(Gf.Vec3d(*location))
shape.AddScaleOp().Set(Gf.Vec3d(*size))
binding = UsdShade.MaterialBindingAPI.Apply(shape.GetPrim())
binding.Bind(material)
if collision:
UsdPhysics.CollisionAPI.Apply(shape.GetPrim())
binding.Bind(material, materialPurpose="physics")
return shape
def main():
carb.settings.get_settings().set_bool("/physics/updateToUsd", True)
stage = Usd.Stage.CreateInMemory()
UsdGeom.SetStageUpAxis(stage, UsdGeom.Tokens.z)
UsdGeom.SetStageMetersPerUnit(stage, 1)
stage.SetDefaultPrim(UsdGeom.Xform.Define(stage, "/World").GetPrim())
physics = UsdPhysics.Scene.Define(stage, "/World/Physics")
physics.CreateGravityDirectionAttr(Gf.Vec3f(0, 0, -1))
physics.CreateGravityMagnitudeAttr(9.81)
table_contact = material(stage, "DeskContact", (.48, .31, .19))
colors = {"snack": material(stage, "Snack", (.93, .22, .045)), "mouse": material(stage, "Mouse", (.08, .17, .30)),
"cup": material(stage, "Cup", (.66, .91, .92)), "bin": material(stage, "Bin", (.20, .52, .26), .45),
"floor": material(stage, "Floor", (.1, .12, .14)), "white": material(stage, "White", (.94, .92, .80))}
cube(stage, "/World/Floor", (0, 0, -.04), (16, 16, .08), colors["floor"], collision=True)
items = {"snack": (-.36, (.11, .14, .04)), "cup": (0, (.09, .09, .13)), "mouse": (.36, (.10, .16, .05))}
scenarios = {}
for name, center in (("ClearEverything", -.82), ("SortTrashOnly", .82)):
prefix = f"/World/{name}"
bench = UsdGeom.Xform.Define(stage, prefix + "/Desk")
bench.GetPrim().GetReferences().AddReference(str(Path(__file__).with_name("workbench-asset.usda").resolve()))
bench.AddTranslateOp().Set(Gf.Vec3d(center, 0, 0))
tabletop = stage.GetPrimAtPath(prefix + "/Desk/Workbench/Tabletop")
assert tabletop.IsValid()
UsdShade.MaterialBindingAPI.Apply(tabletop).Bind(table_contact, materialPurpose="physics")
cube(stage, prefix + "/BinFloor", (center, .79, .42), (1.22, .74, .06), colors["bin"], collision=True)
for index, x in enumerate((center - .62, center + .62)):
cube(stage, prefix + f"/BinSide{index}", (x, .79, .52), (.025, .74, .22), colors["bin"], collision=True)
for index, y in enumerate((.42, 1.16)):
cube(stage, prefix + f"/BinEnd{index}", (center, y, .52), (1.22, .025, .22), colors["bin"], collision=True)
body_paths = {}
for item, (x, size) in items.items():
path = prefix + "/" + item.capitalize()
body = UsdGeom.Xform.Define(stage, path)
body.AddTranslateOp().Set(Gf.Vec3d(center + x, -.065, .803 + size[2] / 2))
cube(stage, path + "/Shape", (0, 0, 0), size, colors[item], collision=True)
accent_z = size[2] / 2 + .001
cube(stage, path + "/Accent", (0, 0, accent_z), (size[0] * .46, size[1] * .34, .003), colors["white"])
rigid = UsdPhysics.RigidBodyAPI.Apply(body.GetPrim())
rigid.CreateVelocityAttr(Gf.Vec3f(0, 2.30 if name == "ClearEverything" or item == "snack" else 0, 0))
UsdPhysics.MassAPI.Apply(body.GetPrim()).CreateMassAttr(.15)
body_paths[item] = path
scenarios[name] = (center, body_paths)
camera = UsdGeom.Camera.Define(stage, "/World/Camera")
pose = Gf.Matrix4d(1).SetLookAt(Gf.Vec3d(0, 4.25, 3.05), Gf.Vec3d(0, .24, .62), Gf.Vec3d(0, 0, 1)).GetInverse()
camera.AddTransformOp().Set(pose)
camera.CreateFocalLengthAttr(24)
ambient = UsdLux.DomeLight.Define(stage, "/World/Ambient")
ambient.CreateIntensityAttr(420)
light = UsdLux.DistantLight.Define(stage, "/World/Key")
light.CreateIntensityAttr(1350)
light.CreateAngleAttr(14)
light.AddRotateXYZOp().Set(Gf.Vec3f(34, -17, 0))
print("DESK_READY | Two CAD-defined workbenches | Same initial objects", flush=True)
cache = UsdUtils.StageCache.Get()
stage_id = cache.Insert(stage).ToLongInt()
driver = get_physx_simulation_interface()
query = get_physx_interface()
body_order = [path for _, body_paths in scenarios.values() for path in body_paths.values()]
poses = []
driver.attach_stage(stage_id)
try:
for frame in range(150):
driver.simulate(1 / 60, frame / 60)
driver.fetch_results()
positions = []
for path in body_order:
pose = query.get_rigidbody_transformation(path)
assert pose["ret_val"], path
positions.append(list(pose["position"]) + list(pose["rotation"]))
poses.append(positions)
if frame in (16, 33, 59, 149):
print(f"PHYSX_STEP {frame + 1}/150", flush=True)
results = {}
for name, (center, body_paths) in scenarios.items():
objects = {}
for item, path in body_paths.items():
pose = query.get_rigidbody_transformation(path)
assert pose["ret_val"], path
x, y, z = list(pose["position"])
on_desk = abs(x - center) < .60 and abs(y) < .30 and z >= .80
in_bin = abs(x - center) < .60 and .42 < y < 1.16 and .44 < z < .66
objects[item] = {"xyz_m": [x, y, z], "on_desk": on_desk, "in_bin": in_bin}
results[name] = {"objects": objects, "remaining_on_desk": sum(entry["on_desk"] for entry in objects.values()),
"task_passed": objects["snack"]["in_bin"] and objects["mouse"]["on_desk"] and objects["cup"]["on_desk"]}
record = {"physics": "Isaac Sim 6.2.0 / PhysX", "simulated_seconds": 2.5,
"actuation": "initial linear velocities on selected objects", "scenarios": results,
"sample_rate_hz": 60, "body_paths": body_order,
"geometry_sizes_m": {kind: list(size) for kind, (_, size) in items.items()}}
assert np.asarray(poses).shape == (150, 6, 7)
assert np.isfinite(poses).all()
np.save(output / "poses.npy", np.asarray(poses, dtype=np.float32))
(output / "result.json").write_text(json.dumps(record, indent=2) + "\n")
print(f"RESULT {json.dumps(record)}", flush=True)
finally:
driver.detach_stage()
cache.Erase(stage)
try:
main()
finally:
app.close()
aisle.py
View files on GitHub"""Compare three OpenUSD layouts using the measured bounds of the CAD workbench."""
import argparse
import hashlib
import json
from pathlib import Path
from pxr import Gf, Sdf, Usd, UsdGeom
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("source", type=Path)
parser.add_argument("output", type=Path)
args = parser.parse_args()
root = args.output
root.mkdir(parents=True, exist_ok=True)
digest = hashlib.sha256(args.source.read_bytes()).hexdigest()
source = Usd.Stage.Open(str(args.source))
asset_layer = Sdf.Layer.CreateAnonymous()
for path in ("/World", "/World/Looks", "/World/Workbench"):
Sdf.CreatePrimInLayer(asset_layer, path)
for path in ("/World/Looks", "/World/Workbench"):
Sdf.CopySpec(source.GetRootLayer(), path, asset_layer, path)
asset = Usd.Stage.Open(asset_layer)
asset.GetPrimAtPath("/World").SetSpecifier(Sdf.SpecifierDef)
asset.SetDefaultPrim(asset.GetPrimAtPath("/World"))
UsdGeom.SetStageUpAxis(asset, UsdGeom.Tokens.z)
UsdGeom.SetStageMetersPerUnit(asset, 1.0)
asset.Export(str(root / "workbench-asset.usda"))
stage = Usd.Stage.CreateInMemory()
scene = UsdGeom.Xform.Define(stage, "/Cell").GetPrim()
stage.SetDefaultPrim(scene)
UsdGeom.SetStageUpAxis(stage, UsdGeom.Tokens.z)
UsdGeom.SetStageMetersPerUnit(stage, 1)
fixed = UsdGeom.Xform.Define(stage, "/Cell/Fixed")
moved = UsdGeom.Xform.Define(stage, "/Cell/Moved")
# Resolve both references from a file-backed version of the stage once authored.
for bench in (fixed, moved):
bench.GetPrim().GetReferences().AddReference(str((root / "workbench-asset.usda").resolve()))
variants = scene.GetVariantSets().AddVariantSet("aisle")
for name, separation in (("narrow", 1.15), ("borderline", 1.35), ("wide", 1.55)):
variants.AddVariant(name)
variants.SetVariantSelection(name)
with variants.GetVariantEditContext():
UsdGeom.Xformable(moved).AddTranslateOp().Set(Gf.Vec3d(0, separation, 0))
variants.SetVariantSelection("wide")
stage.GetRootLayer().Export(str(root / "aisle-layouts.usda"))
path = root / "aisle-layouts.usda"
path.write_text(path.read_text().replace(str((root / "workbench-asset.usda").resolve()), "./workbench-asset.usda"))
measured = Usd.Stage.Open(str(path))
variants = measured.GetDefaultPrim().GetVariantSets().GetVariantSet("aisle")
measurements = []
required = 0.8 # 0.60 m demo cart plus 0.10 m clearance per side.
for name in ("narrow", "borderline", "wide"):
variants.SetVariantSelection(name)
bounds = UsdGeom.BBoxCache(Usd.TimeCode.Default(), [UsdGeom.Tokens.default_])
boxes = []
for prim_path in ("/Cell/Fixed/Workbench", "/Cell/Moved/Workbench"):
prim = measured.GetPrimAtPath(prim_path)
assert prim.IsValid(), prim_path
world = bounds.ComputeWorldBound(prim).ComputeAlignedRange()
boxes.append([list(world.GetMin()), list(world.GetMax())])
gap = boxes[1][0][1] - boxes[0][1][1]
row = {"variant": name, "clear_m": round(gap, 5), "fits_demo_cart": gap >= required, "bounds_m": boxes}
print(f"{name:11s} | clear {gap:.3f} m | {'PASS' if row['fits_demo_cart'] else 'FAIL'}", flush=True)
measurements.append(row)
assert [item["fits_demo_cart"] for item in measurements] == [False, False, True]
assert digest == hashlib.sha256(args.source.read_bytes()).hexdigest()
(root / "result.json").write_text(json.dumps({
"asset_sha256": digest, "cart_width_m": 0.60, "clearance_per_side_m": 0.10,
"required_clearance_m": required, "layouts": measurements,
}, indent=2) + "\n")
if __name__ == "__main__":
main()
