Create a short video from an Isaac Sim image, then run a separate driving model on six camera views from a self-authored intersection. You can inspect the saved model products without a GPU. These are two independent experiments: the Cosmos video is not fed into Alpamayo.
How it works
- InputTwo independent inputs
Cosmos: RGB + camera actions; Alpamayo: 22 RGB frames + ego historyBoth examples use scenes we authored. They test different model interfaces.
- ProcessRun released model weights
Cosmos3-Edge · Alpamayo2-EdgeCosmos generates pixels. Alpamayo proposes future vehicle positions and rotations.
- OutputInspect each result
61-frame MP4 · 64 future posesCheck decoded frames and finite tensors. The car shown in Isaac Sim replays the predicted trajectory.
An image is a condition, not a simulated physical state
Cosmos3-Edge receives one RGB image and a description of what should happen. It denoises a latent video and decodes RGB frames. Our input contains a yellow cube, a green cube and a dark block on a workbench.
Isaac Sim camera → one RGB image + motion prompt → Cosmos3-Edge → generated RGB video
same image and seed, change only "left" / "right"
The requested output has 49 frames at 24 FPS, about 2.04 seconds. The model does not return a USD scene, contact forces or verified collision events. A decoded video confirms that inference finished; visual review determines whether the requested motion happened. Official inference contract
Give the world model explicit camera actions
The action-conditioned example adds a float32 [60, 9] sequence to the same image. Each row carries three native normalized translation channels and a 6D rotation representation. These values are already prepared for the model; they are not raw displacements in meters and must not be normalized again. We preserve NVIDIA's published camera example unchanged, verify its SHA-256 (547b977c…83c5595), and let Framework pad the nine channels to its model action width. Pinned official camera example
Our RGB image + NVIDIA's 60×9 camera-action sequence → forward_dynamics → 61 RGB frames
30 FPS
The first frame is the observation; the following 60 are predicted views. The recorded output is 832×464, after the model's padding/cropping, at the requested 480 resolution tier. It changes the camera view visibly. It still provides generated RGB, not a reconstructed 3D scene or a physics-based guarantee that the requested camera path is exact.
Camera history becomes a proposed trajectory
Alpamayo2-Edge combines camera observations, recent ego motion and navigation text. Its vision-language backbone generates driving reasoning, and its action expert predicts motion. The upstream decoder integrates its predicted acceleration and curvature into future poses. Our intersection uses original primitive geometry and no gated driving recordings. Official model card
6 cameras / 22 RGB frames ─┐
20 historical ego poses ───┼→ Alpamayo2-Edge → reasoning text + 64 future poses
"Continue straight..." ───┘ → inspect the saved tensors
| Data in this lab | Shape and meaning |
|---|---|
| Camera images | CPU uint8 [22, 1, 3, 360, 640]; 22 frames, one frame per entry, RGB, height, width. |
| Camera IDs and timestamps | int64 [22]; IDs 0/1/2/3/5/6 have 3/5/3/3/3/5 frames. The two front cameras cover −2, −1, −0.5, −0.1 and 0 seconds; other views cover −1, −0.1 and 0 seconds. |
| Position history | float32 [1, 20, 3]; ego-frame XYZ at 0.1-second steps, accompanied by [1, 20, 3, 3] rotation matrices. Our synthetic camera rig moves at 5 m/s. |
| Predicted positions | [1, 1, 1, 64, 3]; one batch, one trajectory set, one candidate, 64 XYZ poses from 0.1 to 6.4 seconds. Rotation output is [1, 1, 1, 64, 3, 3]. |
For example, a predicted position [5.0, 0.4, 0.0] means 5 m forward and 0.4 m left of the reference ego pose. This example explains the coordinate convention; it is not a measured prediction. Our rendered road is a synthetic interface test, outside the model's real driving-data evaluation. It checks actual weight loading and valid outputs. It does not test a car in closed loop.
First time? Get the lab files
Open a terminal on the Isaac Sim machine for GPU labs, or on your own computer for CPU-only steps. Press Enter after each line. You can copy a multi-line command block as a whole.
cd "$HOME"
git clone https://github.com/james-joobs/awesome-nvidia-physical-ai.git
cd awesome-nvidia-physical-ai
pwd
This example clones into your home folder. If you already cloned the repo, skip git clone and enter your project folder. In each new terminal, check that pwd ends in awesome-nvidia-physical-ai so commands can find the labs/ files.
Run
1. Prepare a dedicated model folder on a Linux GPU machine
Open a terminal in this repository on the GPU machine. Check GPU and driver setup, then choose an empty folder outside the repository for model weights and virtual environments. The recorded machine has an RTX 5090 with 32 GiB. The commands pin the source and model revisions in model-pins.json.
export WORLD_DRIVING_ROOT="$HOME/physical-ai-experiments/world-driving"
bash labs/world-driving/setup.sh
Setup creates separate Cosmos and Alpamayo environments. Cosmos uses the locked CUDA 13.0 inference dependencies; Alpamayo uses its Python 3.12 lockfile. The public weights total about 13.5 GiB; the recorded environments add about 16 GiB, with caches and a separately downloaded VAE requiring more space. Allow at least 60 GiB for this subset. Read the exact checkpoint terms before using the weights. Cosmos setup · Alpamayo setup
2. Generate two videos from one image
Run each line in the same terminal. Both runs use seed 19 and the same bundled, self-authored simulator image. Only the requested slide direction changes. Each script waits for the session's shared GPU lock, so it can run alongside other queued labs without competing for GPU memory.
bash labs/world-driving/run_cosmos.sh right
bash labs/world-driving/run_cosmos.sh left
Open outputs/cosmos-right/workbench-right/vision.mp4 under WORLD_DRIVING_ROOT. Keep both clips, even if one does not follow its prompt. The script turns off the optional guardrail models for this controlled synthetic input; deployed applications need their own guardrail configuration.
To add the explicit camera actions, run this third command. It downloads only the pinned 12 KB action sidecar, checks its digest and shape, then calls the same released Edge checkpoint.
bash labs/world-driving/run_camera_action.sh
3. Render your own driving-model inputs in Isaac Sim
Set ISAAC_LAB_PYTHON to a Python executable with Isaac Lab and Isaac Sim installed. The recorded input-rendering environment uses Isaac Sim 6.1.0.0 / Isaac Lab 17.0.2 and AppLauncher --viz kit. It is separate from the site's Isaac Sim 6.2 workbench guide. Follow the simulator's license prompt on first use.
export ISAAC_LAB_PYTHON="/path/to/IsaacLab/.venv/bin/python"
flock /tmp/physical-ai-four-domains.gpu.lock \
"$ISAAC_LAB_PYTHON" labs/world-driving/make_driving_scene.py \
--viz kit --output "$WORLD_DRIVING_ROOT/inputs/synthetic-intersection"
The GUI opens the road scene and renders 22 images from the camera positions and times in the table. The folder also receives input.json and synthetic-intersection.usda. To record the actual simulator window on Linux X11, add --record-window /absolute/path/to/ffmpeg. Use a terminal in your graphical desktop session so DISPLAY and XAUTHORITY refer to that session.
4. Run the released driving-model weights
This reads your generated camera images and history, loads the pinned BF16 checkpoint and writes both reasoning and trajectories. It uses the default PyTorch SDPA attention backend, without accelerated-inference compilation.
python3 labs/cli.py --json run alpamayo-synthetic \
--python "$WORLD_DRIVING_ROOT/alpamayo2-edge/.venv/bin/python" \
--input "$WORLD_DRIVING_ROOT/inputs/synthetic-intersection" \
--model "$WORLD_DRIVING_ROOT/checkpoints/Alpamayo2-Edge" \
--seed 19 --output "$WORLD_DRIVING_ROOT/outputs/alpamayo-cli"
Open outputs/alpamayo-cli/output.json for reasoning and output.safetensors for numeric predictions. The receipt records GPU, package versions, memory and exact input/output shapes. The runner verifies both weight shards against their published SHA-256 values before loading. In the recorded run the adaptive reasoning output is <NOMINAL>, rather than a longer explanation. To perform an actual closed-loop driving evaluation, use the separate AlpaSim guide and obtain the required scene access.
Check the result
Inspect saved products on a CPU
You only need Python and FFmpeg/FFprobe for the checker. Point it at a generation folder or an Alpamayo output folder. It fully decodes the video, checks requested frame count and FPS, and independently reads the saved trajectory tensors to reject nonfinite coordinates or wrong shapes.
python3 labs/world-driving/check.py \
--cosmos labs/world-driving/evidence/cosmos-camera-actions \
--alpamayo labs/world-driving/evidence/alpamayo-synthetic
Those paths use the bundled public outputs, so the command works without installing the model runtimes. Replace them with your own output folders after a GPU run. The recorded CLI prediction took 13.37 seconds including input processing, with 5.76 GiB peak PyTorch allocation; this is one synthetic sample on one RTX 5090, not a general performance benchmark.
The saved evidence/prediction-scene/alpamayo-trajectory.usda draws the 64 predicted positions on our road and animates a cosmetic vehicle proxy. Its transforms come from the saved model tensors. The Isaac Sim recording of this scene is trajectory replay, with no new driver decisions or vehicle dynamics. Replay generator · Recorded checks
Compare the right things
- Cosmos: compare the initial and last frames, cube direction, camera motion and object identity. Report a prompt-following failure even when
sample_outputs.jsonsayssuccess. - Alpamayo: check that all weights loaded, the saved tensors are finite, and positions use the documented ego frame. A plausible explanation and a curved path do not establish safe driving.
- Access: the November 19 audit confirmed NuRec scene access on the recording machine, while the separate PhysicalAI-AV source dataset returned HTTP 403. The public synthetic test does not need either dataset. Their access approvals and terms are independent; dataset-derived video and metrics remain local.
Source and version pins · Cosmos · Alpamayo · AlpaSim closed loop
Example run and recording
Source code
setup.sh
View files on GitHub#!/usr/bin/env bash
set -euo pipefail
task_root="${WORLD_DRIVING_ROOT:?Choose an isolated folder with at least 60 GiB free for weights, runtimes and caches}"
uv_bin="${UV_BIN:-$HOME/.local/bin/uv}"
mkdir -p "$task_root"
cd "$task_root"
clone_pin() {
local directory="$1" url="$2" revision="$3"
if [[ ! -d "$directory/.git" ]]; then
git clone "$url" "$directory"
fi
if [[ -n "$(git -C "$directory" status --porcelain)" ]]; then
printf 'Stop: %s has local changes. Choose an empty task folder.\n' "$directory" >&2
return 1
fi
git -C "$directory" fetch origin "$revision"
git -C "$directory" checkout --detach "$revision"
}
clone_pin cosmos-framework https://github.com/NVIDIA/cosmos-framework.git afd7d4c4087a8929c84523ff60172642fd11690b
clone_pin alpamayo2-edge https://github.com/NVlabs/alpamayo2-edge.git 02f55a7aeabc1a2b9f7a80445c08e40bddc89f5e
(
cd cosmos-framework
"$uv_bin" tool run --from uv==0.11.29 uv sync --locked --group cu130 --extra guardrail --no-default-groups --python 3.13
)
(
cd alpamayo2-edge
"$uv_bin" sync --locked --extra dataset --no-default-groups
)
cosmos-framework/.venv/bin/hf download nvidia/Cosmos3-Edge \
--revision 6afc1256321c3a8e623d89be3bc5117a1e90f96b \
--local-dir checkpoints/Cosmos3-Edge
alpamayo2-edge/.venv/bin/hf download nvidia/Alpamayo2-Edge \
--revision b92b6aa326bd3e2235db28f248ad5a30b5890e64 \
--local-dir checkpoints/Alpamayo2-Edge
model-pins.json
View files on GitHub{
"checked_at": "2026-11-19",
"camera_action_sidecar": {
"url": "https://media.githubusercontent.com/media/nvidia-cosmos/cosmos-dependencies/2b17a2413bd86b2cf9b03823637108851e4ddf2d/inputs/action/camera_action_44.json",
"sha256": "547b977ccd6d435d579ba83e82e58618d0bebf2e8e7d8d577bf2e9c8883c5595",
"shape": [60, 9],
"semantics": "Published native normalized camera-pose sequence: three translation channels and six rotation channels. Preserve unchanged; do not assume translation is raw meters."
},
"cosmos": {
"source": "https://github.com/NVIDIA/cosmos-framework",
"source_revision": "afd7d4c4087a8929c84523ff60172642fd11690b",
"model": "nvidia/Cosmos3-Edge",
"model_revision": "6afc1256321c3a8e623d89be3bc5117a1e90f96b",
"gated": false,
"weights_gib": 8.5,
"terms": "OpenMDW-1.1; the Hugging Face metadata license field is other, with a linked license",
"weights_license_url": "https://openmdw.ai/license/1-1/",
"code_license_url": "https://github.com/NVIDIA/cosmos-framework/blob/afd7d4c4087a8929c84523ff60172642fd11690b/LICENSE",
"requirements": "Dedicated Python 3.13, locked cu130 inference group. Driver must support CUDA 13.0."
},
"alpamayo": {
"source": "https://github.com/NVlabs/alpamayo2-edge",
"source_revision": "02f55a7aeabc1a2b9f7a80445c08e40bddc89f5e",
"model": "nvidia/Alpamayo2-Edge",
"model_revision": "b92b6aa326bd3e2235db28f248ad5a30b5890e64",
"gated": false,
"weights_gib": 4.94,
"code_license": "Apache-2.0",
"weights_license": "OpenMDW-1.1",
"weights_license_url": "https://huggingface.co/nvidia/Alpamayo2-Edge/blob/b92b6aa326bd3e2235db28f248ad5a30b5890e64/LICENSE",
"code_license_url": "https://github.com/NVlabs/alpamayo2-edge/blob/02f55a7aeabc1a2b9f7a80445c08e40bddc89f5e/LICENSE",
"weight_sha256": {
"00000.safetensors": "d13c1a427218baf7b7d365098c0f5496c0e5e6a6bf662836b8eebbe7eb7cfe7f",
"00001.safetensors": "cfe424a7320ce729cd927b5ceaab4323a62443b50f461a2f5c0dd0b10634e0b5"
},
"requirements": "Python 3.12, locked torch 2.8.0 and Transformers 5.17.0; upstream recommends >=16 GiB for BF16 example, validated there on H100."
},
"large_models_deferred": {
"nvidia/Cosmos3-Nano-Sim-Bimanual": {
"model_revision": "da4d2c09175e2a6dd5b237265318066d5134312e",
"weights_gib": 29.57,
"reason": "Weights alone almost fill a 32 GiB GPU; runtime memory was not validated."
},
"nvidia/Cosmos3-Nano-Sim-Transfer": {
"model_revision": "45510d5b8cefa338b7abc01eb702cf1e9cdf6742",
"weights_gib": 29.54,
"reason": "Weights alone almost fill a 32 GiB GPU; runtime memory was not validated."
}
}
}
make_cosmos_input.py
View files on GitHub"""Prepare a short, explicitly generative Cosmos test from our own simulator image."""
import argparse
import hashlib
import json
from pathlib import Path
parser = argparse.ArgumentParser()
parser.add_argument("--image", type=Path, required=True)
parser.add_argument("--output", type=Path, required=True)
parser.add_argument("--direction", choices=["left", "right"], default="right")
args = parser.parse_args()
args.output.parent.mkdir(parents=True, exist_ok=True)
manifest = {
"model_mode": "image2video",
"prompt": (
"A fixed camera watches a wooden workbench in a clean robotics simulation. "
"A yellow cube is in the center, a mint green cube at the rear left and a tall dark block on the right. "
f"The yellow cube slides slowly to the {args.direction} across the table. "
"The green cube and the dark block remain stationary. "
"Keep the camera fixed, preserve the tabletop, lighting, object colors and sizes. "
"Continuous motion, no cuts, no added objects."
),
"vision_path": str(args.image.resolve()),
"format_prompt_as_json": False,
"prompt_upsampling": False,
"num_frames": 49,
"fps": 24,
"resolution": "480",
"num_steps": 16,
}
args.output.write_text(json.dumps(manifest, indent=2) + "\n")
receipt = {
"kind": "image-conditioned video generation; not a physics simulation or robot policy",
"input_sha256": hashlib.sha256(args.image.read_bytes()).hexdigest(),
"input_origin": "Our self-authored Isaac Sim Replicator workbench scene, frame 05",
"manifest": manifest,
}
args.output.with_suffix(".provenance.json").write_text(json.dumps(receipt, indent=2) + "\n")
make_camera_action_input.py
View files on GitHub"""Pair our simulator image with NVIDIA's pinned native camera-action example.
Use the published normalized sequence unchanged: inventing metric units or
normalizing this sequence a second time would change the input contract.
"""
import argparse
import hashlib
import json
import math
from pathlib import Path
import subprocess
parser = argparse.ArgumentParser()
parser.add_argument("--image", type=Path, required=True)
parser.add_argument("--output", type=Path, required=True)
args = parser.parse_args()
args.output.parent.mkdir(parents=True, exist_ok=True)
revision = "2b17a2413bd86b2cf9b03823637108851e4ddf2d"
url = f"https://media.githubusercontent.com/media/nvidia-cosmos/cosmos-dependencies/{revision}/inputs/action/camera_action_44.json"
payload = subprocess.check_output(["curl", "-fsSL", "--max-time", "60", url])
digest = hashlib.sha256(payload).hexdigest()
assert digest == "547b977ccd6d435d579ba83e82e58618d0bebf2e8e7d8d577bf2e9c8883c5595", digest
actions = json.loads(payload)
assert len(actions) == 60 and all(len(row) == 9 for row in actions)
assert all(math.isfinite(value) for row in actions for value in row)
action_path = args.output.with_name("camera-action-60x9.json")
action_path.write_bytes(payload)
spec = {
"name": "workbench-camera-actions",
"action_chunk_size": 60,
"action_path": str(action_path.resolve()),
"domain_name": "camera_pose",
"fps": 30,
"view_point": "ego_view",
"guidance": 1.0,
"image_size": 480,
"resolution": "480",
"model_mode": "forward_dynamics",
"num_frames": 61,
"num_steps": 30,
"prompt": "A wooden workbench holds a yellow cube, a mint green cube and a tall dark block. The camera moves around the stationary objects.",
"shift": 5.0,
"vision_path": str(args.image.resolve()),
}
args.output.write_text(json.dumps(spec, indent=2) + "\n")
args.output.with_suffix(".provenance.json").write_text(json.dumps({
"input_origin": "Our self-authored Isaac Sim Replicator workbench scene, frame 05",
"action_origin": url,
"action_sha256": digest,
"action_shape": [60, 9],
"action_columns": "native normalized camera-pose channels: 3 translation + 6D rotation; not raw metric displacements",
"normalization": "Preserve NVIDIA's published values unchanged. Framework pads the native channels to its model action width.",
"output": "Initial observation plus 60 predicted RGB frames; not a physics solver or a new 3D reconstruction",
}, indent=2) + "\n")
make_driving_scene.py
View files on GitHub"""Render our own six-camera, 22-frame input in Isaac Sim; no driving dataset needed."""
import argparse
import hashlib
import json
from importlib.metadata import version
import math
import os
from pathlib import Path
import signal
import subprocess
from isaaclab.app import AppLauncher
parser = argparse.ArgumentParser()
parser.add_argument("--output", type=Path, required=True)
parser.add_argument("--record-window", type=Path, help="FFmpeg executable for native Isaac Sim GUI recording")
AppLauncher.add_app_launcher_args(parser)
args = parser.parse_args()
args.enable_cameras = True
launcher = AppLauncher(args)
app = launcher.app
import numpy as np
from PIL import Image
import omni.replicator.core as rep
import omni.usd
from omni.kit.viewport.utility import get_active_viewport
from pxr import Gf, Sdf, UsdGeom, UsdLux, UsdShade
args.output.mkdir(parents=True, exist_ok=True)
omni.usd.get_context().new_stage()
stage = omni.usd.get_context().get_stage()
UsdGeom.SetStageUpAxis(stage, UsdGeom.Tokens.z)
UsdGeom.SetStageMetersPerUnit(stage, 1.0)
world = UsdGeom.Xform.Define(stage, "/World")
stage.SetDefaultPrim(world.GetPrim())
def material(name, rgb):
mat = UsdShade.Material.Define(stage, "/World/Looks/" + name)
shader = UsdShade.Shader.Define(stage, mat.GetPath().AppendChild("Shader"))
shader.CreateIdAttr("UsdPreviewSurface")
shader.CreateInput("diffuseColor", Sdf.ValueTypeNames.Color3f).Set(Gf.Vec3f(*rgb))
shader.CreateInput("roughness", Sdf.ValueTypeNames.Float).Set(.7)
mat.CreateSurfaceOutput().ConnectToSource(shader.ConnectableAPI(), "surface")
return mat
def block(name, center, size, mat):
shape = UsdGeom.Cube.Define(stage, "/World/" + name)
shape.CreateSizeAttr(1)
shape.AddTranslateOp().Set(Gf.Vec3d(*center))
shape.AddScaleOp().Set(Gf.Vec3d(*size))
UsdShade.MaterialBindingAPI.Apply(shape.GetPrim()).Bind(mat)
return shape
asphalt = material("Asphalt", (.12, .14, .16))
grass = material("Grass", (.13, .24, .13))
white = material("White", (.9, .91, .88))
yellow = material("Yellow", (.96, .7, .05))
red = material("RedCar", (.62, .045, .035))
blue = material("BlueCar", (.02, .18, .4))
glass = material("Glass", (.05, .12, .16))
concrete = material("Concrete", (.42, .45, .43))
block("Ground", (20, 0, -.12), (160, 160, .2), grass)
block("MainRoad", (20, 0, .005), (150, 8, .05), asphalt)
block("CrossRoad", (28, 0, .02), (8, 100, .05), asphalt)
for index, x in enumerate(range(-45, 96, 6)):
if 22 < x < 34:
continue
block(f"CenterLine{index}", (x, 0, .04), (3, .12, .015), yellow)
for index, y in enumerate((-3.65, 3.65)):
for side, x in enumerate((-12, 65)):
block(f"Edge{index}_{side}", (x, y, .04), (68, .1, .015), white)
for index, y in enumerate(np.arange(-3.1, 3.2, .75)):
block(f"Crosswalk{index}", (22.8, float(y), .06), (.8, .42, .015), white)
for index, (x, y, h) in enumerate(((42, -15, 12), (56, -14, 8), (40, 17, 10), (-18, 16, 6), (-25, -14, 7))):
block(f"Building{index}", (x, y, h / 2), (9, 9, h), concrete)
for index, (x, y, body) in enumerate(((17, 2.05, blue), (28, -12, red))):
block(f"Car{index}Body", (x, y, .65), (4.2, 1.8, .9), body)
block(f"Car{index}Cabin", (x - .2, y, 1.2), (2, 1.6, .7), glass)
light = UsdLux.DomeLight.Define(stage, "/World/Sky")
light.CreateIntensityAttr(1400)
sun = UsdLux.DistantLight.Define(stage, "/World/Sun")
sun.CreateIntensityAttr(2600)
sun.AddRotateXYZOp().Set(Gf.Vec3f(315, 15, 20))
width, height = 640, 360
camera = UsdGeom.Camera.Define(stage, "/World/Camera")
camera.CreateHorizontalApertureAttr(20.955)
camera.CreateClippingRangeAttr(Gf.Vec2f(.1, 250))
camera_op = camera.AddTransformOp()
product = rep.create.render_product(str(camera.GetPath()), (width, height))
annotator = rep.AnnotatorRegistry.get_annotator("rgb")
annotator.attach([product])
get_active_viewport().camera_path = str(camera.GetPath())
profiles = {0: (55, (-1, -.1, 0), 12), 1: (0, (-2, -1, -.5, -.1, 0), 14),
2: (-55, (-1, -.1, 0), 12), 3: (145, (-1, -.1, 0), 12),
5: (-145, (-1, -.1, 0), 12), 6: (0, (-2, -1, -.5, -.1, 0), 30)}
shots = []
recorder = None
record_log = None
if args.record_window:
camera_op.Set(Gf.Matrix4d(1).SetLookAt(Gf.Vec3d(0, -1.8, 1.6), Gf.Vec3d(30, -1.8, 1.4), Gf.Vec3d(0, 0, 1)).GetInverse())
for _ in range(35):
app.update()
windows = subprocess.check_output(["xwininfo", "-root", "-tree"], text=True)
rows = [row for row in windows.splitlines() if '"Isaac Lab ' in row and "mutter-x11-frames" not in row]
if not rows:
raise RuntimeError("Native Isaac Sim window was not found; use --viz kit")
identifier = str(int(rows[0].split()[0], 16))
record_log = (args.output / "record-window.log").open("w")
recorder = subprocess.Popen([
str(args.record_window), "-nostdin", "-hide_banner", "-loglevel", "warning", "-y",
"-f", "x11grab", "-draw_mouse", "0", "-framerate", "24", "-window_id", identifier,
"-i", f"{os.environ['DISPLAY']}.0", "-an", "-c:v", "libx264", "-preset", "fast", "-crf", "22",
"-pix_fmt", "yuv420p", "-movflags", "+faststart", str(args.output / "six-camera-isaac-gui.mp4"),
], stdout=record_log, stderr=subprocess.STDOUT)
for camera_id, (degrees, times, focal) in profiles.items():
for offset in times:
eye = Gf.Vec3d(offset * 5, -1.8, 1.6)
yaw = math.radians(degrees)
target = eye + Gf.Vec3d(math.cos(yaw) * 30, math.sin(yaw) * 30, -.2)
camera_op.Set(Gf.Matrix4d(1).SetLookAt(eye, target, Gf.Vec3d(0, 0, 1)).GetInverse())
camera.GetFocalLengthAttr().Set(focal)
for _ in range(14):
app.update()
rgb = np.asarray(annotator.get_data())[:, :, :3].copy()
assert rgb.shape == (height, width, 3) and rgb.mean() > 3, (camera_id, offset, rgb.shape)
name = f"camera-{camera_id}-t{round((offset + 2) * 1000000):07d}.png"
Image.fromarray(rgb).save(args.output / name)
shots.append({"camera_id": camera_id, "timestamp_us": round((offset + 2) * 1000000),
"relative_time_s": offset, "image": name,
"sha256": hashlib.sha256((args.output / name).read_bytes()).hexdigest()})
print(f"RENDER camera={camera_id} t={offset:+.1f}s file={name}", flush=True)
stage.GetRootLayer().Export(str(args.output / "synthetic-intersection.usda"))
(args.output / "input.json").write_text(json.dumps({
"kind": "self-authored synthetic interface smoke; not a driving benchmark",
"isaac_sim_version": version("isaacsim"),
"isaac_lab_version": version("isaaclab"),
"coordinate_frame": "ego x forward, y left, z up; ego reference at query time",
"width": width, "height": height, "ego_speed_m_s": 5, "shots": shots,
"history_times_s": [round(-1.9 + .1 * i, 3) for i in range(20)],
"history_xyz_m": [[round((-1.9 + .1 * i) * 5, 3), 0, 0] for i in range(20)],
}, indent=2) + "\n")
annotator.detach([product])
if recorder:
recorder.send_signal(signal.SIGINT)
recorder.wait(timeout=20)
record_log.close()
app.close()
run_cosmos.sh
View files on GitHub#!/usr/bin/env bash
set -euo pipefail
task_root="${WORLD_DRIVING_ROOT:?Set WORLD_DRIVING_ROOT to the isolated GPU-machine folder}"
lab_root="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
direction="${1:-right}"
input_image="${2:-$lab_root/input/workbench.png}"
mkdir -p "$task_root/inputs" "$task_root/logs"
python3 "$lab_root/make_cosmos_input.py" --image "$input_image" \
--output "$task_root/inputs/workbench-$direction.json" --direction "$direction"
export COSMOS_TRAINING=0
export PATH="$task_root/cosmos-framework/.venv/bin:$PATH"
cd "$task_root/cosmos-framework"
# Use the same lock for every other GPU job in this recording session.
flock /tmp/physical-ai-four-domains.gpu.lock \
.venv/bin/python -m cosmos_framework.scripts.inference \
-i "$task_root/inputs/workbench-$direction.json" \
-o "$task_root/outputs/cosmos-$direction" \
--checkpoint-path "$task_root/checkpoints/Cosmos3-Edge" \
--parallelism-preset=latency --seed=19 \
--no-use-torch-compile --no-guardrails
run_camera_action.sh
View files on GitHub#!/usr/bin/env bash
set -euo pipefail
task_root="${WORLD_DRIVING_ROOT:?Set WORLD_DRIVING_ROOT to the isolated GPU-machine folder}"
lab_root="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
python3 "$lab_root/make_camera_action_input.py" --image "$lab_root/input/workbench.png" \
--output "$task_root/inputs/workbench-camera-actions.json"
export COSMOS_TRAINING=0
export PATH="$task_root/cosmos-framework/.venv/bin:$PATH"
cd "$task_root/cosmos-framework"
flock /tmp/physical-ai-four-domains.gpu.lock \
.venv/bin/python -m cosmos_framework.scripts.inference \
-i "$task_root/inputs/workbench-camera-actions.json" \
-o "$task_root/outputs/cosmos-camera-actions" \
--checkpoint-path "$task_root/checkpoints/Cosmos3-Edge" \
--parallelism-preset=latency --seed=19 \
--no-use-torch-compile --no-guardrails
run_alpamayo_synthetic.py
View files on GitHub"""Use released Alpamayo2-Edge weights on self-authored Isaac Sim camera frames.
This checks the model interface and output shapes, not driving quality or safety.
"""
import argparse
import hashlib
import json
from importlib.metadata import version
from pathlib import Path
import subprocess
import time
import numpy as np
from PIL import Image
import torch
from alpamayo2_edge.checkpoint import load_model
from alpamayo2_edge.inference import run_data
parser = argparse.ArgumentParser()
parser.add_argument("--input", type=Path, required=True)
parser.add_argument("--model", type=Path, required=True)
parser.add_argument("--output", type=Path, required=True)
parser.add_argument("--nav-text", default="Continue straight through the intersection.")
parser.add_argument("--seed", type=int, default=43)
args = parser.parse_args()
pins = json.loads(Path(__file__).with_name("model-pins.json").read_text())["alpamayo"]
weight_digests = {}
for filename, expected in pins["weight_sha256"].items():
digest = hashlib.sha256()
with (args.model / filename).open("rb") as stream:
for block in iter(lambda: stream.read(1024 * 1024), b""):
digest.update(block)
weight_digests[filename] = digest.hexdigest()
if digest.hexdigest() != expected:
raise ValueError(f"Checkpoint shard {filename} differs from the documented model revision")
manifest = json.loads((args.input / "input.json").read_text())
images = np.stack([np.asarray(Image.open(args.input / row["image"]).convert("RGB"))
for row in manifest["shots"]])
data = {
"image_frames": torch.from_numpy(images).permute(0, 3, 1, 2).unsqueeze(1).contiguous(),
"camera_indices": torch.tensor([row["camera_id"] for row in manifest["shots"]], dtype=torch.int64),
"absolute_timestamps": torch.tensor([row["timestamp_us"] for row in manifest["shots"]], dtype=torch.int64),
"ego_history_xyz": torch.tensor([manifest["history_xyz_m"]], dtype=torch.float32),
"ego_history_rot": torch.eye(3).reshape(1, 1, 3, 3).repeat(1, 20, 1, 1),
"nav_text": [args.nav_text],
}
args.output.mkdir(parents=True, exist_ok=True)
began = time.perf_counter()
torch.cuda.reset_peak_memory_stats()
model, loading_info = load_model(str(args.model.resolve()), device="cuda", dtype="bfloat16",
attn_implementation="sdpa", local_files_only=True)
settings = dict(seed=args.seed, top_p=.98, top_k=None, temperature=.6, num_traj_sets=1,
num_traj_samples=1, max_generation_length=192,
diffusion_kwargs={"inference_step": 10})
result = run_data(model, data, "isaac-synthetic-intersection", args.output, settings, "cuda")
receipt = {
"scope": "actual Alpamayo2-Edge model inference on a self-authored synthetic interface test; not driving evaluation",
"upstream_source": "https://github.com/NVlabs/alpamayo2-edge",
"upstream_revision": "02f55a7aeabc1a2b9f7a80445c08e40bddc89f5e",
"model_id": "nvidia/Alpamayo2-Edge",
"model_revision": "b92b6aa326bd3e2235db28f248ad5a30b5890e64",
"verified_weight_sha256": weight_digests,
"gpu": torch.cuda.get_device_name(0),
"peak_allocated_gib": torch.cuda.max_memory_allocated() / 2**30,
"total_elapsed_s": time.perf_counter() - began,
"versions": {key: version(key) for key in ("torch", "torchvision", "transformers", "safetensors")},
"input_shapes": {key: list(value.shape) for key, value in data.items() if isinstance(value, torch.Tensor)},
"navigation": args.nav_text,
"seed": args.seed,
"result": result,
}
(args.output / "receipt.json").write_text(json.dumps(receipt, indent=2) + "\n")
print(json.dumps(receipt), flush=True)
run_alpamayo.sh
View files on GitHub#!/usr/bin/env bash
set -euo pipefail
task_root="${WORLD_DRIVING_ROOT:?Set WORLD_DRIVING_ROOT to the isolated GPU-machine folder}"
lab_root="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
test -f "$task_root/inputs/synthetic-intersection/input.json"
mkdir -p "$task_root/outputs/alpamayo-synthetic"
flock /tmp/physical-ai-four-domains.gpu.lock \
"$task_root/alpamayo2-edge/.venv/bin/python" "$lab_root/run_alpamayo_synthetic.py" \
--input "$task_root/inputs/synthetic-intersection" \
--model "$task_root/checkpoints/Alpamayo2-Edge" \
--output "$task_root/outputs/alpamayo-synthetic"
make_prediction_scene.py
View files on GitHub"""Show saved Alpamayo predictions as an animated USD proxy, without a model or GPU.
This is a replay of predicted poses. No vehicle physics or closed-loop policy
execution is added. Requires numpy and the USD Python bindings (usd-core).
"""
import argparse
import csv
import hashlib
import json
import math
from pathlib import Path
import struct
import numpy as np
from pxr import Gf, Sdf, Usd, UsdGeom, UsdShade
parser = argparse.ArgumentParser()
parser.add_argument("--scene", type=Path, required=True)
parser.add_argument("--prediction", type=Path, required=True)
parser.add_argument("--output", type=Path, required=True)
args = parser.parse_args()
args.output.mkdir(parents=True, exist_ok=True)
content = args.prediction.read_bytes()
header_size = struct.unpack("<Q", content[:8])[0]
header = json.loads(content[8:8 + header_size])
numeric = content[8 + header_size:]
def tensor(name):
spec = header[name]
assert spec["dtype"] == "F32", (name, spec)
begin, end = spec["data_offsets"]
return np.frombuffer(numeric[begin:end], dtype="<f4").reshape(spec["shape"])
xyz = tensor("pred_xyz")[0, 0, 0]
rot = tensor("pred_rot")[0, 0, 0]
assert xyz.shape == (64, 3) and rot.shape == (64, 3, 3)
assert np.isfinite(xyz).all() and np.isfinite(rot).all()
stage = Usd.Stage.Open(str(args.scene))
stage.SetStartTimeCode(0)
stage.SetEndTimeCode(192)
stage.SetTimeCodesPerSecond(30)
stage.SetFramesPerSecond(30)
def material(name, color, emissive=False):
mat = UsdShade.Material.Define(stage, "/World/Looks/" + name)
shader = UsdShade.Shader.Define(stage, mat.GetPath().AppendChild("Shader"))
shader.CreateIdAttr("UsdPreviewSurface")
shader.CreateInput("diffuseColor", Sdf.ValueTypeNames.Color3f).Set(Gf.Vec3f(*color))
shader.CreateInput("roughness", Sdf.ValueTypeNames.Float).Set(.55)
if emissive:
shader.CreateInput("emissiveColor", Sdf.ValueTypeNames.Color3f).Set(Gf.Vec3f(*color))
mat.CreateSurfaceOutput().ConnectToSource(shader.ConnectableAPI(), "surface")
return mat
orange = material("PredictedRoute", (1, .36, .015), True)
lime = material("PredictedCar", (.3, .8, .045))
glass = material("PredictedGlass", (.02, .09, .12))
points = [Gf.Vec3f(0, -1.8, .13)] + [Gf.Vec3f(float(x), float(y - 1.8), float(z + .13)) for x, y, z in xyz]
path = UsdGeom.BasisCurves.Define(stage, "/World/PredictedRoute")
path.CreateTypeAttr("linear")
path.CreateCurveVertexCountsAttr([len(points)])
path.CreatePointsAttr(points)
path.CreateWidthsAttr([.15])
path.SetWidthsInterpolation(UsdGeom.Tokens.constant)
UsdShade.MaterialBindingAPI.Apply(path.GetPrim()).Bind(orange)
proxy = UsdGeom.Xform.Define(stage, "/World/PredictedVehicleProxy")
position = proxy.AddTranslateOp()
orientation = proxy.AddOrientOp()
for name, location, scale, look in (("Body", (.3, 0, .65), (4.2, 1.8, .9), lime),
("Cabin", (-.1, 0, 1.22), (2.0, 1.6, .65), glass)):
block = UsdGeom.Cube.Define(stage, proxy.GetPath().AppendChild(name))
block.CreateSizeAttr(1)
block.AddTranslateOp().Set(Gf.Vec3d(*location))
block.AddScaleOp().Set(Gf.Vec3d(*scale))
UsdShade.MaterialBindingAPI.Apply(block.GetPrim()).Bind(look)
position.Set(Gf.Vec3d(0, -1.8, 0), 0)
orientation.Set(Gf.Quatf(1), 0)
rows = []
for index, (point, rotation) in enumerate(zip(xyz, rot, strict=True)):
seconds = (index + 1) / 10
yaw = math.atan2(float(rotation[1, 0]), float(rotation[0, 0]))
position.Set(Gf.Vec3d(float(point[0]), float(point[1] - 1.8), float(point[2])), seconds * 30)
orientation.Set(Gf.Quatf(math.cos(yaw / 2), Gf.Vec3f(0, 0, math.sin(yaw / 2))), seconds * 30)
rows.append([seconds, *map(float, point), yaw])
center_x = float((xyz[:, 0].min() + xyz[:, 0].max()) / 2)
center_y = float((xyz[:, 1].min() + xyz[:, 1].max()) / 2) - 1.8
span = max(float(np.ptp(xyz[:, 0])), float(np.ptp(xyz[:, 1])), 20)
target = Gf.Vec3d(center_x, center_y, .6)
eye = target + Gf.Vec3d(-span * .8, -span * .9, span)
camera = UsdGeom.Camera.Define(stage, "/World/FilmCamera")
camera.CreateFocalLengthAttr(24)
camera.CreateHorizontalApertureAttr(20.955)
camera.CreateClippingRangeAttr(Gf.Vec2f(.1, 300))
camera.AddTransformOp().Set(Gf.Matrix4d(1).SetLookAt(eye, target, Gf.Vec3d(0, 0, 1)).GetInverse())
stage.Export(str(args.output / "alpamayo-trajectory.usda"))
(args.output / "takes.json").write_text(json.dumps([
["alpamayo-trajectory", "alpamayo-trajectory", "FilmCamera", 0.0, 1.0, 7.5]
], indent=2) + "\n")
with (args.output / "prediction.csv").open("w") as stream:
writer = csv.writer(stream, lineterminator="\n")
writer.writerow(["time_s", "ego_x_m", "ego_y_m", "ego_z_m", "body_yaw_rad"])
writer.writerows(rows)
metadata = {
"mode": "model-trajectory-replay; no closed-loop driving or dynamics",
"prediction_sha256": hashlib.sha256(content).hexdigest(),
"points": 64,
"time_step_s": .1,
"ego_origin_in_usd_m": [0, -1.8, 0],
"proxy": "Cosmetic car dimensions; transforms replay model positions and yaw-only body rotations",
"first_predicted_xyz_m": xyz[0].tolist(),
"last_predicted_xyz_m": xyz[-1].tolist(),
"scene": "alpamayo-trajectory.usda",
}
(args.output / "prediction-scene.json").write_text(json.dumps(metadata, indent=2) + "\n")
print(json.dumps(metadata), flush=True)
check.py
View files on GitHub"""Check saved model products on a CPU, without importing either model runtime."""
import argparse
import json
import math
from pathlib import Path
import struct
import subprocess
parser = argparse.ArgumentParser()
parser.add_argument("--cosmos", type=Path, help="Folder containing sample_outputs.json and vision.mp4")
parser.add_argument("--alpamayo", type=Path, help="Folder containing output.json and output.safetensors")
parser.add_argument("--ffmpeg", default="ffmpeg")
parser.add_argument("--ffprobe", default="ffprobe")
args = parser.parse_args()
if not args.cosmos and not args.alpamayo:
parser.error("Choose --cosmos, --alpamayo, or both")
checks = {}
if args.cosmos:
receipt = json.loads((args.cosmos / "sample_outputs.json").read_text())
assert receipt["status"] == "success", receipt.get("message")
probe = json.loads(subprocess.check_output([
args.ffprobe, "-v", "error", "-select_streams", "v:0", "-count_frames",
"-show_entries", "stream=width,height,r_frame_rate,nb_read_frames", "-of", "json",
str(args.cosmos / "vision.mp4"),
], text=True))["streams"][0]
assert int(probe["nb_read_frames"]) == receipt["args"]["num_frames"], probe
numerator, denominator = map(int, probe["r_frame_rate"].split("/"))
assert abs(numerator / denominator - receipt["args"]["fps"]) < 1e-6, probe
subprocess.run([args.ffmpeg, "-v", "error", "-xerror", "-i", str(args.cosmos / "vision.mp4"),
"-f", "null", "-"], check=True)
checks["cosmos"] = {"decoded_video": probe,
"scope": "generation completed; prompt adherence still requires visual review"}
if args.alpamayo:
record = json.loads((args.alpamayo / "output.json").read_text())
expected_shapes = {"pred_xyz": [1, 1, 1, 64, 3], "pred_rot": [1, 1, 1, 64, 3, 3]}
content = (args.alpamayo / "output.safetensors").read_bytes()
header_length = struct.unpack("<Q", content[:8])[0]
header = json.loads(content[8:8 + header_length])
numeric = content[8 + header_length:]
for name, shape in expected_shapes.items():
tensor = header[name]
assert tensor["shape"] == shape and tensor["dtype"] == "F32", (name, tensor)
begin, end = tensor["data_offsets"]
values = [entry[0] for entry in struct.iter_unpack("<f", numeric[begin:end])]
assert len(values) == math.prod(shape) and all(map(math.isfinite, values)), name
assert record["finite"] is True
assert record["pred_xyz_shape"] == expected_shapes["pred_xyz"]
assert record["pred_rot_shape"] == expected_shapes["pred_rot"]
checks["alpamayo"] = {"finite_saved_trajectories": True, "shapes": expected_shapes,
"scope": "model interface smoke on synthetic observations; not driving evaluation"}
print(json.dumps(checks, indent=2))
