Try Cosmos video generation and an Alpamayo driving-model interface

A real Cosmos camera-conditioned generation, then an Alpamayo prediction replay in Isaac Sim. Separate model experiments on our own scenes.

  • Cosmos3-Edge
  • Alpamayo2-Edge
  • Isaac Sim
  • OpenUSD
World generation and driving-model outputs
Recorded runExample run and recording

Create a short video from an Isaac Sim image, then run a separate driving model on six camera views from a self-authored intersection. You can inspect the saved model products without a GPU. These are two independent experiments: the Cosmos video is not fed into Alpamayo.

How it works

How the data moves
  1. InputTwo independent inputsCosmos: RGB + camera actions; Alpamayo: 22 RGB frames + ego history

    Both examples use scenes we authored. They test different model interfaces.

  2. ProcessRun released model weightsCosmos3-Edge · Alpamayo2-Edge

    Cosmos generates pixels. Alpamayo proposes future vehicle positions and rotations.

  3. OutputInspect each result61-frame MP4 · 64 future poses

    Check decoded frames and finite tensors. The car shown in Isaac Sim replays the predicted trajectory.

An image is a condition, not a simulated physical state

Cosmos3-Edge receives one RGB image and a description of what should happen. It denoises a latent video and decodes RGB frames. Our input contains a yellow cube, a green cube and a dark block on a workbench.

Text
Isaac Sim camera → one RGB image + motion prompt → Cosmos3-Edge → generated RGB video
                   same image and seed, change only "left" / "right"

The requested output has 49 frames at 24 FPS, about 2.04 seconds. The model does not return a USD scene, contact forces or verified collision events. A decoded video confirms that inference finished; visual review determines whether the requested motion happened. Official inference contract

Give the world model explicit camera actions

The action-conditioned example adds a float32 [60, 9] sequence to the same image. Each row carries three native normalized translation channels and a 6D rotation representation. These values are already prepared for the model; they are not raw displacements in meters and must not be normalized again. We preserve NVIDIA's published camera example unchanged, verify its SHA-256 (547b977c…83c5595), and let Framework pad the nine channels to its model action width. Pinned official camera example

Text
Our RGB image + NVIDIA's 60×9 camera-action sequence → forward_dynamics → 61 RGB frames
                                                                        30 FPS

The first frame is the observation; the following 60 are predicted views. The recorded output is 832×464, after the model's padding/cropping, at the requested 480 resolution tier. It changes the camera view visibly. It still provides generated RGB, not a reconstructed 3D scene or a physics-based guarantee that the requested camera path is exact.

Camera history becomes a proposed trajectory

Alpamayo2-Edge combines camera observations, recent ego motion and navigation text. Its vision-language backbone generates driving reasoning, and its action expert predicts motion. The upstream decoder integrates its predicted acceleration and curvature into future poses. Our intersection uses original primitive geometry and no gated driving recordings. Official model card

Text
6 cameras / 22 RGB frames ─┐
20 historical ego poses ───┼→ Alpamayo2-Edge → reasoning text + 64 future poses
"Continue straight..." ───┘                  → inspect the saved tensors
Data in this lab Shape and meaning
Camera images CPU uint8 [22, 1, 3, 360, 640]; 22 frames, one frame per entry, RGB, height, width.
Camera IDs and timestamps int64 [22]; IDs 0/1/2/3/5/6 have 3/5/3/3/3/5 frames. The two front cameras cover −2, −1, −0.5, −0.1 and 0 seconds; other views cover −1, −0.1 and 0 seconds.
Position history float32 [1, 20, 3]; ego-frame XYZ at 0.1-second steps, accompanied by [1, 20, 3, 3] rotation matrices. Our synthetic camera rig moves at 5 m/s.
Predicted positions [1, 1, 1, 64, 3]; one batch, one trajectory set, one candidate, 64 XYZ poses from 0.1 to 6.4 seconds. Rotation output is [1, 1, 1, 64, 3, 3].

For example, a predicted position [5.0, 0.4, 0.0] means 5 m forward and 0.4 m left of the reference ego pose. This example explains the coordinate convention; it is not a measured prediction. Our rendered road is a synthetic interface test, outside the model's real driving-data evaluation. It checks actual weight loading and valid outputs. It does not test a car in closed loop.

First time? Get the lab files

Open a terminal on the Isaac Sim machine for GPU labs, or on your own computer for CPU-only steps. Press Enter after each line. You can copy a multi-line command block as a whole.

Shell
cd "$HOME"
git clone https://github.com/james-joobs/awesome-nvidia-physical-ai.git
cd awesome-nvidia-physical-ai
pwd

This example clones into your home folder. If you already cloned the repo, skip git clone and enter your project folder. In each new terminal, check that pwd ends in awesome-nvidia-physical-ai so commands can find the labs/ files.

Run

1. Prepare a dedicated model folder on a Linux GPU machine

Open a terminal in this repository on the GPU machine. Check GPU and driver setup, then choose an empty folder outside the repository for model weights and virtual environments. The recorded machine has an RTX 5090 with 32 GiB. The commands pin the source and model revisions in model-pins.json.

Shell
export WORLD_DRIVING_ROOT="$HOME/physical-ai-experiments/world-driving"
bash labs/world-driving/setup.sh

Setup creates separate Cosmos and Alpamayo environments. Cosmos uses the locked CUDA 13.0 inference dependencies; Alpamayo uses its Python 3.12 lockfile. The public weights total about 13.5 GiB; the recorded environments add about 16 GiB, with caches and a separately downloaded VAE requiring more space. Allow at least 60 GiB for this subset. Read the exact checkpoint terms before using the weights. Cosmos setup · Alpamayo setup

2. Generate two videos from one image

Run each line in the same terminal. Both runs use seed 19 and the same bundled, self-authored simulator image. Only the requested slide direction changes. Each script waits for the session's shared GPU lock, so it can run alongside other queued labs without competing for GPU memory.

Shell
bash labs/world-driving/run_cosmos.sh right
bash labs/world-driving/run_cosmos.sh left

Open outputs/cosmos-right/workbench-right/vision.mp4 under WORLD_DRIVING_ROOT. Keep both clips, even if one does not follow its prompt. The script turns off the optional guardrail models for this controlled synthetic input; deployed applications need their own guardrail configuration.

To add the explicit camera actions, run this third command. It downloads only the pinned 12 KB action sidecar, checks its digest and shape, then calls the same released Edge checkpoint.

Shell
bash labs/world-driving/run_camera_action.sh

3. Render your own driving-model inputs in Isaac Sim

Set ISAAC_LAB_PYTHON to a Python executable with Isaac Lab and Isaac Sim installed. The recorded input-rendering environment uses Isaac Sim 6.1.0.0 / Isaac Lab 17.0.2 and AppLauncher --viz kit. It is separate from the site's Isaac Sim 6.2 workbench guide. Follow the simulator's license prompt on first use.

Shell
export ISAAC_LAB_PYTHON="/path/to/IsaacLab/.venv/bin/python"
flock /tmp/physical-ai-four-domains.gpu.lock \
  "$ISAAC_LAB_PYTHON" labs/world-driving/make_driving_scene.py \
  --viz kit --output "$WORLD_DRIVING_ROOT/inputs/synthetic-intersection"

The GUI opens the road scene and renders 22 images from the camera positions and times in the table. The folder also receives input.json and synthetic-intersection.usda. To record the actual simulator window on Linux X11, add --record-window /absolute/path/to/ffmpeg. Use a terminal in your graphical desktop session so DISPLAY and XAUTHORITY refer to that session.

4. Run the released driving-model weights

This reads your generated camera images and history, loads the pinned BF16 checkpoint and writes both reasoning and trajectories. It uses the default PyTorch SDPA attention backend, without accelerated-inference compilation.

Shell
python3 labs/cli.py --json run alpamayo-synthetic \
  --python "$WORLD_DRIVING_ROOT/alpamayo2-edge/.venv/bin/python" \
  --input "$WORLD_DRIVING_ROOT/inputs/synthetic-intersection" \
  --model "$WORLD_DRIVING_ROOT/checkpoints/Alpamayo2-Edge" \
  --seed 19 --output "$WORLD_DRIVING_ROOT/outputs/alpamayo-cli"

Open outputs/alpamayo-cli/output.json for reasoning and output.safetensors for numeric predictions. The receipt records GPU, package versions, memory and exact input/output shapes. The runner verifies both weight shards against their published SHA-256 values before loading. In the recorded run the adaptive reasoning output is <NOMINAL>, rather than a longer explanation. To perform an actual closed-loop driving evaluation, use the separate AlpaSim guide and obtain the required scene access.

Check the result

Inspect saved products on a CPU

You only need Python and FFmpeg/FFprobe for the checker. Point it at a generation folder or an Alpamayo output folder. It fully decodes the video, checks requested frame count and FPS, and independently reads the saved trajectory tensors to reject nonfinite coordinates or wrong shapes.

Shell
python3 labs/world-driving/check.py \
  --cosmos labs/world-driving/evidence/cosmos-camera-actions \
  --alpamayo labs/world-driving/evidence/alpamayo-synthetic

Those paths use the bundled public outputs, so the command works without installing the model runtimes. Replace them with your own output folders after a GPU run. The recorded CLI prediction took 13.37 seconds including input processing, with 5.76 GiB peak PyTorch allocation; this is one synthetic sample on one RTX 5090, not a general performance benchmark.

The saved evidence/prediction-scene/alpamayo-trajectory.usda draws the 64 predicted positions on our road and animates a cosmetic vehicle proxy. Its transforms come from the saved model tensors. The Isaac Sim recording of this scene is trajectory replay, with no new driver decisions or vehicle dynamics. Replay generator · Recorded checks

Compare the right things

  • Cosmos: compare the initial and last frames, cube direction, camera motion and object identity. Report a prompt-following failure even when sample_outputs.json says success.
  • Alpamayo: check that all weights loaded, the saved tensors are finite, and positions use the documented ego frame. A plausible explanation and a curved path do not establish safe driving.
  • Access: the November 19 audit confirmed NuRec scene access on the recording machine, while the separate PhysicalAI-AV source dataset returned HTTP 403. The public synthetic test does not need either dataset. Their access approvals and terms are independent; dataset-derived video and metrics remain local.

Source and version pins · Cosmos · Alpamayo · AlpaSim closed loop

Example run and recording

Source code

setup.shView files on GitHub
Shell
#!/usr/bin/env bash
set -euo pipefail

task_root="${WORLD_DRIVING_ROOT:?Choose an isolated folder with at least 60 GiB free for weights, runtimes and caches}"
uv_bin="${UV_BIN:-$HOME/.local/bin/uv}"
mkdir -p "$task_root"
cd "$task_root"
clone_pin() {
  local directory="$1" url="$2" revision="$3"
  if [[ ! -d "$directory/.git" ]]; then
    git clone "$url" "$directory"
  fi
  if [[ -n "$(git -C "$directory" status --porcelain)" ]]; then
    printf 'Stop: %s has local changes. Choose an empty task folder.\n' "$directory" >&2
    return 1
  fi
  git -C "$directory" fetch origin "$revision"
  git -C "$directory" checkout --detach "$revision"
}
clone_pin cosmos-framework https://github.com/NVIDIA/cosmos-framework.git afd7d4c4087a8929c84523ff60172642fd11690b
clone_pin alpamayo2-edge https://github.com/NVlabs/alpamayo2-edge.git 02f55a7aeabc1a2b9f7a80445c08e40bddc89f5e
(
  cd cosmos-framework
  "$uv_bin" tool run --from uv==0.11.29 uv sync --locked --group cu130 --extra guardrail --no-default-groups --python 3.13
)
(
  cd alpamayo2-edge
  "$uv_bin" sync --locked --extra dataset --no-default-groups
)
cosmos-framework/.venv/bin/hf download nvidia/Cosmos3-Edge \
  --revision 6afc1256321c3a8e623d89be3bc5117a1e90f96b \
  --local-dir checkpoints/Cosmos3-Edge
alpamayo2-edge/.venv/bin/hf download nvidia/Alpamayo2-Edge \
  --revision b92b6aa326bd3e2235db28f248ad5a30b5890e64 \
  --local-dir checkpoints/Alpamayo2-Edge
model-pins.jsonView files on GitHub
JSON
{
  "checked_at": "2026-11-19",
  "camera_action_sidecar": {
    "url": "https://media.githubusercontent.com/media/nvidia-cosmos/cosmos-dependencies/2b17a2413bd86b2cf9b03823637108851e4ddf2d/inputs/action/camera_action_44.json",
    "sha256": "547b977ccd6d435d579ba83e82e58618d0bebf2e8e7d8d577bf2e9c8883c5595",
    "shape": [60, 9],
    "semantics": "Published native normalized camera-pose sequence: three translation channels and six rotation channels. Preserve unchanged; do not assume translation is raw meters."
  },
  "cosmos": {
    "source": "https://github.com/NVIDIA/cosmos-framework",
    "source_revision": "afd7d4c4087a8929c84523ff60172642fd11690b",
    "model": "nvidia/Cosmos3-Edge",
    "model_revision": "6afc1256321c3a8e623d89be3bc5117a1e90f96b",
    "gated": false,
    "weights_gib": 8.5,
    "terms": "OpenMDW-1.1; the Hugging Face metadata license field is other, with a linked license",
    "weights_license_url": "https://openmdw.ai/license/1-1/",
    "code_license_url": "https://github.com/NVIDIA/cosmos-framework/blob/afd7d4c4087a8929c84523ff60172642fd11690b/LICENSE",
    "requirements": "Dedicated Python 3.13, locked cu130 inference group. Driver must support CUDA 13.0."
  },
  "alpamayo": {
    "source": "https://github.com/NVlabs/alpamayo2-edge",
    "source_revision": "02f55a7aeabc1a2b9f7a80445c08e40bddc89f5e",
    "model": "nvidia/Alpamayo2-Edge",
    "model_revision": "b92b6aa326bd3e2235db28f248ad5a30b5890e64",
    "gated": false,
    "weights_gib": 4.94,
    "code_license": "Apache-2.0",
    "weights_license": "OpenMDW-1.1",
    "weights_license_url": "https://huggingface.co/nvidia/Alpamayo2-Edge/blob/b92b6aa326bd3e2235db28f248ad5a30b5890e64/LICENSE",
    "code_license_url": "https://github.com/NVlabs/alpamayo2-edge/blob/02f55a7aeabc1a2b9f7a80445c08e40bddc89f5e/LICENSE",
    "weight_sha256": {
      "00000.safetensors": "d13c1a427218baf7b7d365098c0f5496c0e5e6a6bf662836b8eebbe7eb7cfe7f",
      "00001.safetensors": "cfe424a7320ce729cd927b5ceaab4323a62443b50f461a2f5c0dd0b10634e0b5"
    },
    "requirements": "Python 3.12, locked torch 2.8.0 and Transformers 5.17.0; upstream recommends >=16 GiB for BF16 example, validated there on H100."
  },
  "large_models_deferred": {
    "nvidia/Cosmos3-Nano-Sim-Bimanual": {
      "model_revision": "da4d2c09175e2a6dd5b237265318066d5134312e",
      "weights_gib": 29.57,
      "reason": "Weights alone almost fill a 32 GiB GPU; runtime memory was not validated."
    },
    "nvidia/Cosmos3-Nano-Sim-Transfer": {
      "model_revision": "45510d5b8cefa338b7abc01eb702cf1e9cdf6742",
      "weights_gib": 29.54,
      "reason": "Weights alone almost fill a 32 GiB GPU; runtime memory was not validated."
    }
  }
}
make_cosmos_input.pyView files on GitHub
Python
"""Prepare a short, explicitly generative Cosmos test from our own simulator image."""

import argparse
import hashlib
import json
from pathlib import Path

parser = argparse.ArgumentParser()
parser.add_argument("--image", type=Path, required=True)
parser.add_argument("--output", type=Path, required=True)
parser.add_argument("--direction", choices=["left", "right"], default="right")
args = parser.parse_args()
args.output.parent.mkdir(parents=True, exist_ok=True)
manifest = {
    "model_mode": "image2video",
    "prompt": (
        "A fixed camera watches a wooden workbench in a clean robotics simulation. "
        "A yellow cube is in the center, a mint green cube at the rear left and a tall dark block on the right. "
        f"The yellow cube slides slowly to the {args.direction} across the table. "
        "The green cube and the dark block remain stationary. "
        "Keep the camera fixed, preserve the tabletop, lighting, object colors and sizes. "
        "Continuous motion, no cuts, no added objects."
    ),
    "vision_path": str(args.image.resolve()),
    "format_prompt_as_json": False,
    "prompt_upsampling": False,
    "num_frames": 49,
    "fps": 24,
    "resolution": "480",
    "num_steps": 16,
}
args.output.write_text(json.dumps(manifest, indent=2) + "\n")
receipt = {
    "kind": "image-conditioned video generation; not a physics simulation or robot policy",
    "input_sha256": hashlib.sha256(args.image.read_bytes()).hexdigest(),
    "input_origin": "Our self-authored Isaac Sim Replicator workbench scene, frame 05",
    "manifest": manifest,
}
args.output.with_suffix(".provenance.json").write_text(json.dumps(receipt, indent=2) + "\n")
make_camera_action_input.pyView files on GitHub
Python
"""Pair our simulator image with NVIDIA's pinned native camera-action example.

Use the published normalized sequence unchanged: inventing metric units or
normalizing this sequence a second time would change the input contract.
"""

import argparse
import hashlib
import json
import math
from pathlib import Path
import subprocess

parser = argparse.ArgumentParser()
parser.add_argument("--image", type=Path, required=True)
parser.add_argument("--output", type=Path, required=True)
args = parser.parse_args()
args.output.parent.mkdir(parents=True, exist_ok=True)
revision = "2b17a2413bd86b2cf9b03823637108851e4ddf2d"
url = f"https://media.githubusercontent.com/media/nvidia-cosmos/cosmos-dependencies/{revision}/inputs/action/camera_action_44.json"
payload = subprocess.check_output(["curl", "-fsSL", "--max-time", "60", url])
digest = hashlib.sha256(payload).hexdigest()
assert digest == "547b977ccd6d435d579ba83e82e58618d0bebf2e8e7d8d577bf2e9c8883c5595", digest
actions = json.loads(payload)
assert len(actions) == 60 and all(len(row) == 9 for row in actions)
assert all(math.isfinite(value) for row in actions for value in row)
action_path = args.output.with_name("camera-action-60x9.json")
action_path.write_bytes(payload)
spec = {
    "name": "workbench-camera-actions",
    "action_chunk_size": 60,
    "action_path": str(action_path.resolve()),
    "domain_name": "camera_pose",
    "fps": 30,
    "view_point": "ego_view",
    "guidance": 1.0,
    "image_size": 480,
    "resolution": "480",
    "model_mode": "forward_dynamics",
    "num_frames": 61,
    "num_steps": 30,
    "prompt": "A wooden workbench holds a yellow cube, a mint green cube and a tall dark block. The camera moves around the stationary objects.",
    "shift": 5.0,
    "vision_path": str(args.image.resolve()),
}
args.output.write_text(json.dumps(spec, indent=2) + "\n")
args.output.with_suffix(".provenance.json").write_text(json.dumps({
    "input_origin": "Our self-authored Isaac Sim Replicator workbench scene, frame 05",
    "action_origin": url,
    "action_sha256": digest,
    "action_shape": [60, 9],
    "action_columns": "native normalized camera-pose channels: 3 translation + 6D rotation; not raw metric displacements",
    "normalization": "Preserve NVIDIA's published values unchanged. Framework pads the native channels to its model action width.",
    "output": "Initial observation plus 60 predicted RGB frames; not a physics solver or a new 3D reconstruction",
}, indent=2) + "\n")
make_driving_scene.pyView files on GitHub
Python
"""Render our own six-camera, 22-frame input in Isaac Sim; no driving dataset needed."""

import argparse
import hashlib
import json
from importlib.metadata import version
import math
import os
from pathlib import Path
import signal
import subprocess

from isaaclab.app import AppLauncher

parser = argparse.ArgumentParser()
parser.add_argument("--output", type=Path, required=True)
parser.add_argument("--record-window", type=Path, help="FFmpeg executable for native Isaac Sim GUI recording")
AppLauncher.add_app_launcher_args(parser)
args = parser.parse_args()
args.enable_cameras = True
launcher = AppLauncher(args)
app = launcher.app

import numpy as np
from PIL import Image
import omni.replicator.core as rep
import omni.usd
from omni.kit.viewport.utility import get_active_viewport
from pxr import Gf, Sdf, UsdGeom, UsdLux, UsdShade

args.output.mkdir(parents=True, exist_ok=True)
omni.usd.get_context().new_stage()
stage = omni.usd.get_context().get_stage()
UsdGeom.SetStageUpAxis(stage, UsdGeom.Tokens.z)
UsdGeom.SetStageMetersPerUnit(stage, 1.0)
world = UsdGeom.Xform.Define(stage, "/World")
stage.SetDefaultPrim(world.GetPrim())


def material(name, rgb):
    mat = UsdShade.Material.Define(stage, "/World/Looks/" + name)
    shader = UsdShade.Shader.Define(stage, mat.GetPath().AppendChild("Shader"))
    shader.CreateIdAttr("UsdPreviewSurface")
    shader.CreateInput("diffuseColor", Sdf.ValueTypeNames.Color3f).Set(Gf.Vec3f(*rgb))
    shader.CreateInput("roughness", Sdf.ValueTypeNames.Float).Set(.7)
    mat.CreateSurfaceOutput().ConnectToSource(shader.ConnectableAPI(), "surface")
    return mat


def block(name, center, size, mat):
    shape = UsdGeom.Cube.Define(stage, "/World/" + name)
    shape.CreateSizeAttr(1)
    shape.AddTranslateOp().Set(Gf.Vec3d(*center))
    shape.AddScaleOp().Set(Gf.Vec3d(*size))
    UsdShade.MaterialBindingAPI.Apply(shape.GetPrim()).Bind(mat)
    return shape


asphalt = material("Asphalt", (.12, .14, .16))
grass = material("Grass", (.13, .24, .13))
white = material("White", (.9, .91, .88))
yellow = material("Yellow", (.96, .7, .05))
red = material("RedCar", (.62, .045, .035))
blue = material("BlueCar", (.02, .18, .4))
glass = material("Glass", (.05, .12, .16))
concrete = material("Concrete", (.42, .45, .43))
block("Ground", (20, 0, -.12), (160, 160, .2), grass)
block("MainRoad", (20, 0, .005), (150, 8, .05), asphalt)
block("CrossRoad", (28, 0, .02), (8, 100, .05), asphalt)
for index, x in enumerate(range(-45, 96, 6)):
    if 22 < x < 34:
        continue
    block(f"CenterLine{index}", (x, 0, .04), (3, .12, .015), yellow)
for index, y in enumerate((-3.65, 3.65)):
    for side, x in enumerate((-12, 65)):
        block(f"Edge{index}_{side}", (x, y, .04), (68, .1, .015), white)
for index, y in enumerate(np.arange(-3.1, 3.2, .75)):
    block(f"Crosswalk{index}", (22.8, float(y), .06), (.8, .42, .015), white)
for index, (x, y, h) in enumerate(((42, -15, 12), (56, -14, 8), (40, 17, 10), (-18, 16, 6), (-25, -14, 7))):
    block(f"Building{index}", (x, y, h / 2), (9, 9, h), concrete)
for index, (x, y, body) in enumerate(((17, 2.05, blue), (28, -12, red))):
    block(f"Car{index}Body", (x, y, .65), (4.2, 1.8, .9), body)
    block(f"Car{index}Cabin", (x - .2, y, 1.2), (2, 1.6, .7), glass)
light = UsdLux.DomeLight.Define(stage, "/World/Sky")
light.CreateIntensityAttr(1400)
sun = UsdLux.DistantLight.Define(stage, "/World/Sun")
sun.CreateIntensityAttr(2600)
sun.AddRotateXYZOp().Set(Gf.Vec3f(315, 15, 20))

width, height = 640, 360
camera = UsdGeom.Camera.Define(stage, "/World/Camera")
camera.CreateHorizontalApertureAttr(20.955)
camera.CreateClippingRangeAttr(Gf.Vec2f(.1, 250))
camera_op = camera.AddTransformOp()
product = rep.create.render_product(str(camera.GetPath()), (width, height))
annotator = rep.AnnotatorRegistry.get_annotator("rgb")
annotator.attach([product])
get_active_viewport().camera_path = str(camera.GetPath())
profiles = {0: (55, (-1, -.1, 0), 12), 1: (0, (-2, -1, -.5, -.1, 0), 14),
            2: (-55, (-1, -.1, 0), 12), 3: (145, (-1, -.1, 0), 12),
            5: (-145, (-1, -.1, 0), 12), 6: (0, (-2, -1, -.5, -.1, 0), 30)}
shots = []
recorder = None
record_log = None
if args.record_window:
    camera_op.Set(Gf.Matrix4d(1).SetLookAt(Gf.Vec3d(0, -1.8, 1.6), Gf.Vec3d(30, -1.8, 1.4), Gf.Vec3d(0, 0, 1)).GetInverse())
    for _ in range(35):
        app.update()
    windows = subprocess.check_output(["xwininfo", "-root", "-tree"], text=True)
    rows = [row for row in windows.splitlines() if '"Isaac Lab ' in row and "mutter-x11-frames" not in row]
    if not rows:
        raise RuntimeError("Native Isaac Sim window was not found; use --viz kit")
    identifier = str(int(rows[0].split()[0], 16))
    record_log = (args.output / "record-window.log").open("w")
    recorder = subprocess.Popen([
        str(args.record_window), "-nostdin", "-hide_banner", "-loglevel", "warning", "-y",
        "-f", "x11grab", "-draw_mouse", "0", "-framerate", "24", "-window_id", identifier,
        "-i", f"{os.environ['DISPLAY']}.0", "-an", "-c:v", "libx264", "-preset", "fast", "-crf", "22",
        "-pix_fmt", "yuv420p", "-movflags", "+faststart", str(args.output / "six-camera-isaac-gui.mp4"),
    ], stdout=record_log, stderr=subprocess.STDOUT)
for camera_id, (degrees, times, focal) in profiles.items():
    for offset in times:
        eye = Gf.Vec3d(offset * 5, -1.8, 1.6)
        yaw = math.radians(degrees)
        target = eye + Gf.Vec3d(math.cos(yaw) * 30, math.sin(yaw) * 30, -.2)
        camera_op.Set(Gf.Matrix4d(1).SetLookAt(eye, target, Gf.Vec3d(0, 0, 1)).GetInverse())
        camera.GetFocalLengthAttr().Set(focal)
        for _ in range(14):
            app.update()
        rgb = np.asarray(annotator.get_data())[:, :, :3].copy()
        assert rgb.shape == (height, width, 3) and rgb.mean() > 3, (camera_id, offset, rgb.shape)
        name = f"camera-{camera_id}-t{round((offset + 2) * 1000000):07d}.png"
        Image.fromarray(rgb).save(args.output / name)
        shots.append({"camera_id": camera_id, "timestamp_us": round((offset + 2) * 1000000),
                      "relative_time_s": offset, "image": name,
                      "sha256": hashlib.sha256((args.output / name).read_bytes()).hexdigest()})
        print(f"RENDER camera={camera_id} t={offset:+.1f}s file={name}", flush=True)
stage.GetRootLayer().Export(str(args.output / "synthetic-intersection.usda"))
(args.output / "input.json").write_text(json.dumps({
    "kind": "self-authored synthetic interface smoke; not a driving benchmark",
    "isaac_sim_version": version("isaacsim"),
    "isaac_lab_version": version("isaaclab"),
    "coordinate_frame": "ego x forward, y left, z up; ego reference at query time",
    "width": width, "height": height, "ego_speed_m_s": 5, "shots": shots,
    "history_times_s": [round(-1.9 + .1 * i, 3) for i in range(20)],
    "history_xyz_m": [[round((-1.9 + .1 * i) * 5, 3), 0, 0] for i in range(20)],
}, indent=2) + "\n")
annotator.detach([product])
if recorder:
    recorder.send_signal(signal.SIGINT)
    recorder.wait(timeout=20)
    record_log.close()
app.close()
run_cosmos.shView files on GitHub
Shell
#!/usr/bin/env bash
set -euo pipefail

task_root="${WORLD_DRIVING_ROOT:?Set WORLD_DRIVING_ROOT to the isolated GPU-machine folder}"
lab_root="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
direction="${1:-right}"
input_image="${2:-$lab_root/input/workbench.png}"
mkdir -p "$task_root/inputs" "$task_root/logs"
python3 "$lab_root/make_cosmos_input.py" --image "$input_image" \
  --output "$task_root/inputs/workbench-$direction.json" --direction "$direction"
export COSMOS_TRAINING=0
export PATH="$task_root/cosmos-framework/.venv/bin:$PATH"
cd "$task_root/cosmos-framework"
# Use the same lock for every other GPU job in this recording session.
flock /tmp/physical-ai-four-domains.gpu.lock \
  .venv/bin/python -m cosmos_framework.scripts.inference \
  -i "$task_root/inputs/workbench-$direction.json" \
  -o "$task_root/outputs/cosmos-$direction" \
  --checkpoint-path "$task_root/checkpoints/Cosmos3-Edge" \
  --parallelism-preset=latency --seed=19 \
  --no-use-torch-compile --no-guardrails
run_camera_action.shView files on GitHub
Shell
#!/usr/bin/env bash
set -euo pipefail
task_root="${WORLD_DRIVING_ROOT:?Set WORLD_DRIVING_ROOT to the isolated GPU-machine folder}"
lab_root="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
python3 "$lab_root/make_camera_action_input.py" --image "$lab_root/input/workbench.png" \
  --output "$task_root/inputs/workbench-camera-actions.json"
export COSMOS_TRAINING=0
export PATH="$task_root/cosmos-framework/.venv/bin:$PATH"
cd "$task_root/cosmos-framework"
flock /tmp/physical-ai-four-domains.gpu.lock \
  .venv/bin/python -m cosmos_framework.scripts.inference \
  -i "$task_root/inputs/workbench-camera-actions.json" \
  -o "$task_root/outputs/cosmos-camera-actions" \
  --checkpoint-path "$task_root/checkpoints/Cosmos3-Edge" \
  --parallelism-preset=latency --seed=19 \
  --no-use-torch-compile --no-guardrails
run_alpamayo_synthetic.pyView files on GitHub
Python
"""Use released Alpamayo2-Edge weights on self-authored Isaac Sim camera frames.

This checks the model interface and output shapes, not driving quality or safety.
"""

import argparse
import hashlib
import json
from importlib.metadata import version
from pathlib import Path
import subprocess
import time

import numpy as np
from PIL import Image
import torch

from alpamayo2_edge.checkpoint import load_model
from alpamayo2_edge.inference import run_data

parser = argparse.ArgumentParser()
parser.add_argument("--input", type=Path, required=True)
parser.add_argument("--model", type=Path, required=True)
parser.add_argument("--output", type=Path, required=True)
parser.add_argument("--nav-text", default="Continue straight through the intersection.")
parser.add_argument("--seed", type=int, default=43)
args = parser.parse_args()
pins = json.loads(Path(__file__).with_name("model-pins.json").read_text())["alpamayo"]
weight_digests = {}
for filename, expected in pins["weight_sha256"].items():
    digest = hashlib.sha256()
    with (args.model / filename).open("rb") as stream:
        for block in iter(lambda: stream.read(1024 * 1024), b""):
            digest.update(block)
    weight_digests[filename] = digest.hexdigest()
    if digest.hexdigest() != expected:
        raise ValueError(f"Checkpoint shard {filename} differs from the documented model revision")
manifest = json.loads((args.input / "input.json").read_text())
images = np.stack([np.asarray(Image.open(args.input / row["image"]).convert("RGB"))
                   for row in manifest["shots"]])
data = {
    "image_frames": torch.from_numpy(images).permute(0, 3, 1, 2).unsqueeze(1).contiguous(),
    "camera_indices": torch.tensor([row["camera_id"] for row in manifest["shots"]], dtype=torch.int64),
    "absolute_timestamps": torch.tensor([row["timestamp_us"] for row in manifest["shots"]], dtype=torch.int64),
    "ego_history_xyz": torch.tensor([manifest["history_xyz_m"]], dtype=torch.float32),
    "ego_history_rot": torch.eye(3).reshape(1, 1, 3, 3).repeat(1, 20, 1, 1),
    "nav_text": [args.nav_text],
}
args.output.mkdir(parents=True, exist_ok=True)
began = time.perf_counter()
torch.cuda.reset_peak_memory_stats()
model, loading_info = load_model(str(args.model.resolve()), device="cuda", dtype="bfloat16",
                                attn_implementation="sdpa", local_files_only=True)
settings = dict(seed=args.seed, top_p=.98, top_k=None, temperature=.6, num_traj_sets=1,
                num_traj_samples=1, max_generation_length=192,
                diffusion_kwargs={"inference_step": 10})
result = run_data(model, data, "isaac-synthetic-intersection", args.output, settings, "cuda")
receipt = {
    "scope": "actual Alpamayo2-Edge model inference on a self-authored synthetic interface test; not driving evaluation",
    "upstream_source": "https://github.com/NVlabs/alpamayo2-edge",
    "upstream_revision": "02f55a7aeabc1a2b9f7a80445c08e40bddc89f5e",
    "model_id": "nvidia/Alpamayo2-Edge",
    "model_revision": "b92b6aa326bd3e2235db28f248ad5a30b5890e64",
    "verified_weight_sha256": weight_digests,
    "gpu": torch.cuda.get_device_name(0),
    "peak_allocated_gib": torch.cuda.max_memory_allocated() / 2**30,
    "total_elapsed_s": time.perf_counter() - began,
    "versions": {key: version(key) for key in ("torch", "torchvision", "transformers", "safetensors")},
    "input_shapes": {key: list(value.shape) for key, value in data.items() if isinstance(value, torch.Tensor)},
    "navigation": args.nav_text,
    "seed": args.seed,
    "result": result,
}
(args.output / "receipt.json").write_text(json.dumps(receipt, indent=2) + "\n")
print(json.dumps(receipt), flush=True)
run_alpamayo.shView files on GitHub
Shell
#!/usr/bin/env bash
set -euo pipefail
task_root="${WORLD_DRIVING_ROOT:?Set WORLD_DRIVING_ROOT to the isolated GPU-machine folder}"
lab_root="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
test -f "$task_root/inputs/synthetic-intersection/input.json"
mkdir -p "$task_root/outputs/alpamayo-synthetic"
flock /tmp/physical-ai-four-domains.gpu.lock \
  "$task_root/alpamayo2-edge/.venv/bin/python" "$lab_root/run_alpamayo_synthetic.py" \
  --input "$task_root/inputs/synthetic-intersection" \
  --model "$task_root/checkpoints/Alpamayo2-Edge" \
  --output "$task_root/outputs/alpamayo-synthetic"
make_prediction_scene.pyView files on GitHub
Python
"""Show saved Alpamayo predictions as an animated USD proxy, without a model or GPU.

This is a replay of predicted poses. No vehicle physics or closed-loop policy
execution is added. Requires numpy and the USD Python bindings (usd-core).
"""

import argparse
import csv
import hashlib
import json
import math
from pathlib import Path
import struct

import numpy as np
from pxr import Gf, Sdf, Usd, UsdGeom, UsdShade

parser = argparse.ArgumentParser()
parser.add_argument("--scene", type=Path, required=True)
parser.add_argument("--prediction", type=Path, required=True)
parser.add_argument("--output", type=Path, required=True)
args = parser.parse_args()
args.output.mkdir(parents=True, exist_ok=True)
content = args.prediction.read_bytes()
header_size = struct.unpack("<Q", content[:8])[0]
header = json.loads(content[8:8 + header_size])
numeric = content[8 + header_size:]


def tensor(name):
    spec = header[name]
    assert spec["dtype"] == "F32", (name, spec)
    begin, end = spec["data_offsets"]
    return np.frombuffer(numeric[begin:end], dtype="<f4").reshape(spec["shape"])


xyz = tensor("pred_xyz")[0, 0, 0]
rot = tensor("pred_rot")[0, 0, 0]
assert xyz.shape == (64, 3) and rot.shape == (64, 3, 3)
assert np.isfinite(xyz).all() and np.isfinite(rot).all()
stage = Usd.Stage.Open(str(args.scene))
stage.SetStartTimeCode(0)
stage.SetEndTimeCode(192)
stage.SetTimeCodesPerSecond(30)
stage.SetFramesPerSecond(30)


def material(name, color, emissive=False):
    mat = UsdShade.Material.Define(stage, "/World/Looks/" + name)
    shader = UsdShade.Shader.Define(stage, mat.GetPath().AppendChild("Shader"))
    shader.CreateIdAttr("UsdPreviewSurface")
    shader.CreateInput("diffuseColor", Sdf.ValueTypeNames.Color3f).Set(Gf.Vec3f(*color))
    shader.CreateInput("roughness", Sdf.ValueTypeNames.Float).Set(.55)
    if emissive:
        shader.CreateInput("emissiveColor", Sdf.ValueTypeNames.Color3f).Set(Gf.Vec3f(*color))
    mat.CreateSurfaceOutput().ConnectToSource(shader.ConnectableAPI(), "surface")
    return mat


orange = material("PredictedRoute", (1, .36, .015), True)
lime = material("PredictedCar", (.3, .8, .045))
glass = material("PredictedGlass", (.02, .09, .12))
points = [Gf.Vec3f(0, -1.8, .13)] + [Gf.Vec3f(float(x), float(y - 1.8), float(z + .13)) for x, y, z in xyz]
path = UsdGeom.BasisCurves.Define(stage, "/World/PredictedRoute")
path.CreateTypeAttr("linear")
path.CreateCurveVertexCountsAttr([len(points)])
path.CreatePointsAttr(points)
path.CreateWidthsAttr([.15])
path.SetWidthsInterpolation(UsdGeom.Tokens.constant)
UsdShade.MaterialBindingAPI.Apply(path.GetPrim()).Bind(orange)
proxy = UsdGeom.Xform.Define(stage, "/World/PredictedVehicleProxy")
position = proxy.AddTranslateOp()
orientation = proxy.AddOrientOp()
for name, location, scale, look in (("Body", (.3, 0, .65), (4.2, 1.8, .9), lime),
                                    ("Cabin", (-.1, 0, 1.22), (2.0, 1.6, .65), glass)):
    block = UsdGeom.Cube.Define(stage, proxy.GetPath().AppendChild(name))
    block.CreateSizeAttr(1)
    block.AddTranslateOp().Set(Gf.Vec3d(*location))
    block.AddScaleOp().Set(Gf.Vec3d(*scale))
    UsdShade.MaterialBindingAPI.Apply(block.GetPrim()).Bind(look)
position.Set(Gf.Vec3d(0, -1.8, 0), 0)
orientation.Set(Gf.Quatf(1), 0)
rows = []
for index, (point, rotation) in enumerate(zip(xyz, rot, strict=True)):
    seconds = (index + 1) / 10
    yaw = math.atan2(float(rotation[1, 0]), float(rotation[0, 0]))
    position.Set(Gf.Vec3d(float(point[0]), float(point[1] - 1.8), float(point[2])), seconds * 30)
    orientation.Set(Gf.Quatf(math.cos(yaw / 2), Gf.Vec3f(0, 0, math.sin(yaw / 2))), seconds * 30)
    rows.append([seconds, *map(float, point), yaw])
center_x = float((xyz[:, 0].min() + xyz[:, 0].max()) / 2)
center_y = float((xyz[:, 1].min() + xyz[:, 1].max()) / 2) - 1.8
span = max(float(np.ptp(xyz[:, 0])), float(np.ptp(xyz[:, 1])), 20)
target = Gf.Vec3d(center_x, center_y, .6)
eye = target + Gf.Vec3d(-span * .8, -span * .9, span)
camera = UsdGeom.Camera.Define(stage, "/World/FilmCamera")
camera.CreateFocalLengthAttr(24)
camera.CreateHorizontalApertureAttr(20.955)
camera.CreateClippingRangeAttr(Gf.Vec2f(.1, 300))
camera.AddTransformOp().Set(Gf.Matrix4d(1).SetLookAt(eye, target, Gf.Vec3d(0, 0, 1)).GetInverse())
stage.Export(str(args.output / "alpamayo-trajectory.usda"))
(args.output / "takes.json").write_text(json.dumps([
    ["alpamayo-trajectory", "alpamayo-trajectory", "FilmCamera", 0.0, 1.0, 7.5]
], indent=2) + "\n")
with (args.output / "prediction.csv").open("w") as stream:
    writer = csv.writer(stream, lineterminator="\n")
    writer.writerow(["time_s", "ego_x_m", "ego_y_m", "ego_z_m", "body_yaw_rad"])
    writer.writerows(rows)
metadata = {
    "mode": "model-trajectory-replay; no closed-loop driving or dynamics",
    "prediction_sha256": hashlib.sha256(content).hexdigest(),
    "points": 64,
    "time_step_s": .1,
    "ego_origin_in_usd_m": [0, -1.8, 0],
    "proxy": "Cosmetic car dimensions; transforms replay model positions and yaw-only body rotations",
    "first_predicted_xyz_m": xyz[0].tolist(),
    "last_predicted_xyz_m": xyz[-1].tolist(),
    "scene": "alpamayo-trajectory.usda",
}
(args.output / "prediction-scene.json").write_text(json.dumps(metadata, indent=2) + "\n")
print(json.dumps(metadata), flush=True)
check.pyView files on GitHub
Python
"""Check saved model products on a CPU, without importing either model runtime."""

import argparse
import json
import math
from pathlib import Path
import struct
import subprocess

parser = argparse.ArgumentParser()
parser.add_argument("--cosmos", type=Path, help="Folder containing sample_outputs.json and vision.mp4")
parser.add_argument("--alpamayo", type=Path, help="Folder containing output.json and output.safetensors")
parser.add_argument("--ffmpeg", default="ffmpeg")
parser.add_argument("--ffprobe", default="ffprobe")
args = parser.parse_args()
if not args.cosmos and not args.alpamayo:
    parser.error("Choose --cosmos, --alpamayo, or both")
checks = {}
if args.cosmos:
    receipt = json.loads((args.cosmos / "sample_outputs.json").read_text())
    assert receipt["status"] == "success", receipt.get("message")
    probe = json.loads(subprocess.check_output([
        args.ffprobe, "-v", "error", "-select_streams", "v:0", "-count_frames",
        "-show_entries", "stream=width,height,r_frame_rate,nb_read_frames", "-of", "json",
        str(args.cosmos / "vision.mp4"),
    ], text=True))["streams"][0]
    assert int(probe["nb_read_frames"]) == receipt["args"]["num_frames"], probe
    numerator, denominator = map(int, probe["r_frame_rate"].split("/"))
    assert abs(numerator / denominator - receipt["args"]["fps"]) < 1e-6, probe
    subprocess.run([args.ffmpeg, "-v", "error", "-xerror", "-i", str(args.cosmos / "vision.mp4"),
                    "-f", "null", "-"], check=True)
    checks["cosmos"] = {"decoded_video": probe,
                        "scope": "generation completed; prompt adherence still requires visual review"}
if args.alpamayo:
    record = json.loads((args.alpamayo / "output.json").read_text())
    expected_shapes = {"pred_xyz": [1, 1, 1, 64, 3], "pred_rot": [1, 1, 1, 64, 3, 3]}
    content = (args.alpamayo / "output.safetensors").read_bytes()
    header_length = struct.unpack("<Q", content[:8])[0]
    header = json.loads(content[8:8 + header_length])
    numeric = content[8 + header_length:]
    for name, shape in expected_shapes.items():
        tensor = header[name]
        assert tensor["shape"] == shape and tensor["dtype"] == "F32", (name, tensor)
        begin, end = tensor["data_offsets"]
        values = [entry[0] for entry in struct.iter_unpack("<f", numeric[begin:end])]
        assert len(values) == math.prod(shape) and all(map(math.isfinite, values)), name
    assert record["finite"] is True
    assert record["pred_xyz_shape"] == expected_shapes["pred_xyz"]
    assert record["pred_rot_shape"] == expected_shapes["pred_rot"]
    checks["alpamayo"] = {"finite_saved_trajectories": True, "shapes": expected_shapes,
                          "scope": "model interface smoke on synthetic observations; not driving evaluation"}
print(json.dumps(checks, indent=2))