What is moving, and what does the camera measure?
- InputChoose a world
Town10HD · Town03 · Town02 · Town04 + weatherCARLA owns the map, vehicles, physics and sensor simulation.
- ProcessAdvance the driver and simulation
Traffic Manager controls → 0.05 s tickThe rule-based driver reads actor/road state. It does not read the RGB video.
- OutputRecord the observation
960×540 RGB + frame ID + XYZ + speedEgo and chase cameras share the tick. Published videos are resized to 768×432.
New actor states return to Traffic Manager for the next control step.
Start with the four-view recording below. These are actual CARLA 0.9.16 RGB sensors, rendered by Unreal Engine, rather than an Isaac Sim replay. City streets, a rainy junction, a wet suburb at sunset, and a foggy highway use four different maps. The bonnet at the bottom of each image locates the camera in the vehicle.
The driver and the camera have different jobs
CARLA Traffic Manager reads the simulated traffic and road network and applies throttle, steering, and braking. It is a rule-based controller. The RGB camera records what an on-board camera would see; Traffic Manager does not infer its actions from this video. This is useful sensor and environment testing, not evidence of a learned driving model's accuracy. Traffic Manager documentation
One simulation tick advances time by 0.05 s. Two attached cameras capture that same tick: ego at (0.3, 0, 1.45) m relative to the vehicle, and chase at (-6, 0, 3) m. The original RGB sensor buffer has shape [540, 960, 4], with uint8 BGRA channels. capture.py removes alpha and reverses BGR to RGB before writing video. A frame ID pairs an image with the vehicle's measured position and speed.
Road network + current actor states → Traffic Manager → vehicle controls
↓
CARLA physics → next actor states
↓
RGB camera + vehicle pose → recorded sample
Read the four cases
| Grid position | Environment | What to compare |
|---|---|---|
| Top left | Town10HD / clear noon | Road markings, traffic, buildings and the driver's view |
| Top right | Town03 / heavy rain | Wet surfaces and visibility at a junction |
| Bottom left | Town02 / wet sunset | Residential streets and changing lighting |
| Bottom right | Town04 / fog | How quickly distant objects disappear |
These are different routes, not a matched-route weather benchmark. Each published take contains 320 frames at 20 FPS, runs at its original speed, and links to measured trajectory records. Fog and rain affect appearance; this experiment does not establish a tire-friction change.
First time? Get the lab files
Open a terminal on the Isaac Sim machine for GPU labs, or on your own computer for CPU-only steps. Press Enter after each line. You can copy a multi-line command block as a whole.
cd "$HOME"
git clone https://github.com/james-joobs/awesome-nvidia-physical-ai.git
cd awesome-nvidia-physical-ai
pwd
This example clones into your home folder. If you already cloned the repo, skip git clone and enter your project folder. In each new terminal, check that pwd ends in awesome-nvidia-physical-ai so commands can find the labs/ files.
Run your own ego-camera recording
1. Start a dedicated server
Use Linux with an NVIDIA graphics driver, Docker and NVIDIA Container Toolkit. Run nvidia-smi first. Stop here if it cannot see your GPU; the environment guide explains that check. The server and Python client must both be 0.9.16. We use this packaged release; the repository's default development branch targets a different Unreal version.
Open terminal A in the repository and run:
docker pull carlasim/carla:0.9.16
docker run --rm --name physical-ai-carla --gpus all --shm-size=2g \
-e NVIDIA_DRIVER_CAPABILITIES=all \
-p 127.0.0.1:24000-24002:24000-24002 \
carlasim/carla:0.9.16 /bin/bash CarlaUE4.sh \
-RenderOffScreen -nosound -carla-rpc-port=24000 -quality-level=Epic
Keep terminal A open. -RenderOffScreen still renders real cameras; it does not open a simulator window. The ports listen only on your machine. Run on a free GPU, independently of other simulator sessions.
2. Attach cameras and record
In terminal B, from the same repository:
python3 -m venv .venv-carla
source .venv-carla/bin/activate
python -m pip install carla==0.9.16 numpy pillow imageio imageio-ffmpeg
python labs/carla-ego/capture.py --output runs/carla-ego
Wait for each synchronized frames message. capture.py loads each map, creates its own vehicles and cameras, advances a synchronous simulation, saves ego/chase MP4s, and destroys only those actors afterward. result.json records every frame's position in meters and speed in m/s. If a route barely moves, the script reports it so you can inspect a red light, traffic, or spawn location before presenting a driving clip.
3. Make the web comparison
With FFmpeg available:
python labs/carla-ego/package.py --input runs/carla-ego --output runs/carla-web
python3 labs/carla-ego/check.py
The packaging step adds platform/map labels, reduces video size, and makes the four-view grid. When finished, stop your server with docker stop physical-ai-carla in another terminal.
Check the result and choose the next experiment
No GPU is needed to watch the videos, read recorded.json, or run check.py on the bundled results. The checker verifies frame counts, time steps, finite positions, travel distances, and published video hashes. It does not certify driving safety or policy performance.
Use CARLA → fVDB reconstruction to turn known camera views into an actual 3DGS scene. Use Alpamayo inference to inspect a learned model's outputs, then the AlpaSim guide for policy evaluation with the required scene access. These are separate execution paths.
Credits and terms: CARLA 0.9.16 source code is MIT; CARLA-specific assets are CC-BY, with no version specified in its README. Unreal Engine and integrations have their own terms. Our recordings were resized and labeled; see media credits.
Example run and recording
Clear city
Original-speed, on-board RGB sensor capture in CARLA. Open the lab for setup, camera placement, and per-frame records.
Heavy-rain junction
Original-speed, on-board RGB sensor capture in CARLA. Open the lab for setup, camera placement, and per-frame records.
Wet suburb at sunset
Original-speed, on-board RGB sensor capture in CARLA. Open the lab for setup, camera placement, and per-frame records.
Foggy highway
Original-speed, on-board RGB sensor capture in CARLA. Open the lab for setup, camera placement, and per-frame records.
Source code
capture.py
View files on GitHub"""Record real CARLA RGB sensors; Traffic Manager is a rule-based driver, not an AI policy.
Run against a dedicated CARLA 0.9.16 server. Only actors created here are destroyed.
"""
import argparse
import hashlib
import json
import math
import queue
from pathlib import Path
import carla
import imageio.v2 as imageio
import numpy as np
from PIL import Image
CASES = [
("city-clear", "Town10HD_Opt", "ClearNoon", 1),
("junction-rain", "Town03", "HardRainNoon", 8),
("suburb-sunset", "Town02", "WetSunset", 0),
("highway-fog", "Town04_Opt", "CloudyNoon", 20),
]
def image_for_frame(q, frame):
while True:
image = q.get(timeout=30)
if image.frame == frame:
return np.frombuffer(image.raw_data, dtype=np.uint8).reshape(image.height, image.width, 4)[:, :, :3][:, :, ::-1].copy()
if image.frame > frame:
raise RuntimeError(f"Missed sensor frame {frame}; received {image.frame}")
def record(client, case, output, fps, seconds, width, height):
name, map_name, weather_name, spawn_index = case
print(f"Loading {name}: {map_name}, {weather_name}", flush=True)
world = client.load_world(map_name)
previous = world.get_settings()
settings = world.get_settings()
settings.synchronous_mode = True
settings.fixed_delta_seconds = 1 / fps
world.apply_settings(settings)
weather = getattr(carla.WeatherParameters, weather_name)
if name == "highway-fog":
weather.fog_density = 55
weather.fog_distance = 8
weather.fog_falloff = 1.0
world.set_weather(weather)
tm = client.get_trafficmanager(24005)
tm.set_synchronous_mode(True)
tm.set_random_device_seed(42)
tm.global_percentage_speed_difference(20)
actors = []
writers = []
try:
blueprints = world.get_blueprint_library()
points = world.get_map().get_spawn_points()
bp = blueprints.find("vehicle.tesla.model3")
bp.set_attribute("role_name", "hero")
ego = world.spawn_actor(bp, points[spawn_index])
actors.append(ego)
for index in range(3, min(len(points), 75), 4):
other = world.try_spawn_actor(bp, points[index])
if other:
actors.append(other)
other.set_autopilot(True, 24005)
ego.set_autopilot(True, 24005)
tm.ignore_lights_percentage(ego, 0)
tm.distance_to_leading_vehicle(ego, 3)
streams = []
folder = output / name
folder.mkdir(parents=True, exist_ok=True)
for view, transform in [
("ego", carla.Transform(carla.Location(x=0.3, z=1.45))),
("chase", carla.Transform(carla.Location(x=-6, z=3), carla.Rotation(pitch=-12))),
]:
camera_bp = blueprints.find("sensor.camera.rgb")
for key, value in {"image_size_x": width, "image_size_y": height, "fov": 90, "sensor_tick": 0}.items():
camera_bp.set_attribute(key, str(value))
camera_bp.set_attribute("motion_blur_intensity", "0")
camera = world.spawn_actor(camera_bp, transform, attach_to=ego)
actors.append(camera)
q = queue.Queue()
camera.listen(q.put)
path = folder / f"{view}.mp4"
writer = imageio.get_writer(path, fps=fps, codec="libx264", quality=None,
pixelformat="yuv420p", macro_block_size=None,
output_params=["-crf", "23", "-preset", "fast", "-movflags", "+faststart"])
writers.append(writer)
streams.append((view, q, writer, path))
samples = []
for i in range(round((seconds + 2) * fps)):
frame = world.tick()
images = [(view, image_for_frame(q, frame), writer, path) for view, q, writer, path in streams]
if i < 2 * fps:
continue
for view, rgb, writer, path in images:
writer.append_data(rgb)
if i == 5 * fps:
Image.fromarray(rgb).save(folder / f"{view}.jpg", quality=92)
transform, velocity = ego.get_transform(), ego.get_velocity()
samples.append({"frame": frame, "simulation_seconds": round((i - 2*fps)/fps, 3),
"position_m": [transform.location.x, transform.location.y, transform.location.z],
"rotation_deg": [transform.rotation.pitch, transform.rotation.yaw, transform.rotation.roll],
"speed_mps": math.sqrt(velocity.x**2 + velocity.y**2 + velocity.z**2)})
for writer in writers:
writer.close()
writers.clear()
displacement = math.dist(samples[0]["position_m"], samples[-1]["position_m"])
distance = sum(math.dist(a["position_m"], b["position_m"]) for a,b in zip(samples, samples[1:]))
report = {"id": name, "map": world.get_map().name, "weather": weather_name,
"fog_density": weather.fog_density, "controller": "CARLA Traffic Manager (rule-based)",
"npc_vehicles": len(actors)-3, "fps": fps, "frames": len(samples), "resolution": [width, height],
"displacement_m": round(displacement, 3), "distance_m": round(distance, 3), "trajectory": samples,
"videos": [{"view": view, "file": str(path.relative_to(output)),
"sha256": hashlib.sha256(path.read_bytes()).hexdigest()} for view,_,_,path in streams]}
(folder / "result.json").write_text(json.dumps(report, indent=2)+"\n")
print(f"{name}: {len(samples)} synchronized frames, {distance:.1f}m travelled", flush=True)
if distance < 5:
raise RuntimeError(f"{name}: ego did not travel enough; review the spawn and traffic")
return report
finally:
for writer in writers:
writer.close()
for actor in reversed(actors):
if actor.is_alive:
if actor.type_id.startswith("sensor."):
actor.stop()
actor.destroy()
tm.set_synchronous_mode(False)
world.apply_settings(previous)
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--host", default="127.0.0.1")
parser.add_argument("--port", type=int, default=24000)
parser.add_argument("--output", type=Path, required=True)
parser.add_argument("--case", choices=[c[0] for c in CASES], action="append")
parser.add_argument("--seconds", type=float, default=16)
parser.add_argument("--fps", type=int, default=20)
parser.add_argument("--width", type=int, default=960)
parser.add_argument("--height", type=int, default=540)
args = parser.parse_args()
client = carla.Client(args.host, args.port)
client.set_timeout(90)
assert client.get_server_version() == "0.9.16", "Use the pinned CARLA server and client"
cases = [c for c in CASES if not args.case or c[0] in args.case]
results = [record(client, c, args.output, args.fps, args.seconds, args.width, args.height) for c in cases]
(args.output / "result.json").write_text(json.dumps({"server_version": client.get_server_version(),
"client_version": client.get_client_version(), "seed": 42, "cases": results}, indent=2)+"\n")
if __name__ == "__main__":
main()
package.py
View files on GitHub"""Make labeled, lightweight web videos from the four original CARLA sensor takes."""
import argparse
import hashlib
import json
import subprocess
from pathlib import Path
from PIL import Image, ImageDraw, ImageFont
CASES=[("city-clear","Town10HD / clear city"),("junction-rain","Town03 / heavy rain"),
("suburb-sunset","Town02 / wet sunset"),("highway-fog","Town04 / fog")]
def main():
parser=argparse.ArgumentParser()
parser.add_argument("--input",type=Path,required=True)
parser.add_argument("--output",type=Path,required=True)
parser.add_argument("--ffmpeg",default="ffmpeg")
args=parser.parse_args();args.output.mkdir(parents=True,exist_ok=True)
font_path=next(Path(p) for p in ["/System/Library/Fonts/Supplemental/Arial.ttf","/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf"] if Path(p).exists())
font=ImageFont.truetype(str(font_path),18)
reports=[]
for name,label in CASES:
overlay=Image.new("RGBA",(768,432))
draw=ImageDraw.Draw(overlay)
draw.rectangle((0,0,768,34),fill=(0,0,0,180))
draw.text((12,7),f"CARLA 0.9.16 | Ego RGB | {label}",font=font,fill="white")
png=args.output/f"carla-{name}-label.png";overlay.save(png)
target=args.output/f"carla-{name}.mp4"
subprocess.run([args.ffmpeg,"-hide_banner","-loglevel","error","-y","-i",str(args.input/name/"ego.mp4"),"-i",str(png),
"-filter_complex","[0:v]scale=768:432[v];[v][1:v]overlay=0:0", "-an","-c:v","libx264","-crf","26","-pix_fmt","yuv420p","-movflags","+faststart",str(target)],check=True)
subprocess.run([args.ffmpeg,"-hide_banner","-loglevel","error","-y","-ss","4","-i",str(target),"-frames:v","1",str(target.with_suffix(".jpg"))],check=True)
report=json.loads((args.input/name/"result.json").read_text())
report["published"]={"file":target.name,"sha256":hashlib.sha256(target.read_bytes()).hexdigest(),"resolution":[768,432],"edit":"20 fps / original speed; label overlay; resize and H.264 CRF26"}
reports.append(report)
png.unlink()
inputs=[]
for name,_ in CASES:
inputs.extend(["-i",str(args.output/f"carla-{name}.mp4")])
grid=args.output/"carla-ego.mp4"
filters=";".join(f"[{i}:v]scale=640:360[v{i}]" for i in range(4))+";[v0][v1][v2][v3]xstack=inputs=4:layout=0_0|640_0|0_360|640_360[out]"
subprocess.run([args.ffmpeg,"-hide_banner","-loglevel","error","-y",*inputs,"-filter_complex",filters,"-map","[out]","-an","-c:v","libx264","-crf","25","-pix_fmt","yuv420p","-movflags","+faststart",str(grid)],check=True)
subprocess.run([args.ffmpeg,"-hide_banner","-loglevel","error","-y","-ss","4","-i",str(grid),"-frames:v","1",str(grid.with_suffix(".jpg"))],check=True)
(args.output/"carla-recorded.json").write_text(json.dumps({"date":"2026-11-20","server_version":"0.9.16","seed":42,
"controller":"Traffic Manager; rule-based closed-loop simulation, not a learned driving policy",
"grid_order":[c[0] for c in CASES],"cases":reports,
"grid":{"file":grid.name,"sha256":hashlib.sha256(grid.read_bytes()).hexdigest(),"frames":320,"fps":20,"resolution":[1280,720]}},indent=2)+"\n")
if __name__=="__main__":
main()
check.py
View files on GitHub"""CPU-only checks of our recorded CARLA trajectories and published sensor videos."""
import hashlib
import json
import math
from pathlib import Path
def main():
root=Path(__file__).resolve().parents[2]
data=json.loads((root/"labs/carla-ego/recorded.json").read_text())
assert data["server_version"]=="0.9.16" and len(data["cases"])==4
for case in data["cases"]:
rows=case["trajectory"]
assert len(rows)==case["frames"]==320 and case["fps"]==20
assert all(b["frame"]-a["frame"]==1 and math.isclose(b["simulation_seconds"]-a["simulation_seconds"],.05,abs_tol=1e-6) for a,b in zip(rows,rows[1:]))
assert all(all(math.isfinite(n) for n in row["position_m"]) and 0<=row["speed_mps"]<50 for row in rows)
distance=sum(math.dist(a["position_m"],b["position_m"]) for a,b in zip(rows,rows[1:]))
assert math.isclose(distance,case["distance_m"],abs_tol=.001) and distance>5
video=root/"site/public/showcase"/case["published"]["file"]
assert hashlib.sha256(video.read_bytes()).hexdigest()==case["published"]["sha256"]
print(f'{case["id"]}: {len(rows)} frames, {distance:.1f}m, video hash OK')
grid=data["grid"]
assert hashlib.sha256((root/"site/public/showcase"/grid["file"]).read_bytes()).hexdigest()==grid["sha256"]
print("Four CARLA sensor runs verified. This is not a learned-policy safety evaluation.")
if __name__=="__main__":
main()




