Drive through four environments from the ego camera

Actual CARLA ego cameras: clear city, heavy rain, wet sunset, and fog. Traffic Manager is a rule-based driver; these are sensor captures, not AI policy results.

  • CARLA 0.9.16
  • Unreal Engine 4.26
  • Traffic Manager
Four environments from the driver’s seat
Recorded runExample run and recording

What is moving, and what does the camera measure?

How the data moves
  1. InputChoose a worldTown10HD · Town03 · Town02 · Town04 + weather

    CARLA owns the map, vehicles, physics and sensor simulation.

  2. ProcessAdvance the driver and simulationTraffic Manager controls → 0.05 s tick

    The rule-based driver reads actor/road state. It does not read the RGB video.

  3. OutputRecord the observation960×540 RGB + frame ID + XYZ + speed

    Ego and chase cameras share the tick. Published videos are resized to 768×432.

New actor states return to Traffic Manager for the next control step.

Start with the four-view recording below. These are actual CARLA 0.9.16 RGB sensors, rendered by Unreal Engine, rather than an Isaac Sim replay. City streets, a rainy junction, a wet suburb at sunset, and a foggy highway use four different maps. The bonnet at the bottom of each image locates the camera in the vehicle.

Input and output flow

The driver and the camera have different jobs

CARLA Traffic Manager reads the simulated traffic and road network and applies throttle, steering, and braking. It is a rule-based controller. The RGB camera records what an on-board camera would see; Traffic Manager does not infer its actions from this video. This is useful sensor and environment testing, not evidence of a learned driving model's accuracy. Traffic Manager documentation

One simulation tick advances time by 0.05 s. Two attached cameras capture that same tick: ego at (0.3, 0, 1.45) m relative to the vehicle, and chase at (-6, 0, 3) m. The original RGB sensor buffer has shape [540, 960, 4], with uint8 BGRA channels. capture.py removes alpha and reverses BGR to RGB before writing video. A frame ID pairs an image with the vehicle's measured position and speed.

Text
Road network + current actor states → Traffic Manager → vehicle controls
                                                     ↓
                              CARLA physics → next actor states
                                                     ↓
                          RGB camera + vehicle pose → recorded sample

Read the four cases

Grid position Environment What to compare
Top left Town10HD / clear noon Road markings, traffic, buildings and the driver's view
Top right Town03 / heavy rain Wet surfaces and visibility at a junction
Bottom left Town02 / wet sunset Residential streets and changing lighting
Bottom right Town04 / fog How quickly distant objects disappear

These are different routes, not a matched-route weather benchmark. Each published take contains 320 frames at 20 FPS, runs at its original speed, and links to measured trajectory records. Fog and rain affect appearance; this experiment does not establish a tire-friction change.

First time? Get the lab files

Open a terminal on the Isaac Sim machine for GPU labs, or on your own computer for CPU-only steps. Press Enter after each line. You can copy a multi-line command block as a whole.

Shell
cd "$HOME"
git clone https://github.com/james-joobs/awesome-nvidia-physical-ai.git
cd awesome-nvidia-physical-ai
pwd

This example clones into your home folder. If you already cloned the repo, skip git clone and enter your project folder. In each new terminal, check that pwd ends in awesome-nvidia-physical-ai so commands can find the labs/ files.

Run your own ego-camera recording

1. Start a dedicated server

Use Linux with an NVIDIA graphics driver, Docker and NVIDIA Container Toolkit. Run nvidia-smi first. Stop here if it cannot see your GPU; the environment guide explains that check. The server and Python client must both be 0.9.16. We use this packaged release; the repository's default development branch targets a different Unreal version.

Open terminal A in the repository and run:

Shell
docker pull carlasim/carla:0.9.16
docker run --rm --name physical-ai-carla --gpus all --shm-size=2g \
  -e NVIDIA_DRIVER_CAPABILITIES=all \
  -p 127.0.0.1:24000-24002:24000-24002 \
  carlasim/carla:0.9.16 /bin/bash CarlaUE4.sh \
  -RenderOffScreen -nosound -carla-rpc-port=24000 -quality-level=Epic

Keep terminal A open. -RenderOffScreen still renders real cameras; it does not open a simulator window. The ports listen only on your machine. Run on a free GPU, independently of other simulator sessions.

2. Attach cameras and record

In terminal B, from the same repository:

Shell
python3 -m venv .venv-carla
source .venv-carla/bin/activate
python -m pip install carla==0.9.16 numpy pillow imageio imageio-ffmpeg
python labs/carla-ego/capture.py --output runs/carla-ego

Wait for each synchronized frames message. capture.py loads each map, creates its own vehicles and cameras, advances a synchronous simulation, saves ego/chase MP4s, and destroys only those actors afterward. result.json records every frame's position in meters and speed in m/s. If a route barely moves, the script reports it so you can inspect a red light, traffic, or spawn location before presenting a driving clip.

3. Make the web comparison

With FFmpeg available:

Shell
python labs/carla-ego/package.py --input runs/carla-ego --output runs/carla-web
python3 labs/carla-ego/check.py

The packaging step adds platform/map labels, reduces video size, and makes the four-view grid. When finished, stop your server with docker stop physical-ai-carla in another terminal.

Check the result and choose the next experiment

No GPU is needed to watch the videos, read recorded.json, or run check.py on the bundled results. The checker verifies frame counts, time steps, finite positions, travel distances, and published video hashes. It does not certify driving safety or policy performance.

Use CARLA → fVDB reconstruction to turn known camera views into an actual 3DGS scene. Use Alpamayo inference to inspect a learned model's outputs, then the AlpaSim guide for policy evaluation with the required scene access. These are separate execution paths.

Credits and terms: CARLA 0.9.16 source code is MIT; CARLA-specific assets are CC-BY, with no version specified in its README. Unreal Engine and integrations have their own terms. Our recordings were resized and labeled; see media credits.

Example run and recording

Clear city

Original-speed, on-board RGB sensor capture in CARLA. Open the lab for setup, camera placement, and per-frame records.

Heavy-rain junction

Original-speed, on-board RGB sensor capture in CARLA. Open the lab for setup, camera placement, and per-frame records.

Wet suburb at sunset

Original-speed, on-board RGB sensor capture in CARLA. Open the lab for setup, camera placement, and per-frame records.

Foggy highway

Original-speed, on-board RGB sensor capture in CARLA. Open the lab for setup, camera placement, and per-frame records.

Source code

capture.pyView files on GitHub
Python
"""Record real CARLA RGB sensors; Traffic Manager is a rule-based driver, not an AI policy.

Run against a dedicated CARLA 0.9.16 server. Only actors created here are destroyed.
"""
import argparse
import hashlib
import json
import math
import queue
from pathlib import Path

import carla
import imageio.v2 as imageio
import numpy as np
from PIL import Image

CASES = [
    ("city-clear", "Town10HD_Opt", "ClearNoon", 1),
    ("junction-rain", "Town03", "HardRainNoon", 8),
    ("suburb-sunset", "Town02", "WetSunset", 0),
    ("highway-fog", "Town04_Opt", "CloudyNoon", 20),
]


def image_for_frame(q, frame):
    while True:
        image = q.get(timeout=30)
        if image.frame == frame:
            return np.frombuffer(image.raw_data, dtype=np.uint8).reshape(image.height, image.width, 4)[:, :, :3][:, :, ::-1].copy()
        if image.frame > frame:
            raise RuntimeError(f"Missed sensor frame {frame}; received {image.frame}")


def record(client, case, output, fps, seconds, width, height):
    name, map_name, weather_name, spawn_index = case
    print(f"Loading {name}: {map_name}, {weather_name}", flush=True)
    world = client.load_world(map_name)
    previous = world.get_settings()
    settings = world.get_settings()
    settings.synchronous_mode = True
    settings.fixed_delta_seconds = 1 / fps
    world.apply_settings(settings)
    weather = getattr(carla.WeatherParameters, weather_name)
    if name == "highway-fog":
        weather.fog_density = 55
        weather.fog_distance = 8
        weather.fog_falloff = 1.0
    world.set_weather(weather)
    tm = client.get_trafficmanager(24005)
    tm.set_synchronous_mode(True)
    tm.set_random_device_seed(42)
    tm.global_percentage_speed_difference(20)
    actors = []
    writers = []
    try:
        blueprints = world.get_blueprint_library()
        points = world.get_map().get_spawn_points()
        bp = blueprints.find("vehicle.tesla.model3")
        bp.set_attribute("role_name", "hero")
        ego = world.spawn_actor(bp, points[spawn_index])
        actors.append(ego)
        for index in range(3, min(len(points), 75), 4):
            other = world.try_spawn_actor(bp, points[index])
            if other:
                actors.append(other)
                other.set_autopilot(True, 24005)
        ego.set_autopilot(True, 24005)
        tm.ignore_lights_percentage(ego, 0)
        tm.distance_to_leading_vehicle(ego, 3)
        streams = []
        folder = output / name
        folder.mkdir(parents=True, exist_ok=True)
        for view, transform in [
            ("ego", carla.Transform(carla.Location(x=0.3, z=1.45))),
            ("chase", carla.Transform(carla.Location(x=-6, z=3), carla.Rotation(pitch=-12))),
        ]:
            camera_bp = blueprints.find("sensor.camera.rgb")
            for key, value in {"image_size_x": width, "image_size_y": height, "fov": 90, "sensor_tick": 0}.items():
                camera_bp.set_attribute(key, str(value))
            camera_bp.set_attribute("motion_blur_intensity", "0")
            camera = world.spawn_actor(camera_bp, transform, attach_to=ego)
            actors.append(camera)
            q = queue.Queue()
            camera.listen(q.put)
            path = folder / f"{view}.mp4"
            writer = imageio.get_writer(path, fps=fps, codec="libx264", quality=None,
                pixelformat="yuv420p", macro_block_size=None,
                output_params=["-crf", "23", "-preset", "fast", "-movflags", "+faststart"])
            writers.append(writer)
            streams.append((view, q, writer, path))
        samples = []
        for i in range(round((seconds + 2) * fps)):
            frame = world.tick()
            images = [(view, image_for_frame(q, frame), writer, path) for view, q, writer, path in streams]
            if i < 2 * fps:
                continue
            for view, rgb, writer, path in images:
                writer.append_data(rgb)
                if i == 5 * fps:
                    Image.fromarray(rgb).save(folder / f"{view}.jpg", quality=92)
            transform, velocity = ego.get_transform(), ego.get_velocity()
            samples.append({"frame": frame, "simulation_seconds": round((i - 2*fps)/fps, 3),
                "position_m": [transform.location.x, transform.location.y, transform.location.z],
                "rotation_deg": [transform.rotation.pitch, transform.rotation.yaw, transform.rotation.roll],
                "speed_mps": math.sqrt(velocity.x**2 + velocity.y**2 + velocity.z**2)})
        for writer in writers:
            writer.close()
        writers.clear()
        displacement = math.dist(samples[0]["position_m"], samples[-1]["position_m"])
        distance = sum(math.dist(a["position_m"], b["position_m"]) for a,b in zip(samples, samples[1:]))
        report = {"id": name, "map": world.get_map().name, "weather": weather_name,
            "fog_density": weather.fog_density, "controller": "CARLA Traffic Manager (rule-based)",
            "npc_vehicles": len(actors)-3, "fps": fps, "frames": len(samples), "resolution": [width, height],
            "displacement_m": round(displacement, 3), "distance_m": round(distance, 3), "trajectory": samples,
            "videos": [{"view": view, "file": str(path.relative_to(output)),
                "sha256": hashlib.sha256(path.read_bytes()).hexdigest()} for view,_,_,path in streams]}
        (folder / "result.json").write_text(json.dumps(report, indent=2)+"\n")
        print(f"{name}: {len(samples)} synchronized frames, {distance:.1f}m travelled", flush=True)
        if distance < 5:
            raise RuntimeError(f"{name}: ego did not travel enough; review the spawn and traffic")
        return report
    finally:
        for writer in writers:
            writer.close()
        for actor in reversed(actors):
            if actor.is_alive:
                if actor.type_id.startswith("sensor."):
                    actor.stop()
                actor.destroy()
        tm.set_synchronous_mode(False)
        world.apply_settings(previous)


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--host", default="127.0.0.1")
    parser.add_argument("--port", type=int, default=24000)
    parser.add_argument("--output", type=Path, required=True)
    parser.add_argument("--case", choices=[c[0] for c in CASES], action="append")
    parser.add_argument("--seconds", type=float, default=16)
    parser.add_argument("--fps", type=int, default=20)
    parser.add_argument("--width", type=int, default=960)
    parser.add_argument("--height", type=int, default=540)
    args = parser.parse_args()
    client = carla.Client(args.host, args.port)
    client.set_timeout(90)
    assert client.get_server_version() == "0.9.16", "Use the pinned CARLA server and client"
    cases = [c for c in CASES if not args.case or c[0] in args.case]
    results = [record(client, c, args.output, args.fps, args.seconds, args.width, args.height) for c in cases]
    (args.output / "result.json").write_text(json.dumps({"server_version": client.get_server_version(),
        "client_version": client.get_client_version(), "seed": 42, "cases": results}, indent=2)+"\n")


if __name__ == "__main__":
    main()
package.pyView files on GitHub
Python
"""Make labeled, lightweight web videos from the four original CARLA sensor takes."""
import argparse
import hashlib
import json
import subprocess
from pathlib import Path

from PIL import Image, ImageDraw, ImageFont

CASES=[("city-clear","Town10HD / clear city"),("junction-rain","Town03 / heavy rain"),
       ("suburb-sunset","Town02 / wet sunset"),("highway-fog","Town04 / fog")]


def main():
    parser=argparse.ArgumentParser()
    parser.add_argument("--input",type=Path,required=True)
    parser.add_argument("--output",type=Path,required=True)
    parser.add_argument("--ffmpeg",default="ffmpeg")
    args=parser.parse_args();args.output.mkdir(parents=True,exist_ok=True)
    font_path=next(Path(p) for p in ["/System/Library/Fonts/Supplemental/Arial.ttf","/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf"] if Path(p).exists())
    font=ImageFont.truetype(str(font_path),18)
    reports=[]
    for name,label in CASES:
        overlay=Image.new("RGBA",(768,432))
        draw=ImageDraw.Draw(overlay)
        draw.rectangle((0,0,768,34),fill=(0,0,0,180))
        draw.text((12,7),f"CARLA 0.9.16  |  Ego RGB  |  {label}",font=font,fill="white")
        png=args.output/f"carla-{name}-label.png";overlay.save(png)
        target=args.output/f"carla-{name}.mp4"
        subprocess.run([args.ffmpeg,"-hide_banner","-loglevel","error","-y","-i",str(args.input/name/"ego.mp4"),"-i",str(png),
            "-filter_complex","[0:v]scale=768:432[v];[v][1:v]overlay=0:0", "-an","-c:v","libx264","-crf","26","-pix_fmt","yuv420p","-movflags","+faststart",str(target)],check=True)
        subprocess.run([args.ffmpeg,"-hide_banner","-loglevel","error","-y","-ss","4","-i",str(target),"-frames:v","1",str(target.with_suffix(".jpg"))],check=True)
        report=json.loads((args.input/name/"result.json").read_text())
        report["published"]={"file":target.name,"sha256":hashlib.sha256(target.read_bytes()).hexdigest(),"resolution":[768,432],"edit":"20 fps / original speed; label overlay; resize and H.264 CRF26"}
        reports.append(report)
        png.unlink()
    inputs=[]
    for name,_ in CASES:
        inputs.extend(["-i",str(args.output/f"carla-{name}.mp4")])
    grid=args.output/"carla-ego.mp4"
    filters=";".join(f"[{i}:v]scale=640:360[v{i}]" for i in range(4))+";[v0][v1][v2][v3]xstack=inputs=4:layout=0_0|640_0|0_360|640_360[out]"
    subprocess.run([args.ffmpeg,"-hide_banner","-loglevel","error","-y",*inputs,"-filter_complex",filters,"-map","[out]","-an","-c:v","libx264","-crf","25","-pix_fmt","yuv420p","-movflags","+faststart",str(grid)],check=True)
    subprocess.run([args.ffmpeg,"-hide_banner","-loglevel","error","-y","-ss","4","-i",str(grid),"-frames:v","1",str(grid.with_suffix(".jpg"))],check=True)
    (args.output/"carla-recorded.json").write_text(json.dumps({"date":"2026-11-20","server_version":"0.9.16","seed":42,
        "controller":"Traffic Manager; rule-based closed-loop simulation, not a learned driving policy",
        "grid_order":[c[0] for c in CASES],"cases":reports,
        "grid":{"file":grid.name,"sha256":hashlib.sha256(grid.read_bytes()).hexdigest(),"frames":320,"fps":20,"resolution":[1280,720]}},indent=2)+"\n")


if __name__=="__main__":
    main()
check.pyView files on GitHub
Python
"""CPU-only checks of our recorded CARLA trajectories and published sensor videos."""
import hashlib
import json
import math
from pathlib import Path


def main():
    root=Path(__file__).resolve().parents[2]
    data=json.loads((root/"labs/carla-ego/recorded.json").read_text())
    assert data["server_version"]=="0.9.16" and len(data["cases"])==4
    for case in data["cases"]:
        rows=case["trajectory"]
        assert len(rows)==case["frames"]==320 and case["fps"]==20
        assert all(b["frame"]-a["frame"]==1 and math.isclose(b["simulation_seconds"]-a["simulation_seconds"],.05,abs_tol=1e-6) for a,b in zip(rows,rows[1:]))
        assert all(all(math.isfinite(n) for n in row["position_m"]) and 0<=row["speed_mps"]<50 for row in rows)
        distance=sum(math.dist(a["position_m"],b["position_m"]) for a,b in zip(rows,rows[1:]))
        assert math.isclose(distance,case["distance_m"],abs_tol=.001) and distance>5
        video=root/"site/public/showcase"/case["published"]["file"]
        assert hashlib.sha256(video.read_bytes()).hexdigest()==case["published"]["sha256"]
        print(f'{case["id"]}: {len(rows)} frames, {distance:.1f}m, video hash OK')
    grid=data["grid"]
    assert hashlib.sha256((root/"site/public/showcase"/grid["file"]).read_bytes()).hexdigest()==grid["sha256"]
    print("Four CARLA sensor runs verified. This is not a learned-policy safety evaluation.")


if __name__=="__main__":
    main()