Physical AI in motion

Explore robot policies, physics, 3D Gaussian worlds, generated indoor scenes and ego-camera driving. These experiments use NVIDIA GPU tools, CARLA and Blender; each recording links to code and checks.

New experiment runs

Stop for a closing gate, then continue

A moving physical collider blocks the passage. The robot waits with zero wheel commands and resumes from new observations when a path opens.

  • WorldGen
  • Isaac Sim 6.2
  • Physics Raycast Sensor
  • A*

A moving box enters the first detour

The initial sensor-generated route goes below the static crate. A box enters this passage, the robot brakes, and a later plan takes it around the other side.

  • WorldGen
  • Isaac Sim 6.2
  • Physics Raycast Sensor
  • A*

Previous controller: the physical gate opens, but map memory remains

The same range sensor and moving gate, using the unchanged previous permanent-map Navigator. The robot stops and still has no route after the gate withdraws.

  • WorldGen
  • Isaac Sim 6.2
  • Physics Raycast Sensor
  • A*

Lower passage blocked: the robot chooses the upper side

The controller receives only ranges, ego pose and a goal. The cyan line is recomputed online; it is not a supplied waypoint list.

  • WorldGen
  • Isaac Sim 6.2
  • Physics Raycast Sensor
  • A*

Upper passage blocked: the choice changes

Move the added box to the opposite side without changing the planner. The measured route changes with the scan.

  • WorldGen
  • Isaac Sim 6.2
  • Physics Raycast Sensor
  • A*

One box, no supplied detour

The original central obstacle is detected from range hits. The planner builds its first route from an empty obstacle map.

  • WorldGen
  • Isaac Sim 6.2
  • Physics Raycast Sensor
  • A*

The fixed-route baseline in the changed layout

The previous waypoint list is unchanged and ignores range data. Live contacts stop the robot at the added box; the goal is not reached.

  • WorldGen
  • Isaac Sim 6.2
  • Physics Raycast Sensor
  • A*

A forward command does not pass through a collider

The same physical robot receives equal wheel-speed targets and stops against the crate. Contact events and measured poses are saved.

  • Isaac Sim 6.2
  • PhysX
  • WorldGen

A closer view of waypoint control

An independent run follows the detour directly. The camera is closer and the generated-room emission is lower than in the main overview. No crate contact was reported.

  • Isaac Sim 6.2
  • PhysX
  • Waypoint feedback

Check contact before you drive

Four 10 cm cubes drop onto the explicit floor. Compare their saved heights with the no-collider and generated-mesh tests in the lab.

  • Isaac Sim 6.2
  • PhysX
  • Collision check

Expand one image into a 360° living room

Actual model inference on an RTX GPU, followed by a translated camera path rendered from the Gaussian PLY. Distance is inferred and not metric ground truth.

  • WorldGen
  • FLUX.1
  • DA-2
  • fVDB renderer

Generate a forest trail from text

Actual model inference on an RTX GPU, followed by a translated camera path rendered from the Gaussian PLY. Distance is inferred and not metric ground truth.

  • WorldGen
  • FLUX.1
  • DA-2
  • fVDB renderer

Clear city

Original-speed, on-board RGB sensor capture in CARLA. Open the lab for setup, camera placement, and per-frame records.

  • CARLA 0.9.16
  • Ego RGB
  • Traffic Manager

Heavy-rain junction

Original-speed, on-board RGB sensor capture in CARLA. Open the lab for setup, camera placement, and per-frame records.

  • CARLA 0.9.16
  • Ego RGB
  • Traffic Manager

Wet suburb at sunset

Original-speed, on-board RGB sensor capture in CARLA. Open the lab for setup, camera placement, and per-frame records.

  • CARLA 0.9.16
  • Ego RGB
  • Traffic Manager

Foggy highway

Original-speed, on-board RGB sensor capture in CARLA. Open the lab for setup, camera placement, and per-frame records.

  • CARLA 0.9.16
  • Ego RGB
  • Traffic Manager

City from one image

SHARP predicts 3D Gaussians from our CARLA image. fVDB renders nearby views. Research-only model weights; no complete 360° scene.

  • Apple SHARP
  • fVDB renderer
  • Single-view reconstruction

Suburb from one image

SHARP predicts 3D Gaussians from our CARLA image. fVDB renders nearby views. Research-only model weights; no complete 360° scene.

  • Apple SHARP
  • fVDB renderer
  • Single-view reconstruction

Four fields on one GPU

Clockwise from the top left: cuRobo paths, Newton solver replay, Alpamayo trajectory replay, and Cosmos generated RGB. Short takes loop; the Newton tile changes examples.

  • cuRobo
  • Newton
  • Cosmos3-Edge
  • Alpamayo2-Edge
  • Isaac Sim

A lift is not yet a placement

November 19 live GR00T run, shown at 2× speed. Both new trials lifted the cube but neither placed it in the bowl.

  • GR00T N1.6-DROID
  • Isaac Lab-Arena
  • Isaac Sim

Same goal, a new obstacle

November 19 GPU replan. Isaac Sim replays the calculated arm collision geometry; sampled sphere clearance is about 2.17 mm.

  • Isaac Sim
  • cuRobo
  • AutoData
  • MimicGen
  • GR00T
  • Isaac Lab
  • Newton

Compare by topic

Robot motion

GR00T places the cube, cuRobo plans around an obstacle, AutoData replays generated demonstrations, and a trained Microduck policy walks in Newton.

  • Isaac Sim
  • cuRobo
  • AutoData
  • MimicGen
  • GR00T
  • Isaac Lab
  • Newton

Physics you can see

Colliders change where objects stop. Friction changes how far they slide, and restitution changes how high they bounce.

  • Isaac Sim
  • PhysX
  • OpenUSD
  • SimReady

From a scene to sensor data

Open a CAD workbench, vary the camera scene in Replicator, and inspect RGB, depth, labels, and a reconstructed point cloud.

  • CadQuery
  • OpenUSD
  • Isaac Sim
  • PhysX
  • Replicator

Experiments around a desk

An alarm rolls away, a snack drawer responds to a card, a cleanup rule puts a mouse in the bin, and a cart tests aisle widths.

  • Isaac Sim
  • PhysX
  • OpenUSD
  • OpenCV

Watch one result

Generate and explore a world from text or an image

WorldGen creates a robotics lab, a forest trail and an expanded living room. Watch translated 3D camera views, then load the generated scene in your browser.

  • WorldGen
  • FLUX.1 / Nunchaku FP4
  • DA-2
  • fVDB renderer

When an obstacle moves, the map must change too

A gate closes during driving and reopens. The old permanent map still blocks the robot; fresh range observations let the new controller wait, clear the map and continue. Actual Isaac Sim GUI recordings at 1×.

  • WorldGen
  • Isaac Sim 6.2
  • Physics Raycast Sensor
  • A*

Fixed waypoints or a route from sensor scans?

A new box blocks the old route. An actual Physics Raycast Sensor feeds an occupancy grid and A* planner, and the same robot chooses the other side. Native Isaac Sim GUI footage at 1× speed.

  • WorldGen
  • Isaac Sim 6.2
  • Physics Raycast Sensor
  • A*

Drive inside the generated WorldGen laboratory

A generated laboratory becomes an Isaac Sim 6.2 scene. Watch an articulated AMR meet a crate, reverse and follow waypoints around it. The recovery is scripted; contacts and motion are live PhysX.

  • WorldGen
  • Isaac Sim 6.2
  • PhysX
  • OpenUSD

Explore a plaza reconstructed with 3DGS

CARLA camera views become an fVDB radiance field. Move the viewpoint in the browser, or watch the full model rendered along a new camera path.

  • fVDB Reality Capture
  • CARLA
  • Spark / World Labs
  • Apple SHARP

Four environments from the driver’s seat

Actual CARLA ego cameras: clear city, heavy rain, wet sunset, and fog. Traffic Manager is a rule-based driver; these are sensor captures, not AI policy results.

  • CARLA 0.9.16
  • Unreal Engine 4.26
  • Traffic Manager

Walk through a procedurally generated house

Infinigen generates a furnished house and plans a camera tour. Blender Cycles renders the geometry; this is procedural generation, not neural world-model inference.

  • Infinigen
  • Blender / Cycles
  • Procedural generation

World generation and driving-model outputs

A real Cosmos camera-conditioned generation, then an Alpamayo prediction replay in Isaac Sim. Separate model experiments on our own scenes.

  • Cosmos3-Edge
  • Alpamayo2-Edge
  • Isaac Sim
  • OpenUSD

Cloth and cables in Newton

Newton computes XPBD cloth, VBD cloth, and twisting cables. Isaac Sim displays the saved solver states.

  • Newton
  • Warp
  • OpenUSD
  • Isaac Sim

GR00T puts the cube in the bowl

Watch the gripper hold, carry, open, and leave the cube resting in the bowl.

  • GR00T N1.6-DROID
  • Isaac Lab-Arena
  • Isaac Sim

Three outputs from one camera

Inspect RGB, camera-plane depth, and semantic masks from 12 captured frames.

  • Isaac Sim
  • Replicator
  • OpenUSD

Microduck walks in Newton

A walking policy trained in Isaac Lab drives the simulated robot.

  • Isaac Lab
  • Newton
  • RSL-RL
  • Microduck

Particles collide in Warp

Warp computes 65,536 interacting particles on the GPU. The viewer draws one in four.

  • NVIDIA Warp
  • CUDA
  • OpenGL
All labs on GitHub Scene and tool credits