How Physical AI tools work
Read the diagrams and data examples in order for the field you want to learn, then follow the linked lab.
48 guides
Set up your environment
Check the GPU and driver first, then learn how to open a stage in the GUI and run a capture from the terminal.
Physics & simulation
Start with a stage and a falling cube. Then compare contact, solver behavior, and GPU particle simulation.
- Isaac SimFollow one USD stage through physics, RTX rendering, and sensor capture.
- Drop your first cubeRun the same scene twice to see how a rigid body and a collider change what happens.
- PhysXUnderstand how gravity and contact change a rigid body and how to read the resulting poses.
- SimReadyRead the physical and semantic properties that turn a USD asset into a simulation input.
- Physics you can seeColliders change where objects stop. Friction changes how far they slide, and restitution changes how high they bounce.
- NewtonFollow rigid-body state through a Newton step and compare its saved poses with PhysX.
- Cloth and cables in NewtonNewton computes XPBD cloth, VBD cloth, and twisting cables. Isaac Sim displays the saved solver states.
- WarpFollow particle arrays from a Python launch through CUDA kernels to simulation and viewer outputs.
- Particles collide in WarpWarp computes 65,536 interacting particles on the GPU. The viewer draws one in four.
3D assets & synthetic data
Build and capture a CAD scene first. Follow with RGB-D reconstruction, then explore advanced capture or reconstruction from photos.
- OpenUSDTrace how reusable asset references and scene edits produce the USD stage Isaac Sim opens.
- Workbench CAD to Isaac SimConvert a STEP file to OpenUSD and check the tabletop’s dimensions and collision geometry.
- ReplicatorConnect a camera to annotators and see how their outputs become files.
- Build and verify a workbench camera datasetTurn CAD into an Isaac Sim scene, capture RGB, depth and masks, then check the result on a CPU.
- From a scene to sensor dataOpen a CAD workbench, vary the camera scene in Replicator, and inspect RGB, depth, labels, and a reconstructed point cloud.
- Three outputs from one cameraInspect RGB, camera-plane depth, and semantic masks from 12 captured frames.
- fVDB Reality CaptureReconstruct a scene from calibrated photos, render another viewpoint, and compare the available exports.
Robotics & control
Read how environments and evaluation work, then follow a policy run, motion planning, demonstration capture, and locomotion training.
- Isaac LabFollow observations, actions, rewards, and resets through parallel robot training environments.
- Experiments around a deskAn alarm rolls away, a snack drawer responds to a card, a cleanup rule puts a mouse in the bin, and a cart tests aisle widths.
- Isaac Lab-ArenaFollow one robot policy episode through task setup, policy adaptation, and milestone scoring.
- Choose and evaluate a robot foundation modelMatch the checkpoint to your robot or benchmark, then verify one rollout.
- GR00T puts the cube in the bowlWatch the gripper hold, carry, open, and leave the cube resting in the bowl.
- cuRoboCompare how a fixed target and a changed obstacle produce different joint trajectories.
- Robot motionGR00T places the cube, cuRobo plans around an obstacle, AutoData replays generated demonstrations, and a trained Microduck policy walks in Newton.
- Isaac CaptureUnderstand how tracked human poses become robot commands and synchronized demonstrations.
- Video-to-DataTrace how human manipulation video becomes a robot training candidate.
- AutoDataSee how a seed demonstration becomes new simulator-checked training examples.
- Microduck walks in NewtonA walking policy trained in Isaac Lab drives the simulated robot.
Autonomous driving
Reconstruct the scene and read the policy output first. Evaluate a closed-loop drive before exploring rewards and policy updates.
- Four environments from the driver’s seatActual CARLA ego cameras: clear city, heavy rain, wet sunset, and fog. Traffic Manager is a rule-based driver; these are sensor captures, not AI policy results.
- NuRecSee how sensor logs become a scene that can render new camera views.
- AlpamayoRead the public 2 Super inference sample from camera and ego history through reasoning and future waypoints.
- AlpaSim closed-loop evaluationRun the baseline policy on one reconstructed driving scene with the NuRec renderer.
- AlpaGymTrace a configured AlpaSim rollout through reward calculation and Cosmos-RL policy updates.
- Cosmos RLSee how the rollout and trainer roles exchange scored experience in AlpaGym.
3D worlds & rendering
Distinguish generated worlds, reconstructed 3DGS and procedural geometry first, then explore video generation, data curation and model serving.
- Choose a 3D world workflowCompare explorable 3D worlds, measured Gaussian scenes, procedural geometry, and action-conditioned video.
- Generate and explore a world from text or an imageWorldGen creates a robotics lab, a forest trail and an expanded living room. Watch translated 3D camera views, then load the generated scene in your browser.
- Drive inside the generated WorldGen laboratoryA generated laboratory becomes an Isaac Sim 6.2 scene. Watch an articulated AMR meet a crate, reverse and follow waypoints around it. The recovery is scripted; contacts and motion are live PhysX.
- Fixed waypoints or a route from sensor scans?A new box blocks the old route. An actual Physics Raycast Sensor feeds an occupancy grid and A* planner, and the same robot chooses the other side. Native Isaac Sim GUI footage at 1× speed.
- When an obstacle moves, the map must change tooA gate closes during driving and reopens. The old permanent map still blocks the robot; fresh range observations let the new controller wait, clear the map and continue. Actual Isaac Sim GUI recordings at 1×.
- Explore a plaza reconstructed with 3DGSCARLA camera views become an fVDB radiance field. Move the viewpoint in the browser, or watch the full model rendered along a new camera path.
- Walk through a procedurally generated houseInfinigen generates a furnished house and plans a camera tour. Blender Cycles renders the geometry; this is procedural generation, not neural world-model inference.
- Cosmos CuratorFollow raw recordings through clipping, filtering, and annotation to a training dataset.
- Cosmos world modelsSee how video and scene conditions become inputs for generation or reasoning.
- World generation and driving-model outputsA real Cosmos camera-conditioned generation, then an Alpamayo prediction replay in Isaac Sim. Separate model experiments on our own scenes.
- Cosmos Data FactoryTrace a blueprint from data sources through generation and quality checks to an exported dataset.
- Metropolis VSSSee how video intervals become evidence for search and answers.
- Route model requests with NeMo SwitchyardFollow a request from a video analysis app to a model endpoint and see what to check before serving it.