Cosmos world models

See how video and scene conditions become inputs for generation or reasoning.

NVIDIA Cosmos logo
Official visualNVIDIA Cosmos

Cosmos is NVIDIA’s world-model platform for physical AI. Cosmos 3 is the model family that combines multimodal reasoning and generation. The platform also provides separate training, curation and evaluation tools.

NVIDIA published four interactive simulation checkpoints under OpenMDW-1.1 in November 2026. Sim-Bimanual predicts video given robot or camera actions, Sim-Transfer renders video from simulator controls, and Sim-Auto and Transfer-Auto serve driving workflows. For models that output robot actions, see the Cosmos Policy entries.

Read the official architecture

How the data moves
  1. InputChoose the conditioningclip + question or scene controls

    Select inputs for the task your checkpoint supports.

  2. ProcessChoose the taskreasoning · generation · prediction

    Each task uses the video or conditioning in a different way.

  3. OutputEvaluate the outputtext, video, or a predicted rollout

    Use checks suited to what the checkpoint produced.

Cosmos 3 two-tower model architecture

The Reasoner and Generator are jointly trained transformer towers with a shared representation. Reasoner grounds language in visual context. Generator uses grounded context and generation conditions to produce outputs supported by the checkpoint. The NIM image exposes separate Reasoner and Generator runtimes, so a unified model architecture does not imply a shared request contract. NVIDIA technical report · NIM architecture

In the attention mask on the right, the purple region is causal. The Reasoner reads its earlier vision and language tokens while predicting text. The orange region shows the Generator's diffusion tokens attending to both token groups. Image and video enter that branch through a visual VAE, with separate encoders for audio and action. This explains how the Generator can use context from the Reasoner while the Reasoner can run by itself. Cosmos 3 technical report, model architecture

Job Send Receive
Reasoning Video and a question about an object's movement Text or a structured observation, depending on serving configuration
Forward dynamics Initial observation and an action sequence in the checkpoint's format Generated rollout conditioned on those actions
Transfer Prompt and supported spatial controls Video conditioned on the supplied scene layout
Policy Views, task and state expected by the embodiment Action prediction in the checkpoint's representation

Check frame limits, media modes and action dimensions for the chosen checkpoint. Super, Nano and Edge share the model family architecture while their supported tasks and inference integrations vary. Official model reference

The same driving clip can lead to different outputs

Given a driving clip and the question “When did the car change lanes?”, a reasoning task produces a description. To generate a rainy variant, pass the scene conditions and prompt supported by the checkpoint, then check track and lane consistency. When conditioning on actions, vary their sequence and compare the predicted videos.

Record the source clip ID, checkpoint, task, condition or question, and output file. Use the selected checkpoint's documentation for the actual API schema. Evaluate adherence to the supplied conditions in the video and use simulator state or observation logs to confirm physical events such as collisions. Cosmos 3 model reference

Worked example — generate driving scene variants

Cosmos 3 world scenario transfer example

NVIDIA’s demo shows a driving control layout alongside generated footage. For an experiment, fix camera paths and layout controls, change the weather prompt, and compare actor position, lane layout and movement timing against the controls. Keep model revision, seed and source sequence ID with every result. Measure any downstream improvement on real drives held out by source session. Generated footage can suggest plausible outcomes. Use a suitable simulator or recorded measurements to validate collision or contact outcomes. Official demos and recipes · Cosmos Evaluator

Availability

The Cosmos 3 repository identifies OpenMDW-1.1 for code and models. The earlier Predict2 and Reason2 repositories document separate terms. Official license section

See Cosmos Framework for checkpoint-specific training, post-training and serving recipes. NVIDIA NIM covers packaged inference endpoints.