Distributed reinforcement learning infrastructure used by AlpaGym.

Limited maintenance

Cosmos RL worker architecture

The figure separates rollout production, policy updates and controller messages. Official figure source

Which process collects experience and which updates the policy?

How the data moves
  1. InputRequest experiencescene/start + rollout settings

    The trainer requests candidate runs for the training problem.

  2. ProcessCollect and scorepolicy actions + environment → reward

    A rollout worker gathers the episode and its score.

  3. OutputUpdate the policyscored experience → training step

    The trainer updates policy weights and logs the result.

Separate collection and training

In distributed reinforcement learning, rollout workers run the policy and collect experience. The trainer uses that experience and its scores to update weights. In AlpaGym, a rollout is a closed-loop drive in AlpaSim. After selecting the task and algorithm, track collection throughput separately from learning and evaluation metrics.

Read rollout settings in a real preset

The published training preset sets rollout.n_generation=2, rollout.batch_size=1, and train.num_epochs=1. These specify generation requests, the rollout batch setting, and training epochs. Inspect AlpaGym rewards and metrics for driving behavior. Cosmos RL project

Architecture and data flow

A controller coordinates policy workers and rollout workers. Rollout replicas produce experiences and policy replicas consume them to train. The documented control messages include rollout, evaluation and weight synchronization. AlpaGym uses the framework for autonomous-driving post-training in AlpaSim.

Physical AI example

Pin the AlpaGym-compatible trainer and run one scene to inspect rewards, termination conditions and weight updates. Compare a checkpoint with the original policy on held-out routes.

What to verify

The upstream repository reports limited maintenance and points to the Cosmos 3 ecosystem for new development. Keep working experiment versions pinned and follow the model-specific recipes when migrating.

License and availability

Apache-2.0 repository code. Model weights, external services and dependencies can have separate terms.

Official sources

Alpagym · NeMo RL · Cosmos Framework