Distributed reinforcement learning infrastructure used by AlpaGym.
Limited maintenance
The figure separates rollout production, policy updates and controller messages. Official figure source
Which process collects experience and which updates the policy?
- InputRequest experience
scene/start + rollout settingsThe trainer requests candidate runs for the training problem.
- ProcessCollect and score
policy actions + environment → rewardA rollout worker gathers the episode and its score.
- OutputUpdate the policy
scored experience → training stepThe trainer updates policy weights and logs the result.
Separate collection and training
In distributed reinforcement learning, rollout workers run the policy and collect experience. The trainer uses that experience and its scores to update weights. In AlpaGym, a rollout is a closed-loop drive in AlpaSim. After selecting the task and algorithm, track collection throughput separately from learning and evaluation metrics.
Read rollout settings in a real preset
The published training preset sets rollout.n_generation=2, rollout.batch_size=1, and train.num_epochs=1. These specify generation requests, the rollout batch setting, and training epochs. Inspect AlpaGym rewards and metrics for driving behavior. Cosmos RL project
Architecture and data flow
A controller coordinates policy workers and rollout workers. Rollout replicas produce experiences and policy replicas consume them to train. The documented control messages include rollout, evaluation and weight synchronization. AlpaGym uses the framework for autonomous-driving post-training in AlpaSim.
Physical AI example
Pin the AlpaGym-compatible trainer and run one scene to inspect rewards, termination conditions and weight updates. Compare a checkpoint with the original policy on held-out routes.
What to verify
The upstream repository reports limited maintenance and points to the Cosmos 3 ecosystem for new development. Keep working experiment versions pinned and follow the model-specific recipes when migrating.
License and availability
Apache-2.0 repository code. Model weights, external services and dependencies can have separate terms.