Hugging Face’s TRL 1.18 promotes AsyncGRPOTrainer to a stable trainer for asynchronous reinforcement learning, with generation and training able to overlap. The release also changes imports, configuration and metric names, so users of the earlier experimental version have a migration to plan, not merely a new version number to admire.
Hugging Face Watch analysis
What happened
The TRL 1.18.0 release notes describe AsyncGRPOTrainer running generation in a background worker connected to a vLLM server, while the training loop scores queued completions and updates the model. The trainer synchronises updated weights with the server over NCCL. The notes provide an end-to-end example using two B200 GPUs, Qwen3.5-2B and the DeepMath-103K dataset, but explicitly call it a sanity check, not a speed comparison.
The trainer now sits at the top level of the package: import AsyncGRPOTrainer and AsyncGRPOConfig from trl. The older trl.experimental.asyncgrpo import still works with a FutureWarning, but is due to be removed in v2.0.0. The rollout loop is now a public extension point, AsyncRolloutLoop, in trl.trainer.asyncgrpotrainer; the vLLM client is in trl.generation.asyncvllmclient.
Why it matters
AsyncGRPO separates model generation from training updates, allowing the two to overlap instead of taking turns. For teams training with vLLM, the stable API and documented extension point make the workflow easier to build around. The example is useful for understanding how the pieces fit, but it is not evidence that this approach is faster or cheaper on other models and hardware.
The upgrade has practical consequences for existing users. Several old configuration options and constructor arguments have been removed or renamed, and metric names have changed. Dashboards and scripts that consume the old metrics may need updating; teams using the old experimental import can move to the top-level path before the planned v2.0.0 removal.
Our read
This is a meaningful developer release: it turns an experimental training workflow into a stable API and explains how to extend it. The best immediate move for current users is to check their imports, configuration, rollout customisations and metric consumers against the notes before upgrading. A stable label is welcome; a quietly broken dashboard is less so.
What to watch
- Whether the stable trainer’s API and documented extension points remain consistent in later releases.
- How the workflow performs on workloads beyond the release notes’ example, which is not a speed benchmark.
- Whether users update old imports, configuration and dashboards before the experimental path is removed in v2.0.0.
Discussion spark: For teams already using experimental AsyncGRPO, is a stable API worth migrating now, or would you wait until the v2.0.0 removal deadline forces the issue?
Sources and evidence
- v1.18.0 (13 November 2026, 15:37 UTC)
not affiliated with or endorsed by Hugging Face