In brief: Single-rollout sampling instead of batch sampling stabilizes asynchronous RL training and outperforms GRPO on agent-based benchmarks.
Researchers have introduced Single-rollout Asynchronous Optimization (SAO), a method for stable asynchronous optimization in reinforcement learning of large language models. SAO replaces the batch sampling strategy with single-rollout sampling, thereby addressing stability and off-policy issues in agent-based training pipelines.
Synchronous batch RL pipelines for post-training of Large Language Models show inefficiencies in long-horizon tasks, where agents must act autonomously across multiple steps. Asynchronous RL systems address this problem through live updates upon arrival of rollouts, but often neglect training stability in favor of throughput. The widely used GRPO framework employs batch sampling — an approach that does not naturally fit asynchronous agent-based training.
The SAO method instead relies on single-rollout sampling: one rollout per prompt rather than multiple. This reduces off-policy effects and improves generalization. This is complemented by practical value-model training practices and a strict two-sided token-level clipping strategy to stabilize optimization. The method trains stably over thousands of steps.
On benchmarks such as SWE-Bench Verified, BeyondAIME, and IMOAnswerBench for coding and reasoning, SAO consistently outperforms GRPO and its variants. The single-rollout strategy proves particularly effective in simulated online-learning settings, where the model must adapt to changing environments. SAO has already been successfully deployed in the agent-based RL pipeline for the 750B-parameter model GLM-5.2 (open-source release).
Source: arxiv.org · Published July 7, 2026
Lumi AI News — AI-assisted curation pursuant to Article 50 EU AI Act. Paraphrase and classification by Lumi News Pipeline v1.7.3.