Key takeaway: SEED leverages self-generated hindsight supervision from language model-native trajectory analysis to bridge the supervision gap between episode-level outcomes and token-level learning signals.
Researchers have developed SEED, a framework that trains large language models more efficiently in reinforcement learning for complex multi-turn tasks. The method automatically extracts capabilities in natural language from completed trajectories and uses them for direct, token-based optimization.
The core challenge in outcome-based reinforcement learning (RL) for interactive agents lies in sparse rewards at the trajectory level: they provide only limited guidance for individual intermediate steps in a multi-turn dialogue, tool use, or environment interaction. This creates a supervision gap between the overall success of an entire episode and the token-by-token learning process of the model.
SEED (Self-Evolving On-Policy Distillation) addresses this through a self-evolving design: the model first analyzes its own completed trajectories to extract natural-language skills — reusable workflows, critical observations, or error-avoidance rules. During training, the same policy simultaneously collects new trajectories and serves as an analyzer for this data. This enables tight coupling: policy updates improve both subsequent decisions and skill extraction itself, while hindsight supervision evolves with the changing policy.
In the second step, the approach re-evaluates the conditioned actions in both normal and skill-augmented contexts. The resulting probability shift is converted into a dense, token-level distillation signal. This signal is jointly optimized with the outcome-based RL objective, keeping additional supervision aligned to the current trajectory distribution and preventing drift to stale data distributions.
Experimental results on text- and vision-based agent-based tasks show consistent improvements in performance and sample efficiency. The framework also demonstrates robust generalization to unseen scenarios. Source code is available on GitHub.
Source: arxiv.org · Published 15 July 2026
Lumi AI News — AI-assisted curation pursuant to Art. 50 EU AI Act. Paraphrase and classification by Lumi News Pipeline v1.7.3.