Bottom line: Direct-OPD transfers RL-induced policy shifts from weaker to stronger models by leveraging the implicit reward signal from the log-ratio of the RL-shifted and original policy.
Reinforcement learning on large language models is computationally intensive. A new approach called Direct On-Policy Distillation (Direct-OPD) enables transferring RL progress from smaller models to more powerful ones — without repeating expensive RL training runs on the larger model.
The problem: RL with verifiable rewards (RLVR) improves the reasoning of language models, but becomes increasingly expensive for larger models — the reason being the many rollouts required during training. Every new, stronger model requires a separate, costly RL run.
Direct-OPD addresses this through a weak-to-strong approach: instead of running RL directly on the large model, it is first performed on a smaller model (significantly cheaper). Then, it learns how this small model has changed. The central mechanism: comparing the RL-trained policy of the small model with its pre-RL version. The log-ratio of these two checkpoints is used as a dense implicit reward signal for the stronger student model. Specifically: the checkpoint pair shows which actions RL training made more likely and which less likely. This information is then applied to the on-policy states of the larger model.
Empirically, Direct-OPD improves Qwen3-1.7B on AIME 2024 from 48.3% to 58.3% in 4 hours on 8 A100 GPUs — a 10-point gain without direct RL on the large model. The approach outperforms step-matched direct RL and allows sequential composition of multiple policy shifts.
For engineers, this means: RL insights can be recycled as implicit reward signals across model scales, not just as final models to imitate. This significantly reduces compute costs for post-training on larger models and opens new optimization strategies for inference scaling.
Source: arxiv.org · Published July 7, 2026
Lumi AI News — AI-assisted curation in accordance with Article 50 EU AI Act. Paraphrase and classification by Lumi News Pipeline v1.7.3.