Key takeaway: A training-inference mismatch in LLM-RL leads to persistent off-policyness; MIPU resolves this through selective acceptance of policy updates based on inference-side improvements.
Researchers identify a fundamental problem in RL training of large language models: because training and inference run on different engines, this leads to inconsistent probabilities for the same sequences and thus to persistent offline trainability, which jeopardizes stability.
Reinforcement learning in LLM post-training is fragile and prone to instability or complete collapse. A central reason is the training-inference mismatch: LLMs use separate engines for generation (efficiency) and training (accuracy), which leads to contradictory probability outputs for identical trajectories – even when model parameters are synchronized. This creates a permanent form of off-policyness that poisons the training.
Previous work has attempted to address this off-policyness and stabilize training policies. However, the new study highlights an overlooked problem: effective policy updates in the training engine do not guarantee improvement of the inference policy – that is, the policy that actually runs in production. This is an objective misalignment.
The authors propose Monotonic Inference Policy Improvement (MIPI) as a new optimization objective and implement it in the framework Monotonic Inference Policy Update (MIPU). MIPU operates in two steps: it constructs sampler-referenced candidate updates and selectively accepts them through a proxy for the inference-side gap. Experiments on two model sizes under high mismatch show improvements in average reasoning performance and training stability.
Source: arxiv.org · Published June 27, 2026
Lumi AI News — AI-assisted curation pursuant to Article 50 of the EU AI Act. Paraphrase and classification by Lumi News Pipeline v1.7.3.