At a glance: Aligning router rows with the principal singular directions of their associated expert matrices improves the efficiency and stability of Mixture-of-Experts models.
Researchers propose a new design principle for routers in Mixture-of-Experts models that aligns router rows with the principal singular directions of their respective expert matrices. The approach, called Manifold Power Iteration, aims to make the selection of activated experts more efficient and stable.
In Mixture-of-Experts (MoE) models, the router serves as a key component for selecting activated experts. The rows of the router matrix compute their similarity to MoE inputs to determine which experts are processed. Ideally, each router row should condense the associated expert matrix into a representative vector whose dot product with a token better captures token-expert affinity. However, there have been no systematic design principles to enforce this condensation.
The proposed solution Manifold Power Iteration (MPI) is based on the principle of aligning each router row with the principal singular direction of the associated expert matrix. This principal mathematical direction provides the most informative description of a matrix. MPI performs a so-called “power-then-retract” process: a power iteration step is applied to the router weights, followed by a retraction to enforce a norm constraint. This ensures that both computational efficiency and numerical stability are preserved.
Theoretically, it can be shown that MPI causes router rows to converge to the principal singular directions of their associated experts. Empirical pretraining experiments across model sizes from 1 billion to 11 billion parameters confirm that this alignment leads to more effective MoE models. The results suggest that optimizing router architecture remains a productive area for further development in large language models.
Source: arxiv.org · Published June 9, 2026
Lumi AI News — AI-assisted curation in accordance with Article 50 EU AI Act. Paraphrase and classification by Lumi News Pipeline v1.6.5.