Paper 2512.23447
Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Loss
- Published
- Dec 2025
- Research lab
- Independent
- Citations
- 9
- GitHub
- Not linked
01 In brief
Summary
The paper introduces the expert-router coupling (ERC) loss, a lightweight auxiliary loss for Mixture-of-Experts (MoE) models that aligns router decisions with expert capabilities.
The method treats each router embedding as a proxy token for the tokens routed to that expert, perturbs it with bounded noise, and feeds it through all experts to compute activation norms.
The loss enforces that each expert responds most strongly to its own proxy token and that each proxy token activates its corresponding expert more than others.
This coupling improves expert specialization and routing accuracy.
The ERC loss adds only 2n²Dd FLOPs, independent of batch size, unlike prior methods that scale with token count.
Experiments on MoE-LLMs from 3B to 15B parameters, trained on trillions of tokens, show consistent performance gains over vanilla MoE and competitive results with the more expensive Autonomy-of-Experts (AoE) baseline, while adding only 0.2–0.8% training overhead.
The loss also enables quantitative control and tracking of expert specialization via hyperparameters α and noise level ϵ, revealing a trade-off between specialization and performance.
Ablations confirm the importance of the noise perturbation and show that the loss outperforms router orthogonalization.
02 From the paper
Abstract
Mixture-of-Experts (MoE) models lack explicit constraints to ensure the router's decisions align well with the experts' capabilities, which ultimately limits model performance. To address this, we propose expert-router coupling (ERC) loss, a lightweight auxiliary loss that tightly couples the router's decisions with expert capabilities. Our approach treats each expert's router embedding as a proxy token for the tokens assigned to that expert, and feeds perturbed router embeddings through the experts to obtain intermediate activations. The ERC loss enforces two constraints on these activations: (1) Each expert must exhibit higher activation for its own proxy token than for the proxy tokens of any other expert. (2) Each proxy token must elicit stronger activation from its corresponding expert than from any other expert. These constraints jointly ensure that each router embedding faithfully represents its corresponding expert's capability, while each expert specializes in processing the tokens actually routed to it. The ERC loss is computationally efficient, operating only on $n^2$ activations, where $n$ is the number of experts. This represents a fixed cost independent of batch size, unlike prior coupling methods that scale with the number of tokens (often millions per batch). Through pre-training MoE-LLMs ranging from 3B to 15B parameters and extensive analysis on trillions of tokens, we demonstrate the effectiveness of the ERC loss. Moreover, the ERC loss offers flexible control and quantitative tracking of expert specialization levels during training, providing valuable insights into MoEs.