Paper 2511.02237
Opportunistic Expert Activation: Batch-Aware Expert Routing for Faster Decode Without Retraining
- Published
- Nov 2025
- Research lab
- Together AI
- Citations
- 4
- GitHub
- Not linked
01 In brief
Summary
The paper introduces Opportunistic Expert Activation (OEA), a batch-aware routing framework that reduces Mixture-of-Experts (MoE) decode latency without retraining.
During autoregressive generation, MoE models become memory-bound at moderate batch sizes because latency scales with the number of unique activated experts.
OEA operates in two phases: first, it selects a baseline set of top experts per token (using a hyperparameter k0) to guarantee quality; second, it opportunistically piggybacks additional experts that are already loaded for other tokens in the batch, recovering performance at no extra latency cost.
Evaluations on Qwen3-30B-A3B and Qwen3-235B-A22B at batch size 16 show latency reductions of 39% and 15% respectively, without statistically significant accuracy loss on benchmarks like AIME24, GPQA, LiveCodeBench, and MATH500.
The simplified algorithm uses only k0, with k0=3 halving the number of activated experts.
The paper also confirms a linear relationship between latency and the number of activated experts (R²>0.99).
OEA is complementary to other expert-reduction methods and provides a tunable quality-performance trade-off.
Future work includes batch-size-dependent k0 selection and layer-specific hyperparameters.
The method is implemented in SGLang and tested on H100 GPUs with tensor parallelism for the larger model.
The authors note that padding tokens can increase latency and suggest masking them.
Overall, OEA offers a practical, training-free solution to improve MoE inference efficiency.
02 From the paper
Abstract
An increasing number of LLMs employ Mixture-of-Experts (MoE) architectures where the feed-forward layer is replaced by a pool of experts and each token only activates a small subset of them. During autoregressive generation, these models often enter a memory-bound regime even for moderate batch sizes because the average expert load grows more slowly than in an equivalent dense feedforward layer. Consequently, MoE latency is governed by the number of activated experts. We introduce a framework for dynamically re-routing token-to-expert mapping to lower this number (and thus, the decode latency) while preserving a comparable quality. Our best results use a batch-aware routing that works by having tokens piggyback experts that have already been loaded into memory due to being crucial to other tokens within the same batch. Empirically, we evaluate our method on the Qwen3-30B and Qwen3-235B models with a batch size of $16$. Without any statistically significant loss in accuracy, our approach achieves latency reductions of $39\%$ and $15\%$ in the MoE layer decode latency, respectively.