Paper 2606.20945
Grouped Query Experts: Mixture-of-Experts on GQA Self-Attention
- Published
- Jun 2026
- Research lab
- Independent
- Citations
- 0
- GitHub
- Not linked
01 In brief
Summary
The paper introduces Grouped Query Experts (GQE), a mixture-of-experts layer applied to grouped-query attention (GQA) in Transformers.
GQE routes each token to k query-head experts within each GQA group, while keeping all key-value (KV) heads dense and unchanged, preserving GQA's KV-cache benefits and reducing active query-head computation.
The method includes a renormalized weighted-sum slot for router learning and an always-on shared head for stability.
On a fixed 30B-token budget at 250M parameters, GQE matches the all-active GQA baseline in downstream accuracy (56.04 vs 55.86 average on HellaSwag, ARC-Easy, PIQA) while activating half of the routed query-head experts (9 of 16 total query-attention computations including the shared head).
GQE achieves 1.7–1.8× prefill speedup at long context lengths (4k to 1024k tokens), with modest speedup at 2k tokens (1.15×).
Ablations show that without the renormalized slot and shared head, accuracy drops below baseline, confirming their necessity.
The method is limited to 250M scale and 30B tokens, with future work planned for larger scales and comparisons to other architectures.
02 From the paper
Abstract
Self-attention is central to Transformer performance and is often the most expensive part of the Transformer at long context lengths because its pairwise token interactions scale quadratically with sequence length. Standard dense attention also applies the same set of attention heads to every token regardless of token difficulty or information content. This uniform activation can waste compute, especially as sequences grow longer and attention cost increases rapidly. We propose Grouped Query Experts (GQE), a mixture-of-experts layer on top of grouped-query attention (GQA). Within each GQA group, a router selects k query-head experts per token while all key-value (KV) heads remain dense and unchanged. Thus, GQE keeps the KV cache benefits of GQA and reduces only the active query-head computation. On a fixed 30B token budget at the 250M parameter scale, GQE matches the all-active GQA baseline in downstream accuracy while activating half the query heads per token.