Paper 2603.15569
Mamba-3: Improved Sequence Modeling using State Space Principles
- Published
- Mar 2026
- Research lab
- Together AI
- Citations
- 68
- GitHub
- Not linked
01 In brief
Summary
Mamba-3 introduces three methodological improvements to state space models (SSMs) for better quality, capability, and inference efficiency.
First, exponential-trapezoidal discretization generalizes prior heuristics, enabling a more expressive recurrence that can replace the short causal convolution.
Second, complex-valued state transitions, implemented via a data-dependent RoPE trick, enable state-tracking tasks like parity and modular arithmetic that prior linear models fail.
Third, a multi-input, multi-output (MIMO) formulation increases decoding FLOPs and arithmetic intensity without raising state size or latency.
At 1.5B scale, Mamba-3 (SISO) improves average downstream accuracy by 0.6 points over Gated DeltaNet, and Mamba-3 (MIMO) adds another 1.2 points.
Mamba-3 with state size 64 matches Mamba-2's perplexity with state size 128, halving latency.
The model also shows strong length extrapolation and competitive retrieval, with hybrid variants improving NIAH performance.
Fast Triton and CuTe kernels are released, showing minimal overhead and competitive decode latency compared to Mamba-2 and GDN.
These advances push the performance-efficiency Pareto frontier for sub-quadratic models, addressing limitations in state tracking and hardware utilization during inference.
The work is grounded in an SSM-centric perspective, offering theoretically motivated improvements over prior linear attention and test-time training frameworks.
Future work may explore further architectural refinements and broader applications of these techniques to other recurrent models.
The authors acknowledge support from Schmidt Sciences, Google, Princeton, Together AI, and Cartesia AI.
The paper includes extensive ablations and benchmarks supporting the claims.
Overall, Mamba-3 demonstrates that principled SSM design can yield both better quality and efficiency, making it a strong candidate for deployment in…
02 From the paper
Abstract
Scaling inference-time compute has emerged as an important driver of LLM performance, making inference efficiency a central focus of model design alongside model quality. While the current Transformer-based models deliver strong model quality, their quadratic compute and linear memory make inference expensive. This has spurred the development of sub-quadratic models with reduced linear compute and constant memory requirements. However, many recent linear models trade off model quality and capability for algorithmic efficiency, failing on tasks such as state tracking. Moreover, their theoretically linear inference remains hardware-inefficient in practice. Guided by an inference-first perspective, we introduce three core methodological improvements inspired by the state space model (SSM) viewpoint of linear models. We combine: (1) a more expressive recurrence derived from SSM discretization, (2) a complex-valued state update rule that enables richer state tracking, and (3) a multi-input, multi-output (MIMO) formulation for better model performance without increasing decode latency. Together with architectural refinements, our Mamba-3 model achieves significant gains across retrieval, state-tracking, and downstream language modeling tasks. At the 1.5B scale, Mamba-3 improves average downstream accuracy by 0.6 percentage points compared to the next best model (Gated DeltaNet), with Mamba-3's MIMO variant further improving accuracy by another 1.2 points for a total 1.8 point gain. Across state-size experiments, Mamba-3 achieves comparable perplexity to Mamba-2 despite using half of its predecessor's state size. Our evaluations demonstrate Mamba-3's ability to advance the performance-efficiency Pareto frontier.