Paper 2511.13653
Weight-sparse transformers have interpretable circuits
- Published
- Nov 2025
- Research lab
- OpenAI
- Citations
- 37
- GitHub
- Not linked
01 In brief
Summary
The paper introduces weight-sparse transformers, where most weights are zero, to improve mechanistic interpretability.
By constraining the L0 norm, models learn disentangled circuits for tasks, which are pruned to isolate minimal circuits.
These circuits are compact and contain neurons and residual channels corresponding to natural concepts, with interpretable connections.
The method yields roughly 16-fold smaller circuits than dense models at comparable loss, and scaling model size improves the capability-interpretability frontier, though scaling beyond tens of millions of nonzero parameters remains challenging.
The authors validate circuits via mean ablation, showing they are necessary and sufficient for task performance.
They also present preliminary results using bridges to adapt the method to explain existing dense models.
Qualitative analysis of three tasks (closing strings, counting nesting depth, tracking variable types) reveals human-understandable algorithms, and adversarial attacks derived from circuit understanding generalize to dense models.
Limitations include computational inefficiency, polysemantic features, and the need for better faithfulness measures.
02 From the paper
Abstract
Finding human-understandable circuits in language models is a central goal of the field of mechanistic interpretability. We train models to have more understandable circuits by constraining most of their weights to be zeros, so that each neuron only has a few connections. To recover fine-grained circuits underlying each of several hand-crafted tasks, we prune the models to isolate the part responsible for the task. These circuits often contain neurons and residual channels that correspond to natural concepts, with a small number of straightforwardly interpretable connections between them. We study how these models scale and find that making weights sparser trades off capability for interpretability, and scaling model size improves the capability-interpretability frontier. However, scaling sparse models beyond tens of millions of nonzero parameters while preserving interpretability remains a challenge. In addition to training weight-sparse models de novo, we show preliminary results suggesting our method can also be adapted to explain existing dense models. Our work produces circuits that achieve an unprecedented level of human understandability and validates them with considerable rigor.