Paper 2510.15511
Language Models are Injective and Hence Invertible
- Published
- Oct 2025
- Research lab
- Independent
- Citations
- 31
- GitHub
- 26 stars
01 In brief
Summary
This paper proves that decoder-only Transformer language models are almost surely injective: distinct input prompts map to distinct last-token hidden representations, both at initialization and after any finite number of gradient descent steps.
The authors establish this by showing the model is a real-analytic function of its parameters, so collisions can only occur on a measure-zero parameter set, which standard initialization and training avoid.
They confirm this empirically with billions of collision tests on models like GPT-2, Gemma, Llama, and Mistral, finding no collisions.
They also introduce SIPIT, an algorithm that exploits injectivity to exactly reconstruct input text from hidden states in provable linear time, achieving 100% accuracy in experiments.
The work has implications for interpretability, privacy, and data protection, as hidden states are lossless encodings of user input.
02 From the paper
Abstract
Transformer components such as non-linear activations and normalization are inherently non-injective, suggesting that different inputs could map to the same output and prevent exact recovery of the input from a model's representations. In this paper, we challenge this view. First, we prove mathematically that transformer language models mapping discrete input sequences to their corresponding sequence of continuous representations are injective and therefore lossless, a property established at initialization and preserved during training. Second, we confirm this result empirically through billions of collision tests on six state-of-the-art language models, and observe no collisions. Third, we operationalize injectivity: we introduce SipIt, the first algorithm that provably and efficiently reconstructs the exact input text from hidden activations, establishing linear-time guarantees and demonstrating exact invertibility in practice. Overall, our work establishes injectivity as a fundamental and exploitable property of language models, with direct implications for transparency, interpretability, and safe deployment.