Paper 2606.07502
Your UnEmbedding Matrix is Secretly a Feature Lens for Text Embeddings
- Published
- Jun 2026
- Research lab
- Independent
- Citations
- 0
- GitHub
- 25 stars
01 In brief
Summary
The paper identifies a cause of LLMs' suboptimal zero-shot text embedding performance: text embeddings align with high-frequency but uninformative tokens when projected onto the vocabulary space.
Using Logit Lens and Logit Spectroscopy, the authors discover an 'edge spectrum' subspace in the unembedding matrix that encodes these frequent tokens.
They propose EmbedFilter, a simple linear transformation that filters out this subspace, improving semantic representations.
EmbedFilter also enables inherent dimensionality reduction, lowering storage and speeding up retrieval.
Experiments on MTEB across Qwen, Llama, and Mistral backbones show up to 14.1% improvement over baselines, with gains maintained even at 1/8 of original dimensions.
Ablations confirm the improvement is not due to dimensionality reduction alone and that filtering the edge spectrum is optimal.
EmbedFilter outperforms whitening without needing calibration data.
The work provides a mechanistic interpretation of LLM embedding limitations and a practical post-processing solution.
02 From the paper
Abstract
Large language models exhibit impressive zero-shot capabilities across a wide range of downstream tasks. However, they struggle to function as off-the-shelf embedding models, leading to suboptimal performance on massive text embedding benchmarks. In this paper, we identify a potential cause underlying this deficiency. Our motivation stems from an unexpected observation: text embeddings tend to align with frequent but uninformative tokens when projected onto the vocabulary space. We argue that this excessive expression of high-frequency tokens suppresses the model's ability to capture nuanced semantics. To address this, we introduce EmbedFilter, a simple linear transformation designed to refine text embeddings derived from LLMs directly. Specifically, we uncover that the unembedding matrix within LLMs encodes a latent space that is actively writing these frequent tokens into embedding space. By filtering out this subspace, EmbedFilter suppress the influence of high-frequency tokens, thereby enhancing semantic representations. As a compelling byproduct, this enables an inherent dimensionality reduction, lowering index storage and speedup retrieval while fully preserving the refined embedding quality. Our experiments across multiple LLM backbones demonstrate that LLMs equipped with EmbedFilter achieve superior zero-shot downstream performance even with significantly reduced embedding dimensions. We hope our findings provide deeper insights into the mechanisms of LLM-based representations and inspire more principled designs to improve text embeddings training. Our code is available at https://github.com/CentreChen/EmbFilter.