Paper 2602.10388
Less is Enough: Synthesizing Diverse Data in LLM Feature Space with Sparse Autoencoders
- Published
- Feb 2026
- Research lab
- Independent
- Citations
- 2
- GitHub
- 111 stars
01 In brief
Summary
The paper introduces Feature Activation Coverage (FAC), a metric that measures data diversity in an interpretable feature space derived from Sparse Autoencoders (SAEs) trained on LLM internal activations.
The authors propose FAC Synthesis, a framework that identifies task-relevant features missing from a seed dataset and generates synthetic samples to activate them, using contrastive pairs to reduce sampling error.
Theoretically, they bound post-training generalization error by a distribution gap and a sampling error, both reducible via feature coverage.
Experiments on toxicity detection, reward modeling, behavior steering, and instruction following show FAC strongly correlates with downstream performance (Pearson r=0.95) and that FAC Synthesis outperforms baselines, achieving comparable AlpacaEval 2.0 win rates to MAGPIE with 150x less data (2K vs 300K samples).
The method transfers across model families (LLaMA, Mistral, Qwen), with LLaMA-sourced features benefiting stronger models.
Hyperparameter analysis shows intermediate temperatures and thresholds around 1.0-2.0 are optimal, and data efficiency decreases with more samples per feature.
Limitations include smaller gains on reasoning-heavy tasks like GSM8K and LiveCodeBench, suggesting single-layer SAE features may be insufficient for multi-layer reasoning circuits.
02 From the paper
Abstract
The diversity of post-training data is critical for effective downstream performance in large language models (LLMs). Many existing approaches to constructing post-training data quantify diversity using text-based metrics that capture linguistic variation, but such metrics provide only weak signals for the task-relevant features that determine downstream performance. In this work, we introduce Feature Activation Coverage (FAC) which measures data diversity in an interpretable feature space. Building upon this metric, we further propose a diversity-driven data synthesis framework, named FAC Synthesis, that first uses a sparse autoencoder to identify missing features from a seed dataset, and then generates synthetic samples that explicitly reflect these features. Experiments show that our approach consistently improves both data diversity and downstream performance on various tasks, including instruction following, toxicity detection, reward modeling, and behavior steering. Interestingly, we identify a shared, interpretable feature space across model families (i.e., LLaMA, Mistral, and Qwen), enabling cross-model knowledge transfer. Our work provides a solid and practical methodology for exploring data-centric optimization of LLMs.