Paper 2509.23873
Winning the Pruning Gamble: A Unified Approach to Joint Sample and Token Pruning for Efficient Supervised Fine-Tuning
- Published
- Sep 2025
- Research lab
- Independent
- Citations
- 9
- GitHub
- 3 stars
01 In brief
Summary
The paper introduces Q-Tuning, a unified framework for joint sample- and token-level pruning during supervised fine-tuning (SFT) of large language models.
It addresses the inefficiency of existing methods that treat sample and token pruning separately.
The authors propose the Error–Uncertainty (EU) Plane, which categorizes training samples into four quadrants based on perplexity (error) and entropy (uncertainty): Q1 (harmful noise), Q2 (valuable misconceptions), Q3 (redundant knowledge), and Q4 (calibration data).
Q-Tuning uses a two-stage strategy: first, it prunes Q1 and Q3 at the sample level; second, it applies token-level pruning to Q2 samples while preserving Q4 samples in full.
Experiments across multiple models (LLaMA, Mistral, Qwen, SmolLM2) and benchmarks show that Q-Tuning consistently outperforms full-data training and existing baselines.
Notably, on SmolLM2-1.7B it achieves a +38% average improvement over full-data SFT using only 12.5% of the data, and on LLaMA3-8B with GSM8K it reaches 48.07 using 35% of data, outperforming full-data training by 6.0 points.
The method also demonstrates robustness under extreme data budgets and efficiency gains, reducing training compute to 0.0625× and wall-clock time by ~28%.
02 From the paper
Abstract
As supervised fine-tuning (SFT) evolves from a lightweight post-training step into a compute-intensive phase rivaling mid-training in scale, data efficiency has become critical for aligning large language models (LLMs) under tight budgets. Existing data pruning methods suffer from a fragmented design: they operate either at the sample level or the token level in isolation, failing to jointly optimize both dimensions. This disconnect leads to significant inefficiencies--high-value samples may still contain redundant tokens, while token-level pruning often discards crucial instructional or corrective signals embedded in individual examples. To address this bottleneck, we introduce the Error-Uncertainty (EU) Plane, a diagnostic framework that jointly characterizes the heterogeneous utility of training data across samples and tokens. Guided by this insight, we propose Quadrant-based Tuning (Q-Tuning), a unified framework that strategically coordinates sample pruning and token pruning. Q-Tuning employs a two-stage strategy: first, it performs sample-level triage to retain examples rich in informative misconceptions or calibration signals; second, it applies an asymmetric token-pruning policy, using a context-aware scoring mechanism to trim less salient tokens exclusively from misconception samples while preserving calibration samples in their entirety. Our method sets a new state of the art across five diverse benchmarks. Remarkably, on SmolLM2-1.7B, Q-Tuning achieves a +38\% average improvement over the full-data SFT baseline using only 12.5\% of the original training data. As the first dynamic pruning approach to consistently outperform full-data training, Q-Tuning provides a practical and scalable blueprint for maximizing data utilization in budget-constrained LLM SFT.