Paper 2510.26697
The End of Manual Decoding: Towards Truly End-to-End Language Models
- Published
- Oct 2025
- Research lab
- Independent
- Citations
- 6
- GitHub
- 73 stars
01 In brief
Summary
The paper introduces AutoDeco, a novel architecture that makes LLM generation truly end-to-end by learning to predict token-level temperature and top-p values during decoding.
This replaces manual hyperparameter tuning with a dynamic, self-regulating process.
AutoDeco uses lightweight heads on top of frozen base models, trained with a differentiable soft top-p mechanism and a cross-entropy loss, requiring only 400 steps and about 6K samples.
Experiments on eight benchmarks across four model families (Llama, Qwen, GPT-OSS) show AutoDeco consistently outperforms greedy search and default sampling, and matches or exceeds an oracle-tuned static baseline.
It also exhibits an emergent ability to follow natural language commands (e.g., 'generate with low randomness') to adjust its sampling parameters, achieving 95%+ consistency after targeted training.
The method adds only 1-2% latency and negligible memory overhead.
The paper also releases AutoDeco heads for larger models like Deepseek-V3.1-Terminus, though without full evaluation due to computational cost.
Future work includes joint training of the base model and heads for finer control.
02 From the paper
Abstract
The "end-to-end" label for LLMs is a misnomer. In practice, they depend on a non-differentiable decoding process that requires laborious, hand-tuning of hyperparameters like temperature and top-p. This paper introduces AutoDeco, a novel architecture that enables truly "end-to-end" generation by learning to control its own decoding strategy. We augment the standard transformer with lightweight heads that, at each step, dynamically predict context-specific temperature and top-p values alongside the next-token logits. This approach transforms decoding into a parametric, token-level process, allowing the model to self-regulate its sampling strategy within a single forward pass. Through extensive experiments on eight benchmarks, we demonstrate that AutoDeco not only significantly outperforms default decoding strategies but also achieves performance comparable to an oracle-tuned baseline derived from "hacking the test set"-a practical upper bound for any static method. Crucially, we uncover an emergent capability for instruction-based decoding control: the model learns to interpret natural language commands (e.g., "generate with low randomness") and adjusts its predicted temperature and top-p on a token-by-token basis, opening a new paradigm for steerable and interactive LLM decoding.