Paper 2509.22637
Variational Reasoning for Language Models
- Published
- Sep 2025
- Research lab
- Independent
- Citations
- 6
- GitHub
- 60 stars
01 In brief
Summary
The paper introduces a variational reasoning framework for language models, treating thinking traces as latent variables and optimizing them via variational inference.
It extends the evidence lower bound (ELBO) to a multi-trace IWAE-style objective and proposes a forward-KL formulation for stable training of the variational posterior.
The authors show that rejection sampling finetuning (RFT) and binary-reward RL methods like GRPO can be interpreted as local forward-KL objectives with an implicit weighting by model accuracy, revealing a bias toward easier questions.
They validate their method on Qwen2.5 and Qwen3 model families across reasoning benchmarks (MATH500, AIME24/25, OlympiadBench, LiveCodeBench, GPQA-Diamond, MMLU-Pro), consistently outperforming baselines.
Ablations confirm the importance of conditioning on answer hints, the accuracy-based estimator for π_θ(Y_x|x,z), and scaling the number of thinking traces.
The framework provides a principled probabilistic perspective unifying variational inference with RL-style methods.
02 From the paper
Abstract
We introduce a variational reasoning framework for language models that treats thinking traces as latent variables and optimizes them through variational inference. Starting from the evidence lower bound (ELBO), we extend it to a multi-trace objective for tighter bounds and propose a forward-KL formulation that stabilizes the training of the variational posterior. We further show that rejection sampling finetuning and binary-reward RL, including GRPO, can be interpreted as local forward-KL objectives, where an implicit weighting by model accuracy naturally arises from the derivation and reveals a previously unnoticed bias toward easier questions. We empirically validate our method on the Qwen 2.5 and Qwen 3 model families across a wide range of reasoning tasks. Overall, our work provides a principled probabilistic perspective that unifies variational inference with RL-style methods and yields stable objectives for improving the reasoning ability of language models. Our code is available at https://github.com/sail-sg/variational-reasoning.