Paper 2510.11696
QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMs
01 In brief
Summary
QeRL is a framework that combines NVFP4 4-bit quantization with Low-Rank Adaptation (LoRA) to accelerate and reduce memory usage in reinforcement learning (RL) for large language models (LLMs).
The authors find that quantization noise increases policy entropy, which enhances exploration during RL, contrary to its detrimental effect in supervised fine-tuning.
To make this noise adaptive, they introduce Adaptive Quantization Noise (AQN), which injects channel-wise Gaussian noise via layer normalization with an exponential decay schedule.
Experiments on Qwen2.5 models (3B to 32B) using GRPO and DAPO on GSM8K and BigMath show that QeRL achieves over 1.5× rollout speedup, enables RL training of a 32B model on a single H100 80GB GPU, and matches or exceeds the performance of 16-bit LoRA and QLoRA, approaching full-parameter fine-tuning accuracy (e.g., GSM8K 90.8% and MATH 500 77.4% for 7B).
QeRL also reduces model memory to 25–30% of 16-bit counterparts and provides 1.2–2.0× end-to-end training speedups, with larger gains on larger models.
The framework is open-sourced at https://github.com/NVlabs/QeRL.
02 From the paper
Abstract
We propose QeRL, a Quantization-enhanced Reinforcement Learning framework for large language models (LLMs). While RL is essential for LLMs' reasoning capabilities, it is resource-intensive, requiring substantial GPU memory and long rollout durations. QeRL addresses these issues by combining NVFP4 quantization with Low-Rank Adaptation (LoRA), accelerating rollout phase of RL while reducing memory overhead. Beyond efficiency, our findings show that quantization noise increases policy entropy, enhancing exploration, and enabling the discovery of better strategies during RL. To further optimize exploration, QeRL introduces an Adaptive Quantization Noise (AQN) mechanism, which dynamically adjusts noise during training. Experiments demonstrate that QeRL delivers over 1.5 times speedup in the rollout phase. Moreover, this is the first framework to enable RL training of a 32B LLM on a single H100 80GB GPU, while delivering overall speedups for RL training. It also achieves faster reward growth and higher final accuracy than 16-bit LoRA and QLoRA, while matching the performance of full-parameter fine-tuning on mathematical benchmarks such as GSM8K (90.8%) and MATH 500 (77.4%) in the 7B model. These results establish QeRL as an efficient and effective framework for RL training in LLMs.