The Year in AI Papers
The year/NVIDIA

Paper 2601.22975

Golden Goose: A Simple Trick to Synthesize Unlimited RLVR Tasks from Unverifiable Internet Text

Ximing Lu, David Acuna, Jaehun Jung, Jian Hu, Di Zhang, Shizhe Diao, Yunheng Zou, Shaokun Zhang, Brandon Cui, Mingjie Liu, Hyunwoo Kim, Prithviraj Ammanabrolu, et al.

Published
Jan 2026
Research lab
NVIDIA
Citations
6
GitHub
Not linked

01 In brief

Summary

The paper introduces Golden Goose, a method to synthesize unlimited Reinforcement Learning with Verifiable Rewards (RLVR) tasks from unverifiable internet text by converting them into multiple-choice fill-in-the-middle questions.

Given a source text, an LLM masks a contiguous span of crucial reasoning steps (the ground-truth answer) and generates diverse, plausible distractors.

This enables using reasoning-rich corpora previously excluded from RLVR, such as AoPS-Instruct (math), rStar-Coder (code without test cases), and MegaScience (STEM textbooks), to create GooseReason-0.7M, a dataset of over 0.7 million tasks.

Empirically, GooseReason revives models saturated on existing RLVR data, yielding sustained gains under continued RL.

For ProRL-1.5B-v2, adding GooseReason-0.7M improved math by 2.71%, coding by 2.12%, and STEM by 3.48% over 1,100 H100 GPU hours, versus 0.63%, 0.95%, and 0.13% with original data.

For Qwen3-4B-Instruct, which plateaued after 300 steps, GooseReason enabled absolute gains of 2.18% (math), 2.24% (coding), and 2.40% (STEM), achieving new state-of-the-art among 4B-Instruct models across 15 benchmarks.

The method also synthesizes RLVR data for cybersecurity from FineWeb scrapes (GooseReason-Cyber, 180K tasks), where training Qwen3-4B-Instruct for 100 RL steps yielded a 4.44% absolute gain across 3 benchmarks, surpassing a 7B domain-specialized model.

The design uses 9-choice MCQ format to maintain medium difficulty, as open-ended formats suffer from poor instruction following.

02 From the paper

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has become a cornerstone for unlocking complex reasoning in Large Language Models (LLMs). Yet, scaling up RL is bottlenecked by limited existing verifiable data, where improvements increasingly saturate over prolonged training. To overcome this, we propose Golden Goose, a simple trick to synthesize unlimited RLVR tasks from unverifiable internet text by constructing a multiple-choice question-answering version of the fill-in-the-middle task. Given a source text, we prompt an LLM to identify and mask key reasoning steps, then generate a set of diverse, plausible distractors. This enables us to leverage reasoning-rich unverifiable corpora typically excluded from prior RLVR data construction (e.g., science textbooks) to synthesize GooseReason-0.7M, a large-scale RLVR dataset with over 0.7 million tasks spanning mathematics, programming, and general scientific domains. Empirically, GooseReason effectively revives models saturated on existing RLVR data, yielding robust, sustained gains under continuous RL and achieving new state-of-the-art results for 1.5B and 4B-Instruct models across 15 diverse benchmarks. Finally, we deploy Golden Goose in a real-world setting, synthesizing RLVR tasks from raw FineWeb scrapes for the cybersecurity domain, where no prior RLVR data exists. Training Qwen3-4B-Instruct on the resulting data GooseReason-Cyber sets a new state-of-the-art in cybersecurity, surpassing a 7B domain-specialized model with extensive domain-specific pre-training and post-training. This highlights the potential of automatically scaling up RLVR data by exploiting abundant, reasoning-rich, unverifiable internet text.