The Year in AI Papers
The year/Independent research

Paper 2508.10975

BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining

DatologyAI, :, Pratyush Maini, Vineeth Dorna, Parth Doshi, Aldo Carranza, Fan Pan, Jack Urbanek, Paul Burstein, Alex Fang, Alvin Deng, Amro Abbas, et al.

Published
Aug 2025
Research lab
Independent
Citations
16
GitHub
Not linked

01 In brief

Summary

BeyondWeb is a synthetic data generation framework for LLM pretraining that outperforms existing synthetic datasets like Cosmopedia and Nemotron-Synth by up to 5.1 and 2.6 percentage points, respectively, across 14 benchmarks.

It achieves up to 7.7x faster training than open web data and 2.7x faster than Nemotron-Synth.

A 3B model trained on BeyondWeb for 180B tokens outperforms an 8B model trained on Cosmopedia with the same token budget.

The paper systematically evaluates synthetic data, finding that simple summarization can match generator-driven approaches, but thoughtful rephrasing yields larger gains.

It shows that synthetic data can break the data wall if well-designed, that high-quality seed data matters more than novelty, that style matching (e.g., conversational data) helps but saturates, and that diversity in generation strategies is critical for scaling to trillions of tokens.

Rephraser model family has little impact on synthetic data quality, and gains saturate beyond 3B rephraser size.

The authors conclude that no single factor is sufficient; high-quality synthetic data requires jointly optimizing many variables.

02 From the paper

Abstract

Recent advances in large language model (LLM) pretraining have shown that simply scaling data quantity eventually leads to diminishing returns, hitting a data wall. In response, the use of synthetic data for pretraining has emerged as a promising paradigm for pushing the frontier of performance. Despite this, the factors affecting synthetic data quality remain poorly understood. In this work, we introduce BeyondWeb, a synthetic data generation framework that produces high-quality synthetic data for pretraining. BeyondWeb significantly extends the capabilities of traditional web-scale datasets, outperforming state-of-the-art synthetic pretraining datasets such as Cosmopedia and Nemotron-CC's high-quality synthetic subset (Nemotron-Synth) by up to 5.1 percentage points (pp) and 2.6pp, respectively, when averaged across a suite of 14 benchmark evaluations. It delivers up to 7.7x faster training than open web data and 2.7x faster than Nemotron-Synth. Remarkably, a 3B model trained for 180B tokens on BeyondWeb outperforms an 8B model trained for the same token budget on Cosmopedia. We also present several insights from BeyondWeb on synthetic data for pretraining: what drives its benefits, which data to rephrase and how, and the impact of model size and family on data quality. Overall, our work shows that there's no silver bullet for generating high-quality synthetic pretraining data. The best outcomes require jointly optimizing many factors, a challenging task that requires rigorous science and practical expertise. Naive approaches can yield modest improvements, potentially at great cost, while well-executed methods can yield transformative improvements, as exemplified by BeyondWeb.