The Year in AI Papers
The year/Topics/Data and synthetic generation

Research collection

Data and synthetic generation

Training data itself: pretraining corpus construction, synthetic data generation, data filtering, deduplication, and data mixture studies.

Papers
30
Research labs
2
Official code
19

1–30 of 30 papers in this collection

01

Independent research

DataClaw0: Agentic Tailoring Multimodal Data from Raw Streams

DataClaw0 is a framework for intent-conditioned multimodal data tailoring, converting raw, noisy streams (e.g., tutorial videos, embodied trajectories, GUI logs) into schema-aligned, evidence-grounded training instances. The authors formalize this as a learnable capability and train models at 4B, 9B, and 27B scales, comparing a jointly trained omni model…

Cong Wan, Zeyu Guo, Zijian Cai, Jiangyang Li, et al.
Published
Jun 2026
Citations
0
Code
117 stars
Read summary Original
02

Independent research

OCC-RAG: Optimal Cognitive Core for Faithful Question Answering

The paper introduces OCC-RAG, a family of small language models (SLMs) specialized for faithful, context-grounded question answering (QA). The models are mid-trained from Qwen3 base models (0.6B and 1.7B) on a synthetic corpus of over 3 million QA examples generated via a novel pipeline. The corpus includes single-hop, multi-hop (single- and…

Maksim Savkin, Mikhail Goncharov, Alexander Gambashidze, Alla Chepurova, et al.
Published
May 2026
Citations
0
Code
43 stars
Read summary Original
03

Independent research

DataPrep-Bench: Benchmarking LLMs as Training Data Preparators

DataPrep-Bench is the first unified benchmark for evaluating LLMs as training data preparators, covering two complementary capabilities: data construction (transforming raw sources into supervised training data) and data quality evaluation (predicting downstream training utility of candidate datasets). The benchmark is grounded in downstream performance…

Hao Liang, Qifeng Cai, Yibo Lin, Jianzhuo Du, et al.
Published
May 2026
Citations
0
Code
Not linked
Read summary Original
04

arXiv.org

MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Image

MulTaBench is a benchmark of 40 datasets (20 image-tabular, 20 text-tabular) designed for Multimodal Tabular Learning (MMTL). It addresses the limitation of existing benchmarks that focus on modality co-occurrence, which masks the benefits of task-specific tuning. The authors propose two criteria for dataset inclusion: Joint Signal (each modality provides…

Alan Arazi, Eilam Shapira, Shoham Grunblat, Mor Ventura, et al.
Published
May 2026
Citations
2
Code
Not linked
Read summary Original
05

arXiv.org

K12-KGraph: A Curriculum-Aligned Knowledge Graph for Benchmarking and Training Educational LLMs

The paper introduces K12-KGraph, a curriculum-aligned knowledge graph built from official Chinese People's Education Press textbooks, covering mathematics, physics, chemistry, and biology across primary, middle, and high school. It includes nine node types and fourteen relation types, capturing both curriculum structure and visual grounding. From this…

Hao Liang, Qihan Lin, Zhaoyang Han, Xiaochen Ma, et al.
Published
May 2026
Citations
0
Code
314 stars
Read summary Original
06

arXiv.org

Programming with Data: Test-Driven Data Engineering for Self-Improving LLMs from Raw Corpora

The paper introduces Programming with Data (ProDa), a paradigm that maps the LLM data-engineering lifecycle onto software development by using a shared three-level knowledge structure (L1 concepts, L2 relations, L3 reasoning chains) extracted from raw corpora. This structure serves as the common foundation for both training data synthesis and benchmark…

Chenkai Pan, Xinglong Xu, Yuhang Xu, Yujun Wu, et al.
Published
Apr 2026
Citations
1
Code
162 stars
Read summary Original
07

arXiv.org

MegaStyle: Constructing Diverse and Scalable Style Dataset via Consistent Text-to-Image Style Mapping

The paper introduces MegaStyle, a scalable data curation pipeline for constructing a large-scale, high-quality style dataset (MegaStyle-1.4M) with intra-style consistency and inter-style diversity. It leverages the consistent text-to-image style mapping of large generative models like Qwen-Image to generate style pairs from curated prompts. The pipeline…

Junyao Gao, Sibo Liu, Jiaxing Li, Yanan Sun, et al.
Published
Apr 2026
Citations
2
Code
131 stars
Read summary Original
08

arXiv.org

Adam's Law: Textual Frequency Law on Large Language Models

The paper proposes the Textual Frequency Law (TFL), which states that for LLMs, higher-frequency textual data should be preferred for both prompting and fine-tuning when meaning is kept constant. The framework includes Textual Frequency Distillation (TFD) to enhance frequency estimation via story completion, and Curriculum Textual Frequency Training (CTFT)…

Hongyuan Adam Lu, Z. L., Victor Wei, Zefan Zhang, et al.
Published
Apr 2026
Citations
3
Code
Not linked
Read summary Original
09

arXiv.org

Project Imaging-X: A Survey of 1000+ Open-Access Medical Imaging Datasets for Foundation Model Development

This survey reviews over 1,000 open-access medical imaging datasets released between 2000 and 2025, analyzing their modalities, tasks, anatomical regions, and limitations. It finds the landscape is fragmented, small-scale, and unevenly distributed, with 2D images dominating, pathology and X-ray being the most common modalities, and classification and…

Zhongying Deng, Cheng Tang, Ziyan Huang, Jiashi Lin, et al.
Published
Mar 2026
Citations
2
Code
472 stars
Read summary Original
10

arXiv.org

DataFlex: A Unified Framework for Data-Centric Dynamic Training of Large Language Models

DataFlex is a unified data-centric dynamic training framework built on LLaMA-Factory, designed to treat data as a first-class optimization variable. It integrates three paradigms—data selection, data mixture optimization, and data reweighting—through modular trainer abstractions (Select, Mix, Weight Trainers) and pluggable components (selectors, mixers,…

Hao Liang, Zhengyang Zhao, Meiyi Qiang, Mingrui Chen, et al.
Published
Mar 2026
Citations
3
Code
1.9K stars
Read summary Original
11

Qwen

HopChain: Multi-Hop Data Synthesis for Generalizable Vision-Language Reasoning

The paper introduces HopChain, a scalable framework for synthesizing multi-hop vision-language reasoning data for reinforcement learning with verifiable rewards (RLVR) training of vision-language models (VLMs). The authors identify that long chain-of-thought (CoT) reasoning in VLMs suffers from diverse, compounding failure modes (perception, reasoning,…

Shenzhi Wang, Shixuan Liu, Jing Zhou, Chang Gao, et al.
Published
Mar 2026
Citations
2
Code
Not linked
Read summary Original
12

arXiv.org

CHIMERA: Compact Synthetic Data for Generalizable LLM Reasoning

The paper introduces CHIMERA, a compact synthetic dataset of 9,225 samples designed to address data-centric challenges in LLM reasoning post-training: cold-start data scarcity, limited domain coverage, and the annotation bottleneck. The dataset is built via a three-stage pipeline: subject expansion using GPT-5 to create a hierarchical taxonomy of 1,179…

Xinyu Zhu, Yihao Feng, Yanchao Sun, Xianzhi Du, et al.
Published
Mar 2026
Citations
2
Code
Not linked
Read summary Original
13

arXiv.org

From Blind Spots to Gains: Diagnostic-Driven Iterative Training for Large Multimodal Models

The paper introduces Diagnostic-driven Progressive Evolution (DPE), a closed-loop training framework for Large Multimodal Models (LMMs) that addresses limitations of static data and heuristic self-evolution methods. DPE iteratively performs diagnosis, targeted data generation, and reinforcement learning. A diagnostic agent identifies capability blind spots…

Hongrui Jia, Chaoya Jiang, Yongrui Heng, Shikun Zhang, et al.
Published
Feb 2026
Citations
0
Code
31 stars
Read summary Original
14

NVIDIA

On Data Engineering for Scaling LLM Terminal Capabilities

This paper addresses the lack of transparency in training data for terminal-capable LLMs by introducing Terminal-Task-Gen, a synthetic task generation pipeline, and Terminal-Corpus, a large-scale open-source dataset. The authors fine-tune Qwen3 models (8B, 14B, 32B) to create Nemotron-Terminal, achieving substantial gains on Terminal-Bench 2.0: 8B improves…

Renjie Pi, Grace Lam, Mohammad Shoeybi, Pooya Jannaty, et al.
Published
Feb 2026
Citations
15
Code
Not linked
Read summary Original
15

Independent research

Less is Enough: Synthesizing Diverse Data in LLM Feature Space with Sparse Autoencoders

The paper introduces Feature Activation Coverage (FAC), a metric that measures data diversity in an interpretable feature space derived from Sparse Autoencoders (SAEs) trained on LLM internal activations. The authors propose FAC Synthesis, a framework that identifies task-relevant features missing from a seed dataset and generates synthetic samples to…

Zhongzhi Li, Xuansheng Wu, Yijiang Li, Lijie Hu, et al.
Published
Feb 2026
Citations
2
Code
111 stars
Read summary Original
16

Qwen

OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration

OPUS is a dynamic data selection framework for LLM pre-training that scores candidate samples by projecting their optimizer-induced effective updates onto a target direction from a stable, in-distribution proxy. It addresses the limitations of static filters and raw-gradient-based dynamic methods by aligning selection with the actual update geometry of…

Shaobo Wang, Xuan Ouyang, Tianyi Xu, Yuzheng Hu, et al.
Published
Feb 2026
Citations
10
Code
Not linked
Read summary Original
17

arXiv.org

Training Data Efficiency in Multimodal Process Reward Models

The paper studies data efficiency for training Multimodal Process Reward Models (MPRMs), which provide step-level supervision for visual reasoning in Multimodal Large Language Models. Training MPRMs typically requires large-scale Monte Carlo (MC)-annotated corpora, which is computationally expensive. The authors find that random subsampling of training…

Jinyuan Li, Chengsong Huang, Langlin Huang, Shaoyang Xu, et al.
Published
Feb 2026
Citations
1
Code
7 stars
Read summary Original
18

arXiv.org

MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods

MMFineReason is a large-scale multimodal reasoning dataset with 1.8M samples and 5.1B solution tokens, built via a three-stage pipeline: data aggregation and standardization, CoT distillation from Qwen3-VL-235B-A22B-Thinking, and quality/difficulty-based selection. The dataset covers mathematics (79.4%), science (13.8%), puzzle/game (4.6%), and general/OCR…

Honglin Lin, Zheng Liu, Yun Zhu, Chonghan Qin, et al.
Published
Jan 2026
Citations
33
Code
6 stars
Read summary Original
19

arXiv.org

Can LLMs Clean Up Your Mess? A Survey of Application-Ready Data Preparation with LLMs

This paper surveys the use of large language models (LLMs) for data preparation, covering data cleaning, integration, and enrichment. It contrasts traditional rule-based and model-specific methods with LLM-enhanced approaches that leverage prompting, retrieval-augmented generation (RAG), fine-tuning, and agentic workflows. The survey identifies three core…

Wei Zhou, Jun Zhou, Haoyu Wang, Zhenghao Li, et al.
Published
Jan 2026
Citations
6
Code
814 stars
Read summary Original
20

Annual Meeting of the Association for Computational Linguistics

RubricHub: A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse-to-Fine Generation

The paper introduces RubricHub, a large-scale (~110k) multi-domain rubric dataset, generated via an automated Coarse-to-Fine framework to address limitations in existing rubric-based evaluation for open-ended tasks. The framework comprises three stages: principle-guided and response-grounded generation, multi-model aggregation, and difficulty evolution,…

Sunzhu Li, Jiale Zhao, Miteto Wei, Huimin Ren, et al.
Published
Jan 2026
Citations
29
Code
66 stars
Read summary Original
21

arXiv.org

DataFlow: An LLM-Driven Framework for Unified Data Preparation and Workflow Automation in the Era of Data-Centric AI

DataFlow is a unified, LLM-driven framework for data preparation in the era of data-centric AI, addressing the fragmentation of ad-hoc scripts and workflows. It provides a PyTorch-style API with nearly 200 reusable operators and six domain-general pipelines (text, math, code, Text-to-SQL, agentic RAG, knowledge extraction). The framework includes a global…

Hao Liang, Xiaochen Ma, Zhou Liu, Zhen Hao Wong, et al.
Published
Dec 2025
Citations
19
Code
7.2K stars
Read summary Original
22

arXiv.org

Diffusion Language Models are Super Data Learners

The paper demonstrates that under data-constrained pre-training, diffusion language models (DLMs) consistently surpass autoregressive (AR) models of equal size, a phenomenon termed the 'Intelligence Crossover.' This crossover shifts later with more or higher-quality data, earlier with larger models, and persists across dense and sparse architectures. The…

Jinjie Ni, Qian Liu, Longxu Dou, Chao Du, et al.
Published
Nov 2025
Citations
50
Code
228 stars
Read summary Original
23

arXiv.org

FineVision: Open Data Is All You Need

FineVision is a large-scale, open corpus of 24 million samples (17 million images, 89 million turns, 9.5 billion answer tokens) for training vision-language models. It unifies over 200 public sources into 185 subsets via a semi-automated, human-in-the-loop pipeline that includes schema mapping, cleaning, de-duplication, and decontamination against 66…

Luis Wiedmann, Orr Zohar, Amir Mahla, Xiaohan Wang, et al.
Published
Oct 2025
Citations
38
Code
Not linked
Read summary Original
24

arXiv.org

Bee: A High-Quality Corpus and Full-Stack Suite to Unlock Advanced Fully Open MLLMs

The paper introduces Honey-Data-15M, a 15-million-sample SFT dataset for fully open multimodal large language models (MLLMs), built to address data noise and a lack of complex reasoning data. It also presents HoneyPipe, a transparent data curation pipeline built on the DataStudio framework, which uses rule-based and model-based filtering, dual-level…

Yi Zhang, Bolin Ni, Xin-Sheng Chen, Heng-Rui Zhang, et al.
Published
Oct 2025
Citations
25
Code
Not linked
Read summary Original
25

arXiv.org

KORMo: Korean Open Reasoning Model for Everyone

This paper introduces KORMo-10B, a 10.8B-parameter fully open bilingual Korean-English language model trained from scratch on a corpus where 68.74% of the Korean portion is synthetic. The authors systematically investigate the feasibility of using synthetic data for non-English fully open models (FOMs), addressing stability, tokenizer design, and bias.…

Minjun Kim, Hyeonseok Lim, Hangyeol Yoo, Inho Won, et al.
Published
Oct 2025
Citations
4
Code
119 stars
Read summary Original
26

arXiv.org

Winning the Pruning Gamble: A Unified Approach to Joint Sample and Token Pruning for Efficient Supervised Fine-Tuning

The paper introduces Q-Tuning, a unified framework for joint sample- and token-level pruning during supervised fine-tuning (SFT) of large language models. It addresses the inefficiency of existing methods that treat sample and token pruning separately. The authors propose the Error–Uncertainty (EU) Plane, which categorizes training samples into four…

Shaobo Wang, Jiaming Wang, Jiajun Zhang, Cong Wang, et al.
Published
Sep 2025
Citations
9
Code
3 stars
Read summary Original
27

arXiv.org

Hala Technical Report: Building Arabic-Centric Instruction & Translation Models at Scale

The HALA technical report introduces a family of Arabic-centric instruction and translation models built with a translate-and-tune pipeline. The pipeline compresses a strong AR↔EN teacher model (CohereLabs/command-a-translate-08-2025) to FP8, achieving ~2x higher throughput with no quality loss. This teacher translates 405K Open-Orca pairs into Arabic, and…

Hasan Abed Al Kader Hammoud, Mohammad Zbeeb, Bernard Ghanem
Published
Sep 2025
Citations
5
Code
5 stars
Read summary Original
28

arXiv.org

MachineLearningLM: Scaling Many-shot In-context Learning via Continued Pretraining

The paper introduces MACHINELEARNINGLM, a continued-pretraining framework that enables large language models (LLMs) to perform many-shot in-context learning (ICL) on tabular machine-learning tasks without gradient updates. The method synthesizes millions of tasks from structural causal models (SCMs), uses a random-forest teacher for a warm-start…

Haoyu Dong, Pengkun Zhang, Mingzhe Lu, Yanzhen Shen, et al.
Published
Sep 2025
Citations
4
Code
59 stars
Read summary Original
29

arXiv.org

Hermes 4 Technical Report

Hermes 4 is a family of hybrid reasoning models (14B, 70B, 405B) that combine structured, multi-turn reasoning with broad instruction-following. The models were trained on a dataset of ~5 million samples (19B tokens), including 3.5M reasoning and 1.6M non-reasoning samples, synthesized via DataForge, a graph-based generator, and rejection sampling with…

Ryan Teknium, Roger Jin, Jai Suphavadeeprasit, Dakota Mahan, et al.
Published
Aug 2025
Citations
3
Code
Not linked
Read summary Original
30

arXiv.org

BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining

BeyondWeb is a synthetic data generation framework for LLM pretraining that outperforms existing synthetic datasets like Cosmopedia and Nemotron-Synth by up to 5.1 and 2.6 percentage points, respectively, across 14 benchmarks. It achieves up to 7.7x faster training than open web data and 2.7x faster than Nemotron-Synth. A 3B model trained on BeyondWeb for…

DatologyAI, :, Pratyush Maini, Vineeth Dorna, et al.
Published
Aug 2025
Citations
16
Code
Not linked
Read summary Original