The Year in AI Papers
The year/Independent research

Paper 2510.13795

Bee: A High-Quality Corpus and Full-Stack Suite to Unlock Advanced Fully Open MLLMs

Yi Zhang, Bolin Ni, Xin-Sheng Chen, Heng-Rui Zhang, Yongming Rao, Houwen Peng, Qinglin Lu, Han Hu, Meng-Hao Guo, Shi-Min Hu

Published
Oct 2025
Research lab
Independent
Citations
25
GitHub
Not linked

01 In brief

Summary

The paper introduces Honey-Data-15M, a 15-million-sample SFT dataset for fully open multimodal large language models (MLLMs), built to address data noise and a lack of complex reasoning data.

It also presents HoneyPipe, a transparent data curation pipeline built on the DataStudio framework, which uses rule-based and model-based filtering, dual-level Chain-of-Thought (CoT) enrichment (12.1M short CoT and 2.9M long CoT samples), and fidelity verification.

To validate the dataset, the authors trained Bee-8B, an 8B model, which achieves state-of-the-art results among fully open MLLMs and is competitive with semi-open models like InternVL3.5-8B, particularly in math and reasoning benchmarks.

Ablation studies confirm that each pipeline stage (noise filtering, CoT enrichment, and data selection) provides significant performance gains.

The full suite—dataset, pipeline code, model weights, training recipes, and evaluation harness—is released to the community.

02 From the paper

Abstract

Fully open multimodal large language models (MLLMs) currently lag behind proprietary counterparts, primarily due to a significant gap in data quality for supervised fine-tuning (SFT). Existing open-source datasets are often plagued by widespread noise and a critical deficit in complex reasoning data, such as Chain-of-Thought (CoT), which hinders the development of advanced model capabilities. Addressing these challenges, our work makes three primary contributions. First, we introduce Honey-Data-15M, a new SFT dataset comprising approximately 15 million QA pairs, processed through multiple cleaning techniques and enhanced with a novel dual-level (short and long) CoT enrichment strategy. Second, we introduce HoneyPipe, the data curation pipeline, and its underlying framework DataStudio, providing the community with a transparent and adaptable methodology for data curation that moves beyond static dataset releases. Finally, to validate our dataset and pipeline, we train Bee-8B, an 8B model on Honey-Data-15M. Experiments show that Bee-8B establishes a new state-of-the-art (SOTA) for fully open MLLMs, achieving performance that is competitive with, and in some cases surpasses, recent semi-open models such as InternVL3.5-8B. Our work delivers to the community a suite of foundational resources, including: the Honey-Data-15M corpus; the full-stack suite comprising HoneyPipe and DataStudio; training recipes; an evaluation harness; and the model weights. This effort demonstrates that a principled focus on data quality is a key pathway to developing fully open MLLMs that are highly competitive with their semi-open counterparts.