Paper 2601.06521
BabyVision: Visual Reasoning Beyond Language
- Published
- Jan 2026
- Research lab
- Independent
- Citations
- 24
- GitHub
- 238 stars
01 In brief
Summary
The paper introduces BABYVISION, a benchmark to evaluate core visual abilities in Multimodal LLMs (MLLMs) that are independent of linguistic knowledge, targeting skills humans develop before language.
It contains 388 questions across 22 subtypes in four categories: Fine-grained Discrimination, Visual Tracking, Spatial Perception, and Visual Pattern Recognition.
Evaluations show that state-of-the-art MLLMs perform significantly below human baselines: the best model, Gemini3-Pro-Preview, scores 49.7% versus 94.1% for adults, and even lags behind 6-year-olds.
The authors identify four failure modes—loss of fine-grained detail, loss of manifold identity, failure of spatial imagination, and failure of visual pattern induction—all stemming from a 'verbalization bottleneck' where visual information is compressed into language.
They also propose BABYVISION-GEN, a generative extension where models solve tasks by producing visual annotations, and an automatic evaluation toolkit with 96.1% agreement with human judges.
Experiments with generation models show promising but limited results.
RLVR fine-tuning on Qwen3-VL-8B yields a +4.8 overall accuracy gain, except in visual tracking.
The benchmark is released for reproduction, aiming to guide progress toward human-level visual intelligence in multimodal systems.
02 From the paper
Abstract
While humans develop core visual skills long before acquiring language, contemporary Multimodal LLMs (MLLMs) still rely heavily on linguistic priors to compensate for their fragile visual understanding. We uncovered a crucial fact: state-of-the-art MLLMs consistently fail on basic visual tasks that humans, even 3-year-olds, can solve effortlessly. To systematically investigate this gap, we introduce BabyVision, a benchmark designed to assess core visual abilities independent of linguistic knowledge for MLLMs. BabyVision spans a wide range of tasks, with 388 items divided into 22 subclasses across four key categories. Empirical results and human evaluation reveal that leading MLLMs perform significantly below human baselines. Gemini3-Pro-Preview scores 49.7, lagging behind 6-year-old humans and falling well behind the average adult score of 94.1. These results show despite excelling in knowledge-heavy evaluations, current MLLMs still lack fundamental visual primitives. Progress in BabyVision represents a step toward human-level visual perception and reasoning capabilities. We also explore solving visual reasoning with generation models by proposing BabyVision-Gen and automatic evaluation toolkit. Our code and benchmark data are released at https://github.com/UniPat-AI/BabyVision for reproduction.