The Year in AI Papers
The year/Independent research

Paper 2511.02779

When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for Visual Chain-of-Thought

Yiyang Zhou, Haoqin Tu, Zijun Wang, Zeyu Wang, Niklas Muennighoff, Fan Nie, Yejin Choi, James Zou, Chaorui Deng, Shen Yan, Haoqi Fan, Cihang Xie, et al.

Published
Nov 2025
Research lab
Independent
Citations
10
GitHub
Not linked

01 In brief

Summary

The paper introduces MIRA (Multimodal Imagination for Reasoning Assessment), a benchmark designed to evaluate multimodal large language models (MLLMs) on tasks that require generating or using intermediate visual images (e.g., sketches, diagrams) for successful reasoning, mirroring human 'drawing to think'.

MIRA contains 546 problems across 20 task types in four domains: Euclidean Geometry, Physics-Based Reasoning, Abstract Spatial & Logical Puzzles, and Causal Transformations.

Each problem includes human-annotated intermediate visual steps and final answers.

The evaluation protocol has three levels: direct input, text-only chain-of-thought (Text-CoT), and simulated Visual-CoT with annotated images.

Results show that even top models (GPT-5, Gemini 2.5 Pro, o3) score below 20% with direct input, and Text-CoT often hurts performance (e.g., Gemini 2.5 Pro drops 18.3%).

Providing Visual-CoT images yields consistent gains, with an average relative improvement of 33.7% across models, especially in Physics tasks (from 20.7% to 40.0% for proprietary models).

Pass@k and majority voting show limited gains, indicating fundamental capability gaps.

The authors conclude that text-only reasoning is insufficient for visually intensive tasks and advocate for unified models that can 'think while drawing'.

02 From the paper

Abstract

We propose MIRA, a new benchmark designed to evaluate models in scenarios where generating intermediate visual images is essential for successful reasoning. Unlike traditional CoT methods that rely solely on text, tasks in MIRA require models to generate and utilize intermediate images - such as sketches, structural diagrams, or path drawings - to guide their reasoning process. This setup closely mirrors how humans solve complex problems through "drawing to think". To solve this, MIRA focuses on tasks that are intrinsically challenging and involve complex structures, spatial relationships, or reasoning steps that are difficult to express through language alone. To ensure that our evaluation data is of high-quality, we include 546 multimodal problems, annotated with intermediate visual images and final answers. We also propose a unified evaluation protocol for MIRA that spans three levels of evaluation input: direct input with image and question only, text-only CoT input with image and thinking prompts, and Visual-CoT input with both annotated image clues and textual thinking prompts. To probe the upper bound of model capacity on our benchmark, we also report pass@k and majority voting accuracies under different k settings. Experimental results show that existing multimodal large language models, including strongest private models as well as strong open-weight models, perform poorly when relying solely on textual prompts. However, when intermediate visual cues are provided, model performance improves consistently, yielding an average relative gain of 33.7% across all models and tasks. We also probe the upper bound by expanding the search space and designing textual prompts aligned with Visual-CoT, but both yield only limited improvements compared to our Visual-CoT setting. These results underscore the critical role of imagined visual information in enabling successful reasoning on MIRA.