Paper 2508.11630
Thyme: Think Beyond Images
- Published
- Aug 2025
- Research lab
- Independent
- Citations
- 119
- GitHub
- 586 stars
01 In brief
Summary
The paper introduces Thyme (Think Beyond Images), a paradigm for multimodal large language models (MLLMs) that autonomously generates and executes code for image manipulations (cropping, rotation, contrast enhancement) and complex computations, going beyond existing 'think with images' approaches.
Thyme uses a two-stage training: Supervised Fine-Tuning (SFT) on a curated 500K-sample dataset (requiring ~200 GPU hours) to teach code generation, followed by Reinforcement Learning (RL) with a proposed GRPO-ATS algorithm that applies different temperatures for text (1.0) and code (0.0) generation to balance exploration and code precision.
A secure sandbox executes code, handling formatting and boundary conditions.
RL data includes 10,000 manually annotated high-resolution images.
Evaluations on nearly 20 benchmarks show significant gains over Qwen2.5-VL-7B, especially in high-resolution perception and reasoning tasks, with improvements like +8.2 on HRbench-4K overall and +9.8 on MME-Real reasoning.
Ablations show masking sandbox outputs, training on last round only, and math data annealing are effective; consistency reward improves RL, while process and code rewards do not.
Limitations include base model capability constraints and lack of benchmarks for rotation/contrast tasks.
02 From the paper
Abstract
Following OpenAI's introduction of the ``thinking with images'' concept, recent efforts have explored stimulating the use of visual information in the reasoning process to enhance model performance in perception and reasoning tasks. However, to the best of our knowledge, no open-source work currently offers a feature set as rich as proprietary models (O3), which can perform diverse image manipulations and simultaneously enhance logical reasoning capabilities through code. In this paper, we make a preliminary attempt in this direction by introducing Thyme (Think Beyond Images), a novel paradigm for enabling MLLMs to transcend existing ``think with images'' approaches by autonomously generating and executing diverse image processing and computational operations via executable code. This approach not only facilitates a rich, on-the-fly set of image manipulations (e.g., cropping, rotation, contrast enhancement) but also allows for mathematical computations, all while maintaining high autonomy in deciding when and how to apply these operations. We activate this capability through a two-stage training strategy: an initial SFT on a curated dataset of 500K samples to teach code generation, followed by a RL phase to refine decision-making. For the RL stage, we manually collect and design high-resolution question-answer pairs to increase the learning difficulty, and we propose GRPO-ATS (Group Relative Policy Optimization with Adaptive Temperature Sampling), an algorithm that applies distinct temperatures to text and code generation to balance reasoning exploration with code execution precision. We conduct extensive experimental analysis and ablation studies. Comprehensive evaluations on nearly 20 benchmarks show that Thyme yields significant and consistent performance gains, particularly in challenging high-resolution perception and complex reasoning tasks.