Paper 2508.05405
DeepPHY: Benchmarking Agentic VLMs on Physical Reasoning
- Published
- Aug 2025
- Research lab
- Independent
- Citations
- 10
- GitHub
- 172 stars
01 In brief
Summary
DeepPHY is a benchmark framework for evaluating interactive physical reasoning in agentic Vision-Language Models (VLMs).
It integrates six physics-based simulation environments: PHYRE, I-PHYRE, Kinetix, Pooltool, Angry Birds, and Cut the Rope, converting continuous action spaces into discrete, structured formats and augmenting observations with annotations.
The benchmark formalizes tasks as trial-based POMDPs, evaluating models on success rate, Pass@K, and average attempts, using two prompt formats: Vision-Language-Action (VLA) and World Model (WM).
Evaluations of 17 open- and closed-source VLMs reveal that even state-of-the-art models struggle, with most open-source models performing near or below a random MOCK baseline.
Top models like GPT-o3 achieve 23.1% success on PHYRE and 81.7% on I-PHYRE, but performance drops significantly in complex tasks like Kinetix and Cut the Rope.
The WM prompt format rarely improves performance and often degrades it.
Models often rely on brute-force heuristics (e.g., in Pooltool) and fail to translate descriptive physical knowledge into precise control, highlighting a fundamental disconnect between descriptive understanding and procedural control.
02 From the paper
Abstract
Although Vision Language Models (VLMs) exhibit strong perceptual abilities and impressive visual reasoning, they struggle with attention to detail and precise action planning in complex, dynamic environments, leading to subpar performance. Real-world tasks typically require complex interactions, advanced spatial reasoning, long-term planning, and continuous strategy refinement, usually necessitating understanding the physics rules of the target scenario. However, evaluating these capabilities in real-world scenarios is often prohibitively expensive. To bridge this gap, we introduce DeepPHY, a novel benchmark framework designed to systematically evaluate VLMs' understanding and reasoning about fundamental physical principles through a series of challenging simulated environments. DeepPHY integrates multiple physical reasoning environments of varying difficulty levels and incorporates fine-grained evaluation metrics. Our evaluation finds that even state-of-the-art VLMs struggle to translate descriptive physical knowledge into precise, predictive control.