Paper 2510.17681
PICABench: How Far Are We from Physically Realistic Image Editing?
- Published
- Oct 2025
- Research lab
- Independent
- Citations
- 10
- GitHub
- 39 stars
01 In brief
Summary
PICABench is a benchmark for evaluating physical realism in image editing, addressing the gap where existing models and benchmarks focus on semantic fidelity but overlook physical effects like shadows, reflections, and state transitions.
It categorizes physical consistency into three dimensions—Optics, Mechanics, and State Transition—spanning eight sub-dimensions.
The benchmark includes 900 editing samples with instructions at three complexity levels.
PICAEval, a region-grounded VQA-based evaluation protocol, uses human-annotated key regions and per-case questions to reduce VLM hallucination and improve alignment with human judgment.
PICA-100K, a synthetic dataset of 105,085 editing examples derived from videos, is used to fine-tune models like FLUX.1-Kontext and Qwen-Image-Edit, yielding improvements in physical consistency (e.g., +1.71% accuracy for FLUX.1-Kontext).
Evaluation of 13 models shows that even state-of-the-art systems remain far from physically realistic editing, with open-source models scoring below 60% accuracy.
The work provides a benchmark, evaluation protocol, and dataset to advance physics-aware image editing.
02 From the paper
Abstract
Image editing has achieved remarkable progress recently. Modern editing models could already follow complex instructions to manipulate the original content. However, beyond completing the editing instructions, the accompanying physical effects are the key to the generation realism. For example, removing an object should also remove its shadow, reflections, and interactions with nearby objects. Unfortunately, existing models and benchmarks mainly focus on instruction completion but overlook these physical effects. So, at this moment, how far are we from physically realistic image editing? To answer this, we introduce PICABench, which systematically evaluates physical realism across eight sub-dimension (spanning optics, mechanics, and state transitions) for most of the common editing operations (add, remove, attribute change, etc.). We further propose the PICAEval, a reliable evaluation protocol that uses VLM-as-a-judge with per-case, region-level human annotations and questions. Beyond benchmarking, we also explore effective solutions by learning physics from videos and construct a training dataset PICA-100K. After evaluating most of the mainstream models, we observe that physical realism remains a challenging problem with large rooms to explore. We hope that our benchmark and proposed solutions can serve as a foundation for future work moving from naive content editing toward physically consistent realism.