Paper 2604.24300
ReVSI: Rebuilding Visual Spatial Intelligence Evaluation for Accurate Assessment of VLM 3D Reasoning
- Published
- Apr 2026
- Research lab
- Independent
- Citations
- 7
- GitHub
- 83 stars
01 In brief
Summary
The paper introduces ReVSI, a benchmark for evaluating vision-language models' (VLMs) 3D spatial reasoning, addressing validity issues in the existing VSI-Bench.
Two key pitfalls are identified: annotation-to-video ground-truth drift (errors from point-cloud-based annotations) and scene-observability mismatch (questions unanswerable under sparse frame sampling).
ReVSI re-annotates objects and geometry across 381 scenes from 5 datasets, regenerates QA pairs with bias mitigation and human verification, and provides frame-budgeted variants (16/32/64/all frames) with object visibility metadata.
Evaluations reveal that proprietary models are under-assessed by VSI-Bench (e.g., on object counting), while fine-tuned models show high hallucination rates on dummy videos where queried objects are removed.
ReVSI exposes systematic failure modes, such as reliance on priors rather than visual evidence, and shows that fine-tuned models' gains on VSI-Bench do not translate to ReVSI.
The benchmark enables controlled diagnostics, revealing that models like InternVL3.5 hallucinate on black videos, while Qwen3-VL does not.
02 From the paper
Abstract
Current evaluations of spatial intelligence can be systematically invalid under modern vision-language model (VLM) settings. First, many benchmarks derive question-answer (QA) pairs from point-cloud-based 3D annotations originally curated for traditional 3D perception. When such annotations are treated as ground truth for video-based evaluation, reconstruction and annotation artifacts can miss objects that are clearly visible in the video, mislabel object identities, or corrupt geometry-dependent answers (e.g., size), yielding incorrect or ambiguous QA pairs. Second, evaluations often assume full-scene access, while many VLMs operate on sparsely sampled frames (e.g., 16-64), making many questions effectively unanswerable under the actual model inputs. We improve evaluation validity by introducing ReVSI, a benchmark and protocol that ensures each QA pair is answerable and correct under the model's actual inputs. To this end, we re-annotate objects and geometry across 381 scenes from 5 datasets to improve data quality, and regenerate all QA pairs with rigorous bias mitigation and human verification using professional 3D annotation tools. We further enhance evaluation controllability by providing variants across multiple frame budgets (16/32/64/all) and fine-grained object visibility metadata, enabling controlled diagnostic analyses. Evaluations of general and domain-specific VLMs on ReVSI reveal systematic failure modes that are obscured by prior benchmarks, yielding a more reliable and diagnostic assessment of spatial intelligence.