Paper 2605.12882
CiteVQA: Benchmarking Evidence Attribution for Trustworthy Document Intelligence
- Published
- May 2026
- Research lab
- Independent
- Citations
- 3
- GitHub
- 69 stars
01 In brief
Summary
CiteVQA is a benchmark for evaluating Multimodal Large Language Models (MLLMs) on document visual question answering, requiring both correct answers and element-level bounding-box citations.
It includes 1,897 questions from 711 PDFs across seven domains and two languages, with an average of 40.6 pages per document.
Ground-truth citations are generated via an automated pipeline using masking ablation and expert review.
The key metric, Strict Attributed Accuracy (SAA), credits predictions only when both answer and cited region are correct.
Auditing 20 MLLMs reveals pervasive 'Attribution Hallucination,' where models give correct answers but cite wrong regions.
The best system, Gemini-3.1-Pro-Preview, achieves an SAA of 76.0, while the best open-source model reaches only 22.5.
The benchmark exposes a reliability gap in current systems, providing tools to improve trustworthy document intelligence.
The repository is available at https://github.com/opendatalab/CiteVQA.
The work also introduces traceability metrics like Recall and Relevance, and shows that attribution quality correlates with answer accuracy, suggesting that better localization could improve overall performance.
02 From the paper
Abstract
Multimodal Large Language Models (MLLMs) have significantly advanced document understanding, yet current Doc-VQA evaluations score only the final answer and leave the supporting evidence unchecked. This answer-only approach masks a critical failure mode: a model can land on the correct answer while grounding it in the wrong passage -- a critical risk in high-stakes domains like law, finance, and medicine, where every conclusion must be traceable to a specific source region. To address this, we introduce CiteVQA, a benchmark that requires models to return element-level bounding-box citations alongside each answer, evaluating both jointly. CiteVQA comprises 1,897 questions across 711 PDFs spanning seven domains and two languages, averaging 40.6 pages per document. To ensure fidelity and scalability, the ground-truth citations are generated by an automated pipeline-which identifies crucial evidence via masking ablation-and are subsequently validated through expert review. At the core of our evaluation is Strict Attributed Accuracy (SAA), which credits a prediction only when the answer and the cited region are both correct. Auditing 20 MLLMs reveals a pervasive Attribution Hallucination: models frequently produce the right answer while citing the wrong region. The strongest system (Gemini-3.1-Pro-Preview) achieves an SAA of only 76.0, and the strongest open-source MLLM reaches just 22.5. Ultimately, towards trustworthy document intelligence, CiteVQA exposes a reliability gap that answer-only evaluations overlook, providing the instrumentation needed to close it. Our repository is available at https://github.com/opendatalab/CiteVQA.