Paper 2607.10400
SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understanding
- Published
- Jul 2026
- Research lab
- Independent
- Citations
- 0
- GitHub
- 8 stars
01 In brief
Summary
SynthDocBench is a fully synthetic benchmark for long-context visual document understanding, designed to systematically control factors like document length, layout, modality, and question type.
It comprises 200 reports (avg.
51.1 pages, 16.7 charts) and 1,788 questions across three subsets: chart-reading, complex multi-hop, and cross-modal.
Documents are generated via an LLM pipeline with six layout archetypes and a 40% random override to prevent spurious correlations.
Evaluation of eight frontier VLMs reveals three failure modes: sharp degradation with increasing difficulty (L1→L5), positional sensitivity where the middle third of documents is hardest for most models, and collapse of precise chart-reading in long contexts.
Gemini-3.1-Pro leads with 0.725 ACC, while Qwen2.5-VL-7B trails at 0.081.
OCR+text baselines outperform vision on complex questions but underperform on chart-reading, isolating visual perception as a key bottleneck.
The benchmark aims to provide controlled diagnostic signals beyond existing benchmarks.
02 From the paper
Abstract
Vision language models (VLMs) have achieved strong performance on visual document understanding benchmarks such as DocVQA, ChartQA, and MMLongBench-Doc. However, real-world documents combine multiple factors such as length, layout complexity, modality, and question difficulty, which makes it difficult to attribute model failures to specific causes. We introduce SynthDocBench, a fully synthetic benchmark for long-context visual document understanding that systematically controls factors including document length, layout structure, modality composition, and question type. The benchmark is constructed using a combinatorial design, each factor is varied independently across generated documents, enabling controlled analysis of model behavior. Documents are generated end to end using an LLM pipeline across six layout archetypes, with a 40 percent random override to prevent models from exploiting spurious correlations. Additionally, SynthDocBench spans long-context documents with substantially greater length and structural diversity than existing benchmarks. Evaluating seven frontier VLMs, we uncover three failure modes that existing benchmarks cannot surface: sharp degradation with document length, a systematic positional sensitivity in which the middle third of a document is hardest for five of six models and five of six models show a negative Early-to-Late trend (steepest decline: 8.3 percentage points), and breakdown of chart comprehension in long-document settings. These results suggest that current models may be overfitting to benchmark artifacts rather than achieving robust long-context visual document understanding.