Paper 2508.21496
ELV-Halluc: Benchmarking Semantic Aggregation Hallucinations in Long Video Understanding
- Published
- Aug 2025
- Research lab
- Independent
- Citations
- 10
- GitHub
- 11 stars
01 In brief
Summary
The paper introduces ELV-Halluc, the first benchmark for evaluating Semantic Aggregation Hallucination (SAH) in long videos.
SAH occurs when models correctly perceive frame-level semantics but misattribute them across events, a problem that intensifies with semantic complexity.
The benchmark uses event-by-event videos (average 672.4 seconds) and adversarial question pairs (ground truth vs.
in-video vs.
out-of-video hallucinations) across four aspects: visual details, actions, objects, and declarative content.
Experiments on 14 open-source and 2 closed-source models show SAH increases with event count and semantic variation rate, and is more frequent on rapidly changing semantics.
The authors propose mitigation strategies: improved positional encoding (VideoRoPE) reduces SAH, and Direct Preference Optimization (DPO) with in-video pairs reduces SAH ratio by 27.7% (from 8.3 to 6.0) while improving VideoMME accuracy by 0.9%.
The benchmark includes 4,800 binary QA pairs from 200 videos, with an additional 148 videos for DPO training.
02 From the paper
Abstract
Video multimodal large language models (Video-MLLMs) have achieved remarkable progress in video understanding. However, they remain vulnerable to hallucination-producing content inconsistent with or unrelated to video inputs. Previous video hallucination benchmarks primarily focus on short-videos. They attribute hallucinations to factors such as strong language priors, missing frames, or vision-language biases introduced by the visual encoder. While these causes indeed account for most hallucinations in short videos, they still oversimplify the cause of hallucinations. Sometimes, models generate incorrect outputs but with correct frame-level semantics. We refer to this type of hallucination as Semantic Aggregation Hallucination (SAH), which arises during the process of aggregating frame-level semantics into event-level semantic groups. Given that SAH becomes particularly critical in long videos due to increased semantic complexity across multiple events, it is essential to separate and thoroughly investigate the causes of this type of hallucination. To address the above issues, we introduce ELV-Halluc, the first benchmark dedicated to long-video hallucination, enabling a systematic investigation of SAH. Our experiments confirm the existence of SAH and show that it increases with semantic complexity. Additionally, we find that models are more prone to SAH on rapidly changing semantics. Moreover, we discuss potential approaches to mitigate SAH. We demonstrate that positional encoding strategy contributes to alleviating SAH, and further adopt DPO strategy to enhance the model's ability to distinguish semantics within and across events. To support this, we curate a dataset of 8K adversarial data pairs and achieve improvements on both ELV-Halluc and Video-MME, including a substantial 27.7% reduction in SAH ratio.