Paper 2512.24271
Taming Hallucinations: Boosting MLLMs' Video Understanding via Counterfactual Video Generation
- Published
- Dec 2025
- Research lab
- Independent
- Citations
- 6
- GitHub
- 55 stars
01 In brief
Summary
The paper addresses visual ungrounded hallucinations in Multimodal Large Language Models (MLLMs), which over-rely on language priors when processing counterfactual videos that defy common sense.
To mitigate this, the authors introduce DualityForge, a framework using diffusion-based controllable video editing to transform real-world videos into counterfactual scenarios, embedding structured context (visual, semantic, commonsense anomalies) to enable automated generation of high-quality QA pairs.
This produces DualityVidQA, a dataset with 144K training QA pairs (104K for SFT, 40K for RL) and a 600-pair human-annotated test set.
They also propose DNA-Train, a two-stage SFT-RL regime where RL applies pair-wise ℓ1 advantage normalization to balance gradient updates between real and counterfactual videos.
Experiments show DNA-Train-7B achieves a 24.0% relative improvement over Qwen2.5-VL-7B on DualityVidQA-Test, and gains on hallucination benchmarks (EventHallusion) and general video understanding benchmarks (TempCompass, MVBench, TOMATO, TVBench).
Ablations confirm the necessity of paired data and the advantage normalization strategy.
The method generalizes across model scales (7B, 32B, 72B) and architectures (Qwen2.5-VL, LLaVA-Next-Video).
02 From the paper
Abstract
Multimodal Large Language Models (MLLMs) have made remarkable progress in video understanding. However, they suffer from a critical vulnerability: an over-reliance on language priors, which can lead to visual ungrounded hallucinations, especially when processing counterfactual videos that defy common sense. This limitation, stemming from the intrinsic data imbalance between text and video, is challenging to address due to the substantial cost of collecting and annotating counterfactual data. To address this, we introduce DualityForge, a novel counterfactual data synthesis framework that employs controllable, diffusion-based video editing to transform real-world videos into counterfactual scenarios. By embedding structured contextual information into the video editing and QA generation processes, the framework automatically produces high-quality QA pairs together with original-edited video pairs for contrastive training. Based on this, we build DualityVidQA, a large-scale video dataset designed to reduce MLLM hallucinations. In addition, to fully exploit the contrastive nature of our paired data, we propose Duality-Normalized Advantage Training (DNA-Train), a two-stage SFT-RL training regime where the RL phase applies pair-wise $\ell_1$ advantage normalization, thereby enabling a more stable and efficient policy optimization. Experiments on DualityVidQA-Test demonstrate that our method substantially reduces model hallucinations on counterfactual videos, yielding a relative improvement of 24.0% over the Qwen2.5-VL-7B baseline. Moreover, our approach achieves significant gains across both hallucination and general-purpose benchmarks, indicating strong generalization capability. We will open-source our dataset and code.