The Year in AI Papers
The year/Independent research

Paper 2605.23271

EvalVerse: Pipeline-Aware and Expert-Calibrated Benchmarking for Professional Cinematic Video Generation

Songlin Yang, Haobin Zhong, Ruilin Zhang, Xiaotong Zhao, Shuai Li, Kai Zheng, Xuyi Yang, Zhe Wang, Zhenchen Tang, Yang Li, Bohai Gu, Zhengwei Peng, et al.

Published
May 2026
Research lab
Independent
Citations
2
GitHub
Not linked

01 In brief

Summary

EvalVerse is a comprehensive evaluation framework for professional cinematic video generation that addresses the gap between basic prompt-following and true cinematic quality.

It introduces a pipeline-aware taxonomy mirroring the filmmaking workflow (pre-production, production, post-production) with 3 stages, 7 aspects, 18 dimensions, 45 sub-dimensions, and 196 rationales.

The framework uses a 'Real-to-Gen' data engine to construct test pairs from a million-scale professional database, covering tasks like text-to-video, reference-to-video, multi-shot, and audio-visual generation.

To align machine scoring with human experts, EvalVerse employs a two-stage VLM fine-tuning (preference alignment and score calibration) and a progressive calibration mechanism with prompt-level, fusion-level, and parameter-level tiers.

Evaluations on 11 models show Seedance 2.0 as the top performer, with high human-machine alignment (SRCC up to 0.95).

EvalVerse extends beyond benchmarking to serve as infrastructure for reward models in RL and expert evaluators for agentic workflows.

02 From the paper

Abstract

The rapid evolution of generative video foundation models has propelled the field toward professional-grade cinematic synthesis. To achieve such demanding quality, the community transitions towards Reinforcement Learning (RL) and agentic workflows. However, reliable evaluation has emerged as a critical bottleneck. Existing benchmarks predominantly evaluate ''whether it is right'' (basic prompt-following) while fundamentally neglecting ''whether it is good'' (cinematic quality, acting, and aesthetics). Furthermore, current automated metrics lack the domain-specific rigor required to provide trustworthy signals, creating a severe credibility gap between human aesthetic perception and machine scoring. To bridge this gap, we introduce EvalVerse, a comprehensive, pipeline-aware, and expert-calibrated evaluation framework. We treat video generation assessment not merely as an engineering task, but as a core scientific problem: the systematic digitization of subjective cinematic expertise. First, we organize domain knowledge into an evaluation taxonomy aligned with the professional filmmaking workflow (pre-production, production, and post-production). Second, we distill human expert judgments into a curated dataset with large-scale human annotations. Third, we inject this knowledge into Vision-Language Models (VLMs) through an expert-calibrated fine-tuning strategy, enabling the VLM to perform explicit Chain-of-Thought reasoning. Compared to previous works, EvalVerse not only retains compatibility with foundational ''rightness'' metrics, but also significantly expands the criteria to ''goodness'' and broaden the task coverage to complex multi-shot sequencing and audio-visual integration. Consequently, by providing granular diagnostic signals, EvalVerse transcends a static leaderboard and establishes a fundamental infrastructure for future work, such as reward models and evaluator agent.