Paper 2510.18701
UniGenBench++: A Unified Semantic Evaluation Benchmark for Text-to-Image Generation
- Published
- Oct 2025
- Research lab
- Independent
- Citations
- 20
- GitHub
- 139 stars
01 In brief
Summary
UniGenBench++ is a unified semantic evaluation benchmark for text-to-image (T2I) generation, addressing limitations in existing benchmarks such as coarse evaluation dimensions, lack of prompt diversity, and missing multilingual support.
It comprises 600 prompts hierarchically organized into 5 primary themes and 20 subthemes, covering 10 primary and 27 sub-evaluation criteria (e.g., style, world knowledge, attribute, action, relationship, logical reasoning, grammar, text generation).
Each prompt is provided in both English and Chinese, in short and long forms, and targets 1–10 test points.
The benchmark uses a streamlined, point-wise evaluation pipeline with Gemini-2.5-Pro as the evaluator, and a dedicated offline evaluation model (trained on 375K samples) that outperforms Qwen2.5-VL-72b.
Comprehensive benchmarking of closed-source (e.g., GPT-4o-1.5, Nano Banana Pro, Seedream-4.5) and open-source (e.g., FLUX.2-dev, Z-Image, Qwen-Image) models reveals that both excel in style and world knowledge but struggle with logical reasoning; open-source models show larger performance fluctuations, particularly in grammar and action dimensions.
02 From the paper
Abstract
Recent progress in text-to-image (T2I) generation underscores the importance of reliable benchmarks in evaluating how accurately generated images reflect the semantics of their textual prompt. However, (1) existing benchmarks lack the diversity of prompt scenarios and multilingual support, both essential for real-world applicability; (2) they offer only coarse evaluations across primary dimensions, covering a narrow range of sub-dimensions, and fall short in fine-grained sub-dimension assessment. To address these limitations, we introduce UniGenBench++, a unified semantic assessment benchmark for T2I generation. Specifically, it comprises 600 prompts organized hierarchically to ensure both coverage and efficiency: (1) spans across diverse real-world scenarios, i.e., 5 main prompt themes and 20 subthemes; (2) comprehensively probes T2I models' semantic consistency over 10 primary and 27 sub evaluation criteria, with each prompt assessing multiple testpoints. To rigorously assess model robustness to variations in language and prompt length, we provide both English and Chinese versions of each prompt in short and long forms. Leveraging the general world knowledge and fine-grained image understanding capabilities of a closed-source Multi-modal Large Language Model (MLLM), i.e., Gemini-2.5-Pro, an effective pipeline is developed for reliable benchmark construction and streamlined model assessment. Moreover, to further facilitate community use, we train a robust evaluation model that enables offline assessment of T2I model outputs. Through comprehensive benchmarking of both open- and closed-sourced T2I models, we systematically reveal their strengths and weaknesses across various aspects.