Paper 2510.12323
RAG-Anything: All-in-One RAG Framework
- Published
- Oct 2025
- Research lab
- Independent
- Citations
- 18
- GitHub
- 23K stars
01 In brief
Summary
RAG-Anything is a unified framework for multimodal Retrieval-Augmented Generation (RAG) that addresses the limitation of existing text-only RAG systems in handling real-world documents containing text, images, tables, and equations.
The framework introduces dual-graph construction, which builds a cross-modal knowledge graph for non-textual content and a text-based knowledge graph, merging them via entity alignment.
It employs cross-modal hybrid retrieval combining structural knowledge navigation with semantic similarity matching, and uses a vision-language model for synthesis.
Evaluated on DocBench and MMLongBench, RAG-Anything outperforms baselines (GPT-4o-mini, LightRAG, MMGraphRAG), with gains increasing on longer documents (e.g., over 13 points on DocBench for 100+ pages).
Ablation studies show graph construction is essential, while reranking provides marginal gains.
Case studies demonstrate precise handling of multi-panel figures and financial tables.
The paper also identifies challenges like text-centric retrieval bias and rigid spatial processing, suggesting future work on layout-aware parsing.
02 From the paper
Abstract
Retrieval-Augmented Generation (RAG) has emerged as a fundamental paradigm for expanding Large Language Models beyond their static training limitations. However, a critical misalignment exists between current RAG capabilities and real-world information environments. Modern knowledge repositories are inherently multimodal, containing rich combinations of textual content, visual elements, structured tables, and mathematical expressions. Yet existing RAG frameworks are limited to textual content, creating fundamental gaps when processing multimodal documents. We present RAG-Anything, a unified framework that enables comprehensive knowledge retrieval across all modalities. Our approach reconceptualizes multimodal content as interconnected knowledge entities rather than isolated data types. The framework introduces dual-graph construction to capture both cross-modal relationships and textual semantics within a unified representation. We develop cross-modal hybrid retrieval that combines structural knowledge navigation with semantic matching. This enables effective reasoning over heterogeneous content where relevant evidence spans multiple modalities. RAG-Anything demonstrates superior performance on challenging multimodal benchmarks, achieving significant improvements over state-of-the-art methods. Performance gains become particularly pronounced on long documents where traditional approaches fail. Our framework establishes a new paradigm for multimodal knowledge access, eliminating the architectural fragmentation that constrains current systems. Our framework is open-sourced at: https://github.com/HKUDS/RAG-Anything.