The Year in AI Papers
The year/Independent research

Paper 2510.04849

When Models Lie, We Learn: Multilingual Span-Level Hallucination Detection with PsiloQA

Elisei Rykov, Kseniia Petrushina, Maksim Savkin, Valerii Olisov, Artem Vazhentsev, Kseniia Titova, Alexander Panchenko, Vasily Konovalov, Julia Belikova

Published
Oct 2025
Research lab
Independent
Citations
10
GitHub
Not linked

01 In brief

Summary

The paper introduces PsiloQA, a large-scale multilingual dataset for span-level hallucination detection in LLMs, covering 14 languages.

It is built via an automated pipeline: generating QA pairs from Wikipedia using GPT-4o, eliciting answers from diverse LLMs without context, annotating hallucinated spans with GPT-4o, and filtering low-quality samples.

The dataset contains 63,792 training samples, with English being the most prevalent language.

Evaluation of detection methods shows that fine-tuned encoder models, especially mmBERT, outperform uncertainty quantification and LLM-based approaches.

PsiloQA demonstrates strong cross-lingual transfer and knowledge transfer to other benchmarks, outperforming human-annotated RAGTruth while being over 17 times cheaper to produce.

Manual verification confirms GPT-4o's annotation reliability.

Limitations include reliance on GPT-4o, task narrowness to QA, and uneven language distribution.

The dataset is released under CC-BY-4.0, and the code is available on GitHub.

The work highlights the feasibility of synthetic data for improving hallucination detection in multilingual settings.

Future work will extend the pipeline to other generation tasks like summarization and data-to-text generation.

The paper also discusses ethical considerations, including data privacy, intended use, and fairness.

The findings suggest that multilingual training on PsiloQA enables superior cross-lingual generalization compared to language-specific training.

The gap between AP and IoU metrics indicates that precise span boundary detection remains a challenge.

The dataset supports robust knowledge transfer, with PsiloQA-trained models showing significant gains on Mu-SHROOM and HalluEntity benchmarks.

The paper concludes that synthetic datasets with automated annotations can improve LLM robustness and factuality.

The authors plan to explore extending the pipeline to other tasks in…

02 From the paper

Abstract

Hallucination detection remains a fundamental challenge for the safe and reliable deployment of large language models (LLMs), especially in applications requiring factual accuracy. Existing hallucination benchmarks often operate at the sequence level and are limited to English, lacking the fine-grained, multilingual supervision needed for a comprehensive evaluation. In this work, we introduce PsiloQA, a large-scale, multilingual dataset annotated with span-level hallucinations across 14 languages. PsiloQA is constructed through an automated three-stage pipeline: generating question-answer pairs from Wikipedia using GPT-4o, eliciting potentially hallucinated answers from diverse LLMs in a no-context setting, and automatically annotating hallucinated spans using GPT-4o by comparing against golden answers and retrieved context. We evaluate a wide range of hallucination detection methods -- including uncertainty quantification, LLM-based tagging, and fine-tuned encoder models -- and show that encoder-based models achieve the strongest performance across languages. Furthermore, PsiloQA demonstrates effective cross-lingual generalization and supports robust knowledge transfer to other benchmarks, all while being significantly more cost-efficient than human-annotated datasets. Our dataset and results advance the development of scalable, fine-grained hallucination detection in multilingual settings.