Paper 2602.12783
SQuTR: A Robustness Benchmark for Spoken Query to Text Retrieval under Acoustic Noise
- Published
- Feb 2026
- Research lab
- Independent
- Citations
- 1
- GitHub
- 10 stars
01 In brief
Summary
SQuTR is a benchmark for evaluating spoken query to text retrieval under controlled acoustic noise.
It aggregates 37,317 unique queries from six English and Chinese text retrieval datasets (FiQA, HotpotQA, Natural Questions, MedicalRetrieval, DuRetrieval, T2Retrieval), synthesizing speech with CosyVoice-3 using 200 speakers.
Four acoustic conditions (Clean, Low Noise at 20dB, Medium Noise at 10dB, High Noise at 0dB) are created by mixing 17 categories of real-world noise from DEMAND and NOISEX-92.
The benchmark includes 149,268 evaluation instances and uses nDCG@10, MRR@10, and Recall@10 as metrics.
Experiments evaluate cascaded systems (ASR + retrieval) with multiple ASR models (Whisper variants, Paraformer, etc.) and 12 retrieval backends (BM25, BGE, Qwen3-Embedding, etc.), plus an end-to-end system (Omni-Embed-Nemotron-3B).
Results show retrieval performance degrades as noise increases, with dense retrievers showing higher variance than BM25.
Larger models like Qwen3-Embedding-8B are more stable, but even they underperform text-only baselines under noise.
The choice of retrieval backend is more critical than ASR size, as dense retrieval compensates for ASR errors.
SQuTR provides a reproducible testbed for benchmarking robustness in spoken query retrieval.
02 From the paper
Abstract
Spoken query retrieval is an important interaction mode in modern information retrieval. However, existing evaluation datasets are often limited to simple queries under constrained noise conditions, making them inadequate for assessing the robustness of spoken query retrieval systems under complex acoustic perturbations. To address this limitation, we present SQuTR, a robustness benchmark for spoken query retrieval that includes a large-scale dataset and a unified evaluation protocol. SQuTR aggregates 37,317 unique queries from six commonly used English and Chinese text retrieval datasets, spanning multiple domains and diverse query types. We synthesize speech using voice profiles from 200 real speakers and mix 17 categories of real-world environmental noise under controlled SNR levels, enabling reproducible robustness evaluation from quiet to highly noisy conditions. Under the unified protocol, we conduct large-scale evaluations on representative cascaded and end-to-end retrieval systems. Experimental results show that retrieval performance decreases as noise increases, with substantially different drops across systems. Even large-scale retrieval models struggle under extreme noise, indicating that robustness remains a critical bottleneck. Overall, SQuTR provides a reproducible testbed for benchmarking and diagnostic analysis, and facilitates future research on robustness in spoken query to text retrieval.