The Year in AI Papers
The year/Independent research

Paper 2603.12572

LMEB: Long-horizon Memory Embedding Benchmark

Xinping Zhao, Xinshuo Hu, Jiaxin Xu, Danyu Tang, Xin Zhang, Mengjia Zhou, Yan Zhong, Yao Zhou, Zifei Shan, Meishan Zhang, Baotian Hu, Min Zhang

Published
Mar 2026
Research lab
Independent
Citations
7
GitHub
14 stars

01 In brief

Summary

The paper introduces LMEB, a benchmark for evaluating embedding models on long-horizon memory retrieval tasks, which are underexplored in existing benchmarks like MTEB.

LMEB comprises 22 datasets and 193 zero-shot retrieval tasks across four memory types: episodic, dialogue, semantic, and procedural.

The authors evaluate 15 embedding models, ranging from 239M to 12B parameters.

Key findings include: (1) LMEB offers a reasonable difficulty level, with the best model (bge-multilingual-gemma2) achieving a Mean (Dataset) NDCG@10 of 61.41 in the w/ inst.

setting; (2) larger models do not always outperform smaller ones, as seen with KaLM-Embedding-Gemma3 and bge-multilingual-gemma2 underperforming smaller models like EmbeddingGemma-300M in the w/o inst.

setting; (3) LMEB and MTEB are largely orthogonal, with Pearson and Spearman correlations of -0.115 and -0.130, respectively, indicating they assess complementary capabilities.

The benchmark provides a standardized, extensible framework with a unified data format and open-source code, aiming to fill a gap in memory embedding evaluation and support the development of memory-augmented systems.

02 From the paper

Abstract

Memory embeddings are crucial for memory-augmented systems, such as OpenClaw, but their evaluation is underexplored in current text embedding benchmarks, which narrowly focus on traditional passage retrieval and fail to assess models' ability to handle long-horizon memory retrieval tasks involving fragmented, context-dependent, and temporally distant information. To address this gap, we introduce the Long-horizon Memory Embedding Benchmark (LMEB), a comprehensive framework for evaluating embedding models on complex, long-horizon memory retrieval. LMEB comprises 22 datasets and 193 zero-shot retrieval tasks spanning four memory types: episodic, dialogue, semantic, and procedural. These memory types differ in terms of level of abstraction and temporal dependency, capturing distinct aspects of memory retrieval that reflect the diverse challenges of the real world. We evaluate 15 widely used embedding models, ranging from hundreds of millions to ten billion parameters. The results reveal that (1) LMEB provides a reasonable level of difficulty; (2) Larger models do not always perform better; (3) LMEB and MTEB measure orthogonal capabilities. This suggests that the field has yet to converge on a universal model capable of excelling across all memory retrieval tasks, and that strong performance on traditional passage retrieval does not necessarily transfer to long-horizon memory retrieval. LMEB provides a standardized and reproducible framework that fills a key gap in memory embedding evaluation and supports future advances in long-term, context-dependent retrieval.