Paper 2601.04745
KnowMe-Bench: Benchmarking Person Understanding for Lifelong Digital Companions
- Published
- Jan 2026
- Research lab
- Independent
- Citations
- 8
- GitHub
- 133 stars
01 In brief
Summary
KnowMe-Bench is a benchmark for evaluating person understanding in lifelong digital companions, built from long-form autobiographical narratives rather than sparse chat logs.
It addresses two gaps in existing benchmarks: evaluation misalignment (retrieval proxies vs.
true person understanding) and data substrate misalignment (low-density, decontextualized traces).
The benchmark reconstructs narratives into a flashback-aware, time-anchored cognitive stream with five textual modalities (visual, auditory, context, background, mind) and evaluates models with evidence-linked questions across three tiers: factual recall, subjective state attribution, and principle-level reasoning.
Experiments on 2,580 queries from three narrative datasets (Knausgård, Neapolitan Novels, Proust) show that retrieval-augmented systems improve factual accuracy but fail on temporally grounded explanations and higher-level inferences.
Chronological logging (MemOS) outperforms entity-state memory (Mem0) on chronology-sensitive tasks, while reflective memory (A-Mem) boosts insight but harms grounding.
Human experts score 96.5/88.0/83.5 on Levels I/II/III versus 75.4/62.5/22.6 for the best model, highlighting the need for memory mechanisms beyond retrieval.
02 From the paper
Abstract
Existing long-horizon memory benchmarks mostly use multi-turn dialogues or synthetic user histories, which makes retrieval performance an imperfect proxy for person understanding. We present \BenchName, a publicly releasable benchmark built from long-form autobiographical narratives, where actions, context, and inner thoughts provide dense evidence for inferring stable motivations and decision principles. \BenchName~reconstructs each narrative into a flashback-aware, time-anchored stream and evaluates models with evidence-linked questions spanning factual recall, subjective state attribution, and principle-level reasoning. Across diverse narrative sources, retrieval-augmented systems mainly improve factual accuracy, while errors persist on temporally grounded explanations and higher-level inferences, highlighting the need for memory mechanisms beyond retrieval. Our data is in \href{KnowMeBench}{https://github.com/QuantaAlpha/KnowMeBench}.