The Year in AI Papers
The year/Independent research

Paper 2508.16153

Memento: Fine-tuning LLM Agents without Fine-tuning LLMs

Huichi Zhou, Yihang Chen, Siyuan Guo, Xue Yan, Kin Hei Lee, Zihan Wang, Ka Yiu Lee, Guchun Zhang, Kun Shao, Linyi Yang, Jun Wang

Published
Aug 2025
Research lab
Independent
Citations
87
GitHub
2.6K stars

01 In brief

Summary

The paper introduces Memento, a learning paradigm for LLM agents that enables continual adaptation without fine-tuning the underlying LLM.

It formalizes a Memory-augmented Markov Decision Process (M-MDP) with a neural case-selection policy, storing past experiences in an episodic case bank.

The policy is updated via online soft Q-learning, with retrieval guided by either non-parametric similarity or a parametric Q-function.

Memento achieves top-1 on GAIA validation (87.88% Pass@3) and 79.40% on the test set, 66.6% F1 and 80.4% PM on DeepResearcher, and 95.0% PM on SimpleQA, outperforming training-based baselines.

Case-based memory adds 4.7–9.6 absolute points on out-of-distribution tasks.

Ablations show that a small, curated memory (K=4) is optimal, and both parametric and non-parametric CBR contribute to gains.

The framework uses a planner–executor architecture with GPT-4.1 as planner and o3/o4-mini as executor, integrating MCP tools for search, crawling, and multimodal processing.

The approach offers a scalable, efficient pathway for continuous learning without gradient updates, advancing open-ended skill acquisition in deep research scenarios.

Code is available at https://github.com/Agent-on-the-Fly/Memento.

02 From the paper

Abstract

In this paper, we introduce a novel learning paradigm for Adaptive Large Language Model (LLM) agents that eliminates the need for fine-tuning the underlying LLMs. Existing approaches are often either rigid, relying on static, handcrafted reflection workflows, or computationally intensive, requiring gradient updates of LLM model parameters. In contrast, our method enables low-cost continual adaptation via memory-based online reinforcement learning. We formalise this as a Memory-augmented Markov Decision Process (M-MDP), equipped with a neural case-selection policy to guide action decisions. Past experiences are stored in an episodic memory, either differentiable or non-parametric. The policy is continually updated based on environmental feedback through a memory rewriting mechanism, whereas policy improvement is achieved through efficient memory reading (retrieval). We instantiate our agent model in the deep research setting, namely \emph{Memento}, which attains top-1 on GAIA validation ($87.88\%$ Pass@$3$) and $79.40\%$ on the test set. It reaches $66.6\%$ F1 and $80.4\%$ PM on the DeepResearcher dataset, outperforming the state-of-the-art training-based method, while case-based memory adds $4.7\%$ to $9.6\%$ absolute points on out-of-distribution tasks. Our approach offers a scalable and efficient pathway for developing generalist LLM agents capable of continuous, real-time learning without gradient updates, advancing machine learning towards open-ended skill acquisition and deep research scenarios. The code is available at https://github.com/Agent-on-the-Fly/Memento.