The Year in AI Papers
The year/Independent research

Paper 2601.09667

Collaborative Multi-Agent Test-Time Reinforcement Learning for Reasoning

Zhiyuan Hu, Yunhai Hu, Juncheng Liu, Shuyue Stella Li, Yucheng Wang, Zhen Xu, See-Kiong Ng, Anh Tuan Luu, Xinxing Xu, Bryan Hooi, Cynthia Breazeal, Hae Won Park

Published
Jan 2026
Research lab
Independent
Citations
3
GitHub
Not linked

01 In brief

Summary

MATTRL is a framework that improves multi-agent LLM reasoning at test time by injecting structured textual experience into multi-agent deliberation, avoiding costly weight updates.

It forms a team of specialist agents, retrieves relevant experiences from a pool, and reaches consensus.

The experience pool is built by scoring utterances with individual and decayed terminal rewards, then distilling high-value turns into textual entries.

Experiments on medicine (RareBench), math (HLE), and education (SuperGPQA) show MATTRL improves accuracy by 3.67% over multi-agent baselines and 8.67% over single-agent baselines.

Ablations compare credit assignment schemes (Naive, Difference Rewards, Shapley), finding Difference Rewards best for precision.

An adaptive router that selects between single-agent and MATTRL yields further gains.

Scaling analysis shows three agents are optimal for precision, while larger teams improve recall.

Few-shot prompting does not match MATTRL's gains, confirming the value of structured experience.

Limitations include inference-time compute and experience pool drift.

02 From the paper

Abstract

Multi-agent systems have evolved into practical LLM-driven collaborators for many applications, gaining robustness from diversity and cross-checking. However, multi-agent RL (MARL) training is resource-intensive and unstable: co-adapting teammates induce non-stationarity, and rewards are often sparse and high-variance. Therefore, we introduce \textbf{Multi-Agent Test-Time Reinforcement Learning (MATTRL)}, a framework that injects structured textual experience into multi-agent deliberation at inference time. MATTRL forms a multi-expert team of specialists for multi-turn discussions, retrieves and integrates test-time experiences, and reaches consensus for final decision-making. We also study credit assignment for constructing a turn-level experience pool, then reinjecting it into the dialogue. Across challenging benchmarks in medicine, math, and education, MATTRL improves accuracy by an average of 3.67\% over a multi-agent baseline, and by 8.67\% over comparable single-agent baselines. Ablation studies examine different credit-assignment schemes and provide a detailed comparison of how they affect training outcomes. MATTRL offers a stable, effective and efficient path to distribution-shift-robust multi-agent reasoning without tuning.