Paper 2607.02255
AgenticSTS: A Bounded-Memory Testbed for Long-Horizon LLM Agents
- Published
- Jul 2026
- Research lab
- Independent
- Citations
- 0
- GitHub
- 123 stars
01 In brief
Summary
The paper introduces AgenticSTS, a bounded-memory testbed for long-horizon LLM agents, instantiated in the game Slay the Spire 2.
Instead of appending raw transcripts, the agent composes each decision prompt from five typed layers (L1-L5): fixed protocol, state schemas, game rules, episodic summaries, and triggered strategic skills.
This contract keeps context bounded and allows individual layer ablation.
In a fixed-A0 ablation (N=10 per cell), the no-scaffold baseline won 3/10 games, while enabling the L5 skill layer (mode-a, mode-b, full-frozen) each won 6/10; this difference is directional (Fisher exact p≈0.37) rather than statistically significant.
A cross-backbone probe showed the frozen L4+L5 stack is backbone-sensitive (Qwen score +84.5%, DeepSeek -18.1%).
An auto-mode ladder reached A6-A8 with postrun memory, versus A2-A4 without.
Comparison with two open-source accumulating-context agents (STS2MCP, CharTyr) showed they won 0/5 games, consumed 66-90x more fresh tokens per score point, and took ~4x longer per floor.
The release includes 298 completed trajectories, condition tags, frozen memory/skill snapshots, prompt records, and analysis scripts.
02 From the paper
Abstract
Memory for a long-horizon LLM agent is a contract about what each future decision is allowed to see. The simplest contract appends past observations, tool calls, and reflections to every prompt, which makes prior context easy to access but also turns it into a jumbled mixture in which the effect of any single memory component is hard to isolate. We introduce and instrument an alternative bounded contract: every decision is made from a fresh user message assembled by typed retrieval, with no raw cross-decision transcript appended. The prompt thus stays bounded across runs of any length, and any single layer can be ablated in isolation. We instantiate the contract in Slay the Spire 2, a closed-rule stochastic deck-building game whose runs require hundreds of tactical and strategic decisions. A public online benchmark of frontier LLMs on the same game reports zero wins at the lowest difficulty across five configurations, and the developer-reported human win rate at the same difficulty is 16%; the task is hard but not saturated. Within our harness, a fixed-A0 ablation shows the largest observed difference when triggered strategic skills are enabled: the no-store baseline wins 3/10 games and adding the skill layer 6/10. At this sample size the comparison is directional rather than statistically decisive (Fisher exact p\approx0.37); a cross-backbone probe and public accumulating-context baselines are reported as operational comparisons rather than controlled tests of the contract variable itself. We release a reproducible testbed: 298 completed trajectories with condition tags, frozen memory/skill snapshots, prompt records, and analysis scripts -- an agent design and a validated, reusable methodology for studying how explicit memory layers shape long-horizon LLM-agent decisions.