MERIT Evaluates the Cost and Utility of Long-Term Memory in Tool-Using LLM Agents

Summary

The paper introduces MERIT, a benchmark and evaluation harness for measuring the marginal utility of long-term memory in agents that execute tasks with tools. Unlike conversational recall benchmarks such as LoCoMo and LongMemEval, MERIT tests whether remembered information changes an agent’s actions. It contains episodic tool-use tasks across three domains, verifies dependence on earlier episodes with an automated leak check, includes a difficulty ladder ending in updated-fact recall, supports controlled memory corruption, and meters every memory operation in tokens and dollars. Across 23,440 scored episodes costing $42.57, a two-generation pilot on gpt-4.1-mini and a preregistered three-model, three-seed grid found that memory raised success on dependent tasks from a leak-verified baseline of 0.00 to 0.55–1.00. Updated facts exposed a sharp reliability gap: embedding retrieval ranged from 0.30 to 0.95 across models, with a maximum seed gap of 0.45, and agents used a correctly retrieved value only 55% of the time. Update-on-write approaches, including a structured fact store and LLM summarization, reached 0.70–1.00, while the hybrid underperformed the fact store alone. A clean full-replay-gated spot check with Claude Sonnet 5 reproduced the pattern. Changing the memory implementation shifted task success by as much as 60 percentage points. Full replay was never economical; the best condition in each domain delivered 2.7–3.9 times its marginal utility per dollar. The authors release the benchmark, harness, and all traces.