Temporal Aggregation Is Crucial for Aggressive KV Cache Eviction

Summary

This paper studies decoding-time key-value (KV) cache compression, focusing on a design choice that is often treated as an implementation detail: how token scores are aggregated across decode steps. Under aggressive compression, exponential moving average (EMA) aggregation makes scorer changes that largely preserve token order produce nearly indistinguishable eviction sets. Value-norm and entropy scorers remain highly correlated with attention and retain almost the same tokens, while KeyDiff, key norm, recency, and a learned scorer change the ranking and substantially degrade. The authors link this stability to the evaluated aggregation rule, which jointly affects layer weighting and temporal retention. They introduce InertiaKV, an EMA-based eviction method, and InertiaKV-Lazy, which refreshes scores periodically rather than at every step. InertiaKV-Lazy reaches 1.34-1.46 times the decoding throughput of full-refresh InertiaKV. The paper separately evaluates Score-Free decoding: it scores the full context only at the first decoding step, freezes that ranking, and removes all later scoring, with an average quality change of +0.03. Experiments span six open-weight backbones and the LongBench, LongBench-v2, and RULER benchmarks. The results identify temporal aggregation and ranking preservation as distinct and consequential factors in aggressive KV eviction. They do not establish that scorer quality is generally unimportant; the conclusion is specific to the evaluated compression and aggregation settings.