GrowPage Dynamically Budgets KV Cache for Efficient LLM Reasoning Serving

Summary

Long-output reasoning makes the key-value (KV) cache a major memory bottleneck in large language model serving. The paper argues that existing KV-compression methods generally use a predefined budget for each request and only change which KV states are retained, while reasoning requests can require different capacities and can change their attention demand during generation. GrowPage treats KV capacity as a runtime resource rather than a fixed allocation. It maintains lightweight query summaries at recent and long-term timescales, then compares their attention working sets to estimate how demand is evolving. When decoding reaches a capacity boundary, the framework either compresses KV states within the current allocation or acquires another physical page when broader demand appears. The design integrates with PagedAttention’s page-level memory abstraction, preserving continuous batching and prefix caching. Experiments on reasoning benchmarks across multiple models report a better performance-throughput trade-off than existing approaches, although the abstract does not provide numerical results or detailed experimental settings.