Study Finds Agentic AI Serving Must Optimize Complete Trajectories

Summary

Gimlet Labs, UIUC, and Intel studied ReAct-style agents running through Claude Code across five workload domains, comparing Gemma4-31B and Qwen3.6-27B in thinking and instant configurations. The study found that agent workloads are demanding for two distinct reasons: they may involve many model turns, large accumulated contexts, or both. Gemma Instant averaged 109 turns on ADE, while Qwen Thinking averaged 79K context tokens on SWE-bench Pro. When KV state remained resident, measured cache-hit rates ranged from 84.6% to 99.5%, and decode consumed 94.4% to 97% of LLM execution time. Under Terminal-Bench concurrency, cache hit rate fell from 98.85% at 15 agents to 16.78% at 20, while per-request throughput dropped about 66%, from 30.08 to 10.30 tokens per second at 25 agents. Tool behavior shifted from exploration toward execution as tasks progressed, providing a possible scheduling signal. Thinking increased output work, while failed runs accumulated substantially more context than successful ones. The authors conclude that serving systems need trajectory-level scheduling, cache management, accounting, safeguards, and benchmarks that model multi-turn state, tool waits, and long-tail demand rather than optimizing isolated requests.