How Revefi Cut Its AI Data Agent’s Spend by 90% Through Prompt Caching

Summary

Revefi describes how it reduced spending for Raden, its AI data agent, after a prompt-generation change caused prefix caching to fail. Raden uses LLMs, algorithmic optimizations, and unsupervised machine-learning models to manage data-warehouse tasks, so repeated long agent sessions made inefficient caching expensive. The company found that a timestamp inserted near the top of the system prompt changed on every LLM call; because prefix caching only reuses an unchanged prefix, the conversation history and other content after the timestamp were billed at the regular input rate. Revefi moved the user context and current time into a system-reminder suffix, placing stable tools, instructions, and conversation history first. After the change, Anthropic Console showed cache reads reaching 99% of input tokens, although Revefi did not observe a corresponding latency improvement because tool calls dominated end-to-end time. Revefi says its broader monitoring found similar issues in customer agents, including Anthropic requests where cache controls had not been enabled; one customer could save approximately $300,000 annually on a single internal agent. The article explains that Anthropic requires explicit caching controls, while OpenAI and Gemini provide automatic caching under stated conditions. It also warns that caching is not always beneficial: one-off requests can incur unnecessary write costs, TTL selection affects pricing, and excessive ephemeral content can make agent behavior harder to understand. Revefi recommends checking provider dashboards or using per-agent, per-model cache-efficiency monitoring.