Speculative Macro Commit Cuts Tool-Using Agent Latency

Summary

Tool-using LLM agents lose wall-clock time in serial action-and-observation loops, where each tool call and environment transition delays the next decision. The paper introduces Speculative Macro Commit (SMC), a runtime mechanism that pairs a large authoritative actor with a faster speculative drafter running on an isolated environment snapshot. SMC mines recurring multi-action skeletons from training traces into a macro library, then matches them against action chains predicted by the drafter. When the actor’s next tool call matches the first drafted action, the system commits the remaining pre-executed actions and their observations to the official trajectory. With Qwen3.5-27B INT4 as actor and Qwen3.5-4B as drafter, SMC preserves the sequential agent’s overall accuracy. On the tau^2-Bench Telecom subset, it cuts latency by 10.23% versus Speculative Actions and 18.59% versus sequential execution. On AppWorld, it reduces wall time by 7.7% versus Speculative Actions and 44.9% versus sequential execution, but causes a small reduction in task completion. The authors present SMC as a way to extend speculative execution from single actions to reusable multi-step chains. Code is publicly available, according to the paper.