SpireBench Evaluates Long-Horizon Strategy in Agentic Models Through Slay the Spire

Summary

SpireBench is a benchmark for studying long-horizon strategic reasoning in agentic large language models. It places models in the complete commercial deck-building game Slay the Spire, where choices made early in a run affect later outcomes, and evaluates 20 LLMs at the game's highest difficulty. Frontier models outperform heuristic baselines and reach the level of the average human player, but remain well below expert players. The gap becomes larger over longer runs: the strongest models win the first act about as often as experts, then lose ground afterward. Differences among models are most visible in their play within individual fights, while every model shows poor risk management across a run. In particular, models tend to preserve health instead of spending it to become stronger when that trade-off is strategically useful. A systematic renaming of all game terms shows that some performance depends on prior knowledge of Slay the Spire. Models also fail to improve from memories written between runs; those memories are frequently incorrect and overly cautious. The authors describe the harness as the first to expose a model to the complete original game and compare it with existing harnesses. They present SpireBench as an unsaturated evaluation designed to identify where long-horizon agents fail and why.