When Should Forecasting Agents Reason? Reliability Stress Tests

Summary

Forecasting agents increasingly combine language-model reasoning, retrieval, ensembling, and calibration, but the reliability of these behaviors depends on the data source. This study evaluates binary forecasting tasks in a ForecastBench-style setting and treats the decision to retrieve, reason, defer to a market prior, or use a historical analog as an observable behavior. The authors find that no single mechanism consistently wins: structured analogs perform best for some data-generating processes, while market or crowd-style signals and conservative baselines are better for others. They introduce ReliabilityRoute, an intervention that routes behavior using historical coverage, market-prior availability, source-prior sharpness, evidence strength, evidence disagreement, and forecast horizon. A fixed rule fitted in 2024 closely matches a hand-built taxonomy without hard-coded source-name decisions. A walk-forward, self-adjusting rule refits its thresholds using previously resolved forecast vintages and achieves the best mean Brier score among the deterministic systems tested across 16 later LLM vintages. The improvement is modest, and historical and search baselines remain highly competitive. The authors therefore frame the work primarily as a behavioral stress test: more reasoning is not always better, and an agent should first estimate which evidence source deserves control. They argue that routing policies should adapt under auditable constraints, and they make reproducibility artifacts available through the linked GitHub repository.