Forecast-Dojo Introduces Replayable Environments for LLM Forecasting Agents

Summary

Forecast-Dojo is a replayable environment for benchmarking and training large language model forecasting agents. It combines 1,568 resolved Polymarket events with 18.8 million dated news articles, letting an agent research an event and revisit its forecast at successive historical dates. Because the tasks, tools, and outcomes are recorded, researchers can evaluate agents repeatedly, collect training interactions, and provide outcome feedback without waiting for new events to resolve. The events are divided chronologically into training and evaluation periods. In tests covering 12 models, research tools reduced Brier scores for every model, and forecasts generally improved as events unfolded, with the largest gains occurring when more newly dated evidence was available. However, all models remained behind historical market forecasts in both Brier score and accuracy. A belief notebook carried across dates reduced research costs but did not consistently improve forecast quality. The release also includes interaction trajectories and outcome feedback for agent learning, with supervised fine-tuning presented as a proof of concept.