Can AI Automate Epoch’s Work?

Summary

Epoch evaluated six AI models on 11 tasks drawn from its own work, including graphic design, data insight and explorer creation, AI data-center research, and research design. Models received substantial context and tool access, then completed tasks without further intervention; Epoch employees manually graded the results against task-specific rubrics. Claude Fable 5.1 and GPT-6 Astra led overall, mainly because they were reliable on well-defined coding, computational, and computer-use tasks. However, even these frontier closed-weight models struggled to infer implicit standards from examples, choose broadly useful topics, design experiments that measured their stated questions, and interpret flawed results appropriately. Open-weight models lagged further behind, sometimes making basic factual or data-filtering errors that closed-weight models avoided. Fable 5.1 also tended to produce unnecessarily information-dense writing and graphics, while several models converged on similar ideas despite open-ended prompts. Epoch concludes that current models cannot yet automate its work end to end, although it plans to expand the benchmark as new models are released. The authors caution that the results are noisy because each model ran once, grading was partly subjective, tasks were specific to Epoch, and different agent harnesses may have affected performance.