FDE-Gym Tests AI Agents on Real-World Customer Delivery

Summary

FDE-Gym is a benchmark designed to test whether AI agents can deliver working solutions for real customers, rather than only solve clearly specified issue reports or task tickets. It contains 95 end-to-end tasks drawn from 20 customer scenarios across 18 sectors and adapted from real-world delivery projects by industry experts. In simulated customer environments, agents must clarify ambiguous requirements, interact with scripted stakeholders, and modify production-like customer systems. Each delivery is scored by the customer’s normalized success-metric improvement, from 0 for no change to 1 for a reference solution, with penalties for messages, trial runs, and waiting time. The benchmark also provides an online practice platform for humans using the same criteria. Across 13 model configurations from seven providers, the best configuration achieved a mean score of 0.56 and fully resolved only 14 of 95 tasks, defined as a score of at least 0.8 without an incident. Discovering unstated constraints was the main bottleneck, but information gathering alone did not explain the gap: even on 23 tasks where the best configuration obtained every key stakeholder fact, it reached a score of at least 0.8 on only 10. The results indicate that strong performance on well-specified tasks does not yet translate into reliable customer delivery.