ERPBench Evaluates Computer-Use Agents on Enterprise ERP Workflows

Summary

ERPBench is a benchmark for evaluating computer-use agents that operate enterprise resource planning software through screenshots and simulated actions. The authors argue that ERP systems create distinct challenges because they combine dense interfaces, coordinated multi-step workflows, and actions that can change persistent business records. The benchmark runs agents on a live, reproducible ERP system and scores each task against ground-truth values stored in its database, avoiding reliance on proprietary platforms or simplified simulations. The paper also introduces a production-grade deployment harness that places human approval gates before agent actions, although ERPBench runs the benchmark autonomously. In evaluations of six closed and open-source agents, strong performance on general graphical user-interface tasks did not translate into enterprise reliability. An agent could reach the correct form and save it while still writing an incorrect value to the database: across the reported systems, save rates reached 85% of runs, but correct values were as low as 3%. The study further characterizes failure modes specific to enterprise workflows, highlighting why visible task completion is insufficient for judging agents that modify business data.