Agent Incident Registry Maps AI Agent Failures and Evaluation Gaps

Summary

AI agents increasingly operate through tools and delegated authority, but general incident databases often lack the detail needed to compare real failures with agent-security evaluations. The paper introduces the Agent Incident Registry (AIR), a source-linked catalog of disclosed agent-related events. Each record has supporting evidence, a stable identifier, and labels that account for missing information about causal role, disclosure class, mechanism, and outcome. Among the primary generative-system records in which an agent acted, the abstract reports that a specified share involved realized harm. Those outcomes are concentrated in in-the-wild and safety-failure records, while responsible disclosures and research demonstrations are overwhelmingly demonstrations. The authors therefore caution that the aggregate share reflects the registry’s collection composition, not deployment risk. After initial curation, a second human reviewer checked every record and its existing labels for completeness and correctness. In a deployment-analogue audit, all of InjecAgent’s specified cases fell on three of AIR’s twelve surfaces and were attacker-triggered, while AIR also contained a specified number of no-adversary safety failures. AIR is intended for source-grounded case retrieval and auditing evaluation scope, not for estimating failure rates or control effectiveness.