Why Autonomous Agents Still Fail at Knowing When They Are Wrong

Summary

A five-month log of 84 incidents in a small business run partly by autonomous agents argues that reliability is shifting from code failure to inference failure. The agents wrote code, analyzed nightly data, published content, adjusted ad budgets, and influenced real transactions. Only 10 incidents were caught by automated mechanisms: two by deterministic gates and eight by errors that surfaced during execution. Twenty-eight were found because a person distrusted a number, one followed an operator report, and 45 emerged accidentally during unrelated investigations. As the catalogue grew from 18 to 84 incidents, the automated detection share remained between 9% and 15%, despite new tests and checks. The author says early failures such as syntax errors, missing indexes, and malformed strings are easy for conventional tooling to catch. More recent failures involve correct queries and data being used to produce false conclusions, such as treating ad-attributed conversions as total sales, comparing lifetime ad spend with current targeting after historical edits, or declaring a channel inactive despite its own records showing substantial sales. These failures passed repository checks because the checks did not test whether the conclusions were true. The log itself was later found to contain errors, including an incorrect claim about an unbuilt fix and a machine-generated traffic source counted as human activity. The author warns that the figures describe one business and are heavily shaped by accidental discovery, so the mechanisms matter more than the percentages. The recommended practices are to classify incidents by how they were detected, define automated catches strictly, treat missing data as an unresolved question, and record corrections inline. The central conclusion is that the systems reliably report when machinery breaks, but have little ability to establish whether the resulting analysis is correct.