How to Test Agentic AI Tools Beyond the Final Diff

Summary

This technical essay argues that a passing test suite and correct final diff are insufficient evidence that an agentic coding workflow completed its delegated requirements autonomously. Using a synthetic API-endpoint task, it shows how an agent first edited an API incompatibly, was redirected to the current architectural decision, passed an approved contract test, and reported completion without saving a required migration note. The final result became usable after human correction and a reminder, but the run did not demonstrate that the workflow could succeed without those interventions. The author recommends deciding before a run which decisions, reviews, approvals, and delegated requirements belong to the human or the agent, then marking the first corrective intervention as the boundary between the initial attempt and a continuation. Human actions should be classified by role: deliberately reserved decisions, planned reviews, repairs to delegated work, environment assistance, or repairs to saving and handoff. The evidence should be kept in three related but distinct forms: the starting request and context, an activity record of observable tool calls, messages, approvals, and errors, and the final files, diff, test output, and revision or content hashes. Activity logs cannot prove that an agent never saw information through another route, while final snapshots cannot prove what happened between them; the article therefore emphasizes recording coverage gaps and reviewed revisions. Tests themselves must also be checked: a missing or poorly exercised requirement can make a precise evaluation invalid. The author notes that one run exposes a failure path but cannot estimate reliability, and that repeated workflow changes cannot isolate which change caused improvement. A discussion of permission prompts extends the same caution: silent execution or a visible prompt does not by itself establish the internal cause, so controlled comparisons are needed. These principles shaped the Work Ledger, which routes agents to current decisions, source material, attempted changes, and saved artifacts while retaining limits and reasons to revisit consequential conclusions. The Ledger uses plain-text instructions and configuration, with no required executable in version one.