How AI Agents and Developers Could Share Software Correctness

Summary

The essay models ordinary software development as nested optimization around an implicit and changing software intent. When code fails, an AI agent can reproduce the failure, add a test, and modify the code, turning observed errors into executable constraints. The author distinguishes test-suite completeness, which measures how much intended behavior is covered, from soundness, which measures whether passing tests actually provides evidence that the software is correct. Fast unit and lightweight integration tests are cheaper but observe less of the system, while slow end-to-end and integration tests are closer to production. The proposed inner loop batches candidate branches on a temporary integration branch, uses slow tests to find defects missed by fast tests, and transfers the underlying constraints back into fast tests on the relevant branches. Runtime failures can therefore feed a pipeline from production to slow tests and then to fast tests, increasing coverage without running the slow suite on every pull request. The outer loop belongs to the developer, who reviews whether each test represents the real requirement and whether the fix addresses that requirement rather than merely making the test pass. The author argues that AI can drive completeness upward mechanically, while human understanding is needed to preserve soundness and detect repeated symptoms of deeper problems. Because product intent changes with requests, markets, users, and dependencies, software quality remains a moving target. The essay concludes that unsupervised AI-driven development may eventually capture many observed failures while encoding increasingly unreliable assumptions, making the test suite more complete but less trustworthy. It presents the balance among AI-generated changes, developer review, and external pressure for updates as a condition for continued product evolution.