Trustbut Adds Evidence Checks for AI Coding Agent Changes

Summary

Trustbut is an open-source local command-line tool for checking whether a code change is actually supported by test evidence, whether the change was written by an AI coding agent or a person. It runs tests in a temporary clean Git worktree, reruns new or changed tests against the pre-change code, and reports whether they fail before the change and pass afterward. Tests that pass both before and after are listed as insufficient proof. The tool also detects skipped or focused tests, removed tests and assertions, modified snapshots or test configuration, and code that suppresses type, lint, or coverage checks. It produces a Markdown or JSON receipt with PASS, WARN, or FAIL verdicts, plus a ranked list of files such as authentication, dependencies, CI, infrastructure, and agent configuration that deserve review. A hook can send a FAIL result back to Claude Code, Codex, Gemini CLI, or another supported agent for additional fixes; OpenCode instead receives a follow-up prompt because it cannot block completion. Trustbut supports 11 languages and multiple test runners, requires Node.js 22 or newer, Git, and the project toolchain, and does not use an LLM, server, account, telemetry, or network call in its core. Its GitHub Action can post a sticky pull-request comment and job summary, while fork pull requests receive the summary only. In a pilot covering 201 merged pull requests, trustbut ran on 109: 53 agent-authored and 56 human-authored. No observed run showed a previously passing test failing after the change. Among agent PRs, 97% of new or changed tests failed without the change, compared with 88% for human PRs; 12 of 53 agent PRs and 15 of 56 human PRs had at least one finding. The author says these merged-PR results do not show what happens before an agent claims a task is complete, and the effectiveness of the agent hook has not yet been measured. In an earlier pilot, only five of 71 labeled findings were considered worth reviewer action, while nine were incorrect; the labels were draft, Claude-generated, and not independently confirmed.