RunBoth Detects Behavioral Differences in AI-Changed Code

Summary

RunBoth is a Python developer tool for checking whether a code change alters runtime behavior, including behavior in functions that were not edited. It checks out both versions from Git history, generates inputs from changed function signatures and constants found in bytecode, runs the versions in separate sandboxed subprocesses, and compares return values, exceptions, warnings, stdout, stderr, argument mutation, and object state. When it finds a difference, it reports a separating input and the old and new results. The tool has three outcomes: changed with a witness, no_change within the input budget, and abstained when checking was not possible; it does not treat no difference as proof of safety. A red-team evaluation across eight public repositories covered 2,548 functions and reported zero false positives, while abstention rates ranged from 0% to 14.3% and median runtime per commit ranged from 0.2 to 192.8 seconds. A separate 22-function adversarial corpus also produced no false positives. RunBoth is Python-only at the function level, samples concrete inputs rather than proving behavior, and abstains on nondeterministic, slow, or unconstructible functions. Its sandbox blocks network access and filesystem writes but is not a security boundary against hostile code. It can run as a commit-message hook or GitHub Action, and the repository says the tool uses no AI model and has no dependencies.