Google outlines behavioral evaluations for reliable AI coding agents

Summary

Google Developers argues that end-to-end benchmarks such as Terminal-Bench and DeepSWE are useful for measuring overall agent performance but often do not explain why a result changed. It recommends behavioral evaluations that test discrete, observable actions, such as asking clarifying questions for ambiguous requests, running a validator before completing a build-file change, or consulting web search instead of guessing live information. The article says teams should begin with developer instinct and dogfooding, then introduce evaluations once an agent can handle routine work and needs protection against regressions. Its proposed harness uses fast, deterministic, unit-style checks for intermediate tool calls and file changes. Simple tasks can use strict milestone assertions, while complex tasks may need outcome-based checks or an LLM judge. Because model behavior is nondeterministic, batch runs and aggregate pass rates are preferred to blocking development on a single evaluation. Behavioral tests can also support automated prompt iteration while preserving existing behaviors. Google emphasizes that these micro evaluations complement rather than replace end-to-end benchmarks: the larger tests assess the final result, while behavioral checks provide a feedback loop for safe, rapid harness iteration.