Gemini 4 Leads Tenzai’s Vulnerability Discovery Evaluation but Refuses More Exploits

Summary

Tenzai evaluated Google’s Gemini 4 through the Early Access Program against GLM 5.3 and Opus 5 across six controlled web application labs. The assessment measured vulnerability discovery and reproducible exploit generation in open-ended, black-box environments. Gemini 4 achieved the highest score for vulnerability discovery, which Tenzai describes as the hardest phase because it involves reconnaissance and attack-surface mapping. Its performance declined when it had to produce reproducible exploits: recall fell by 9 percentage points, compared with roughly 1–2 points for GLM 5.3 and Opus 5. Tenzai attributes the gap largely to Gemini’s higher rate of “soft refusals,” in which the agent declines work during normal operation rather than returning a formal safety block. Gemini also processed fewer tokens per assessment than the other tested models. Its tool-use pattern differed substantially, with more file operations and fewer Bash, browser, and code executions; file reads represented roughly 40% of its tool calls, compared with 10% for GLM 5.3 and 4% for Opus 5. The researchers say Gemini often relied on saved artifacts, repeated inspection, and verification before reporting, although one assessment included 108 reads of an API schema across 103 line-range requests. Tenzai concludes that Gemini 4 shows strong autonomous offensive-security capability, but fewer refusals and more consistent behavior would be needed for dependable results.