How to Avoid the Hidden Risks of AI-Generated Code

Summary

AI has lowered the barrier to building software, but Quint CEO Gabriela Moreira argues that production code still requires human judgment. AI can produce large amounts of potentially incorrect code, creating a cognitive gap that makes review and monitoring harder than with ordinary human errors. Concurrent, distributed and non-deterministic systems are especially exposed because they are difficult to test and can contain too many paths for teams to verify. Moreira recommends combining defenses rather than relying on a single safeguard: trustworthy behavioral tests, formal models where useful, mutation testing to measure whether tests catch regressions, monitoring and formal verification when feasible. She is particularly concerned about designs captured mainly in unexecutable markdown, because important flaws may remain hidden until an AI system has generated thousands of lines of implementation. The appropriate level of oversight depends on the system’s risks, but teams should aim to ship with enough confidence that changes do not break working behavior. When AI-generated software becomes too difficult to maintain, her suggested recovery path resembles legacy-code remediation: establish enough tests to regain confidence, then make changes incrementally.