Specification Frames Reduce Defects in LLM-Generated Backend Code

Summary

This paper evaluates whether a fixed specification frame can improve the security and correctness of code generated by large language models. The study focuses on backend tasks where defects in money arithmetic, time handling, retry safety, and access control are consequential, and distinguishes the frame from broad instruction files that have shown no benefit in earlier controlled work. The authors preregistered the hypotheses, refuters, analysis code, and one-shot generation procedure. They ran 50 realistic tasks drawn from finance, healthcare, and insurance practice through five frontier models from five vendor lineages, generating each task once with a bare prompt and once after a filled 267-word specification frame. Nine deterministic AST-based checkers evaluated the outputs, while the independent Bandit security scanner provided a separate assessment. The frame reduced defects for all five models, with mean reductions ranging from 0.16 to 0.70 findings per task; every Holm-adjusted sign test was significant and every bootstrap confidence interval excluded zero. When the two arms differed, the frame version was better in 95 of 100 comparisons, and no model became worse in any tested domain. Bandit identified 53 medium- or high-severity issues in the bare arm versus 11 in the frame arm, with the same direction for every model. The effect was largest where a model’s unprompted defaults were weakest, which the paper interprets as evidence that the frame supplies missing procedural discipline. The authors publish all 500 outputs, prompts, checkers, scoring code, and the preregistration with a DOI so the findings can be independently re-derived.