When LLM Agents Implement Systems: Defects and Evaluation Lessons

Summary

This case study examines an LLM coding agent implementing a multi-component data system. Researchers catalogued five defects and evaluated filtered versus unfiltered retrieval on 100 HotpotQA questions. Filtered recall reached its ceiling at a budget of three, while unfiltered search recovered all required evidence only 69% of the time at a budget of ten. The study also identifies a validation gap where a claimed performance fix was not re-measured on the motivating regression.