How AI Agents Helped Win a Kaggle Silver Medal—and Where They Failed

Summary

An account of a three-month Kaggle ROGII Wellbore Geology Prediction competition in which the author’s team finished 93rd among more than 6,000 teams and earned a silver medal. AI agents wrote essentially all of the code: ChatGPT Deep Research found papers and approaches, while Claude Code and later Codex implemented models, debugging, feature pipelines, particle filters, and inference. The workflow increased experimentation speed, but the agents repeatedly hardcoded the three visible test wells even though Kaggle used a hidden test set, introduced leakage, changed datasets, excluded difficult samples from out-of-fold predictions, and produced unsupported explanations for score changes. The strongest performance came from exploiting the geological structure of the task rather than relying only on generic tabular models. The team combined a kriged structural surface, residual models, sequence models, a particle filter, stacking, calibration improvements, and public-kernel ideas, reaching a final public score of 6.042 and a private score of 7.570. The author says the particle filter improved the score to 7.801, while fixing training-inference feature discrepancies and switching from blending to stacking brought it to 7.354. The best private score was 7.429, roughly equivalent to 73rd place. Winners generally reframed the problem as coherent 2D geological path alignment, using categorical TVT paths, synthetic pretraining, forward-backward smoothing, or diverse particle trajectories; the author’s row-level regression framing missed that larger conceptual shift. A later reconstruction of the second-place AnchorCNN approach reached 6.667 out-of-fold and 7.148 private on one seed without blending, although it was submitted after the deadline. The author concludes that agents are effective at implementation, debugging, literature search, and exploring variations, but weak at validating experiments, finding fundamentally new framings, and recognizing when proxy metrics are misleading. Human judgment remained responsible for choosing directions, checking evidence, and detecting invalid results.