Why Coding Agents Often Prefer Grep to LSP

Summary

A small pilot study compares lexical search with grep and LSP-backed semantic navigation across code-finding and editing tasks involving three Claude models, Python and TypeScript repositories, and several task types. When both tools were available, the models selected semantic navigation in only 0% to 6% of simple localization tasks, and forcing a semantic-first path reduced success from 100% to 89%. For reference-completeness tasks, semantic-tool use rose to 45% to 57%; LSP achieved 1.00 precision versus 0.76 for grep, but recall remained near 0.66 for both, suggesting that agent thoroughness remained the limiting factor. Repository noise changed the outcome: on clean TypeScript code, LSP produced no F1 gain and used 16% more tokens, while on a noisy TypeScript repository it improved F1 by 0.246 and used 12% fewer tokens; a Python repository gained 0.072 F1 at a 19% token cost. The study also changed the LSP response from locations only to locations with inline source context. In multi-file rename tasks, pass@1 rose from 0.67 to 0.83, while follow-up reads fell from 15.2 to 3.2 per episode, without changing the semantic backend. The article argues that semantic precision alone does not make a tool model-friendly: output shape, context, task structure, and familiar action patterns all affect behavior. It distinguishes the demonstrated structural advantage of grep for text-wide edits from the unproven hypothesis that training may make familiar tool loops easier for models to use. The author concludes that teams should evaluate models together with their harness, measure adoption at equal accuracy, return enough context for the next decision, preserve lexical fallbacks, and route tools according to the task and repository. The findings are preliminary, based on small task sets, a few repositories, three Claude models, and two to three rollouts per cell; they do not cover rename-capable LSP features, diagnostics, code actions, or standard SWE-bench tasks.