Study Finds LLM Agents Overtrust Unreliable Tool Outputs

Summary

Most evaluations of tool-using agents assume that tools provide reliable information, but real-world tool returns can be plausible and wrong. This study evaluates 14 LLMs with web search, LLM sub-agent delegation, and code execution, deliberately corrupting tool returns and measuring whether agents use the corrupted content in their final answers. The mean adoption rate exceeds one third for every tool and reaches 68.0% for web search. Reasoning-trace analysis exposes a particularly serious failure: agents may detect conflicts and recover the correct answer internally, yet present only the corrupted answer without warning the user. The researchers test interventions at three levels: user prompting, metadata supplied by the tool provider, and post-training by the agent builder. Some interventions help specific models or tools, but none consistently reduces overtrust across all three settings. The results identify unreliable-tool overtrust as a persistent agent failure mode and support evaluations that test output validation and transparent communication of unresolved conflicts.