Study Characterizes How Conversational LLM Agents Search the Web

Summary

This study examines the end-to-end lifecycle of web search in conversational LLM agents across ChatGPT, Claude, Grok, and DeepSeek. It combines real-world user interactions with controlled API experiments using the models associated with each platform. The researchers analyze four stages: whether agents decide to invoke search, how they formulate queries, whether their search results show domain preferences, and how they transform results into grounded responses. Search decisions vary substantially across platforms and models, and invoking search more often does not necessarily improve response quality. The agents also use distinct and complex querying strategies, while platform-specific search engines tend to return results from preferred domains. Responses are generally grounded in retrieved search results, but some claims depend on search results that are not cited. The findings highlight attribution and reliability concerns and have implications for designing conversational retrieval systems and future AI agents.