jevcrawl Uses a Decision Model to Find Answers for AI Agents

Summary

jevcrawl is an open-source MCP server and CLI for AI agents that crawls toward relevant answers instead of returning whole pages or asking an LLM to choose every link. A small decision model scores links and passages, while the returned text is copied verbatim from source pages with URLs, heading paths, and character offsets. The project reports finding the answer page in 54% of 50 WebWalkerQA tasks, compared with 33% when Claude selected links; its README also describes the result as roughly twice Crawl4AI’s rate, although the Crawl4AI comparison used 20 different tasks. In the same evaluation, jevcrawl answered 40% of tasks, versus 30% for Claude link selection and 8% for breadth-first crawling. The crawl runs up to four pages at a time and is described as about twice as fast as an LLM-driven crawl, while decision costs are about a cent per crawl and roughly eight times lower per task and ten times lower per correct answer than the stated alternatives. A documented example fetched six Python documentation pages, reduced 80,034 raw tokens to 1,873 returned tokens, and cost $0.0069 in 8.3 seconds. The tool exposes crawl, scrape, map, and doctor commands and works with Claude Code, Claude Desktop, Cursor, and other MCP clients through a TypeSafe API key. Its design treats page text as data rather than instructions: the decision model scores content but does not write it, and fixed question strings limit prompt-injection effects. Failed decision requests degrade to breadth-first behavior, with uncertain blocks returned and a degraded status reported. The project does not run JavaScript, read robots.txt, or rate-limit itself beyond four concurrent fetches, leaving crawling compliance to the user. The benchmark caveats include small samples and the fact that Crawl4AI uses a real browser while jevcrawl does not.