Toolproof Finds 8.8% of 752,499 AI Agent-Skill Listings Do Not Load

Summary

Toolproof is a measurement project from Kynth Studios that brings together nine indexes covering AI agent skills, plugins, marketplaces, coding tools, app builders, registries, agent configuration files, answer-engine citations, and document-extraction APIs. Its headline result comes from SkillWorks: 66,294 of 752,499 agent-skill listings read directly from publishing repositories did not load, producing an 8.8% failure share. The project says these failures were counted when fetched files could not be parsed into something an agent could load, rather than being removed from the denominator. The same index covered 30,072 repositories, while its figures were recomputed nightly. Other indexes reported 341 tracked agent tools, with 295 maintained, 20 slowing, and 26 dead; 36 AI coding tools watched nightly, covering 433 models, 33 pricing pages, and 1,477 ranking rows; and 38 starter kits graded, including 16 installed and run, 22 open source, and two archived since listing. StoreReady assessed 14 AI app builders from 47 pieces of evidence, classifying four as able to ship and eight as able to ship with caveats. BlockDex indexed 74,657 items from 973 public shadcn registries, including 12,599 removed in 30 days and 23,656 with source available. RuleStack read 7,134 agent configuration files across eight formats and 1,729 repositories, with cursor-rules the most common format. CiteRank recorded 210 answers from five engines across 24 free checks, while doc-extract-bench scored 1,210 documents across three vendors and five pinned datasets, recording 18 extraction failures. Toolproof argues that stars, installs, and search demand show discovery rather than whether a tool still works. It says major catalogs generally crawl listings or refresh popularity data without executing them, and that one catalog of 22,070 servers publishes no methodology while another comparison of eight marketplaces found no claim that listings were verified to work. Toolproof is the shared masthead, not a tenth index: each project retains its own name, site, and engine, while the common method is stated in advance and conflicts are disclosed.