factlabel Audits AI-Written Data Claims Against Source Records

Summary

factlabel is an open-source Python tool that sits between an AI agent and its readers, checking whether data-heavy drafts accurately describe their source records. It targets invented figures, false attribution, trend reversals, unsupported citations, rhetorical spin, and omissions such as declines or concentration in one source. Its first tier uses deterministic code to recompute metrics, break tables into categories, extract and mask figures, match quotations, identify causal language, and profile material facts that an honest draft should disclose. A second tier sends parallel blinded and grounded questions to Jev, TypeSafe AI’s hosted decision model: blinded calls identify which metric, cause, direction, or wording the draft refers to, while grounded calls assess support against verified facts. A supervisory code layer combines those typed answers with computed comparisons and assigns pass, review, or block; confident violations can block publication, while borderline or conflicting results go to a person. The system never lets a model decide whether a number is correct: code performs the comparison, and Jev helps map language to the relevant source metric. It can annotate an existing HTML page with numbered findings and a Trust Facts badge, correct dashboard values, and optionally use Claude to rewrite and re-audit a draft. The project reports that its tuned benchmark blocked 100% of misleading drafts and passed 100% of honest drafts, but warns that those figures measure fit on a tuned suite rather than generalization. A retail holdout was later used to fix attribution and metric-scope errors, so it is no longer fully held out. A full example audit takes under a second, asks about 50 questions in parallel, and costs about $0.0006. factlabel requires Python 3.10 or later, is not yet on PyPI, and depends on the paid, closed TypeSafe API for Jev; Claude is optional. The authors caution that Jev can be wrong, thresholds need tuning, figure extraction is English-only, long irrelevant source documents reduce accuracy, and the questions were tuned on four synthetic domains.