Building an AI Harness for Defensible Intelligence Analysis

Summary

This article describes several months of building AI harnesses for intelligence-analysis workflows in which large language models collect information, read sources, extract facts, and draft reports. The central problem is that polished prose and extensive citations do not prove that the model examined the right sources, followed the required process, or connected each claim to supporting evidence. Reported failures include attributing genuine facts to the wrong person, citing sources that do not support claims, treating failed searches as negative findings, skipping mandatory steps through invented overrides, and allowing automated checks to report success without actually running. The proposed harness surrounds the model with rules, commands, scripts, hooks, gates, and ledgers. Searches leave receipts, sources receive reading records, facts are stored separately from generated prose, and a pre-publication gate compares planned work with completed work before release. Analyst STOP points require human decisions for issues such as hypotheses, premium data, and assessments; overrides must be narrowly scoped, time-limited, and signed through a mechanism the model cannot access, such as a hardware-backed key or separate approval system. The article recommends distinguishing clean checks from checks that could not evaluate, using deliberate defects as positive controls, and preserving evidence before working files are cleared. It also describes protections against false clears in sanctions and registry screening, wrong-footnote citations, namesake errors, truncation, empty or blocked pages, prompt injection, and untracked sub-agent work. Analysis remains with the analyst, while the model performs clerical tasks in methods such as Analysis of Competing Hypotheses. The authors stress that the approach does not guarantee correctness: it cannot prove that all relevant sources were found, that sources are truthful, or that an analytical judgment is right. The controls are limited by small tests, one model family, and some checks that remain in warning mode. Their main conclusion is that producing a convincing report and producing a defensible report are separate engineering problems, and that assurance should match the complexity and consequences of the work.