AgentMeasure Audit Finds 45+ Verified Billing Bugs in AI Usage Tools

Summary

AgentMeasure reports an audit of about 110 repositories across the Claude Code, Codex, gateway, evaluation, and observability ecosystems. The project says it filed more than 65 evidence-backed findings, with over 45 verified billing bugs and 10 findings already merged or accepted by maintainers as of September 10, 2026. Most audited tools passed, but the audit identifies five recurring failure classes. First, some parsers count each JSONL content-block line as a separate assistant message even when the lines share one message ID and usage total, producing measured overcounts of two to five times; one related implementation instead undercounts by taking only the first value. Second, tools can double-count Codex token-count events that re-emit identical cumulative totals, with measured session-file inflation of 0.5% to 15%. Third, cache pricing is mishandled because Anthropic and OpenAI expose cache-related input-token semantics differently; the report cites 1.25x pricing for five-minute writes, 2x for one-hour writes, and 0.1x for reads, while documenting errors such as charging cache twice or understating one-hour writes by 37.5% for one component. Fourth, embedded price tables can diverge from published rates by 1.2x to 5x, including inherited values in vendored snapshots. Fifth, resume and fork handling can lose earlier history or count a parent prefix twice when Codex creates a new rollout file with the original session ID. The audit used code reading, synthetic fixtures, and each tool’s own parser, with pinned commits and runnable or quoted-static evidence. It publishes an open fixture corpus and says all reproductions were synthetic, with no private logs used; audited projects do not endorse AgentMeasure.