AI Safety Evaluators Move to the Center of the Oversight Debate

Summary

Independent AI evaluators are moving from a relatively obscure part of the AI industry to a central role in debates over how advanced models should be monitored. Groups including METR, Apollo Research and Transluce assess model capabilities and risks, and investigate cases in which systems behave unexpectedly or breach safeguards. Their importance has grown as the federal government favors voluntary industry oversight rather than a formal regulatory system. Anthropic CEO Dario Amodei has proposed embedding independent evaluators inside the company, and OpenAI CEO Sam Altman, President Donald Trump and major technology companies have supported the idea. The arrangement remains unsettled: there are no common standards for evaluator access, reporting, funding or protection from retaliation. Critics say allowing the companies and infrastructure firms that benefit from AI growth to finance the evaluators could create conflicts of interest. METR said it raised about $71 million in commitments over six months, compared with $13.6 million in total 2024 contributions, but it has fewer than 50 full-time employees. Vals AI, a for-profit evaluator focused on industry-specific benchmarks, grew from eight employees to about 30 and announced a $40 million funding round in August. OpenAI said it is finalizing contracts with external assessors and has worked with groups including METR and Redwood Research; it disputed employees' claims that dismissals were linked to third-party evaluator communications. Anthropic said it will fund embedded work by Accenture and is discussing pilot evaluations with METR and other nonprofits, while acknowledging that longer-term funding may need to come from pooled or government sources. Proposals for government-licensed Independent Verification Organizations are included in the federal FRONTIER Act, while California has enacted an AI-auditor registry and an independent-evaluation framework. Industry proposals call for evaluator access comparable to privileged internal staff, transparency, protection from retaliation and the ability to publish key findings. The evaluator community stresses that embedded reviews can complement, but cannot replace, broader external oversight.