Mercor Maps Gaps in AI Benchmark Coverage Across the US Economy

Summary

Mercor’s Benchmark Map examines whether current AI evaluations represent the work performed across the US economy. The project maps 29 publicly available or Mercor APEX benchmarks to the 923 occupations and 18,797 tasks in the US Department of Labor’s O*NET taxonomy. An O*NET task is counted as covered when the mapping assigns it at least five benchmark prompts. By that measure, 208 occupations have at least one sufficiently represented task, while six of 22 job families have no such occupations; Mercor estimates that 90% of American working hours involve tasks with little or no coverage in the index. Coverage is concentrated in AI-exposed tasks and occupations with larger wage bills, and exposed tasks are about six times more likely to appear than unexposed tasks. Important tasks within occupations are also represented more often, while smaller, middle-income knowledge-work roles are less visible. The authors use an LLM judge to map prompts to O*NET tasks, validate part of the mapping with human expert tags on APEX, and combine coverage estimates with other LLM-derived estimates of task time shares. A test on investment banking, law, accounting, and software engineering shows that O*NET-weighted APEX scores are generally close to simple prompt averages, although fewer than one-third of relevant tasks have enough coverage to enter the weighted scores. Mercor stresses that the results are provisional because the benchmark list is incomplete, selection involves subjective criteria, and current evaluations have limited statistical power, uneven task fidelity, and weak coverage of physical work and human-AI augmentation.