Model Marketplace

Epoch evaluated six AI models on 11 tasks drawn from its own work, including graphic design, data insight and explorer creation, AI data-center research, and research design. Models received substantial context and tool access, then completed tasks without further intervention; Epoch employees manually graded the results against task-specific rubrics. Claude Fable 5.1 and GPT-6 Astra led overall, mainly because they were reliable on well-defined coding, computational, and computer-use tasks. However, even these frontier closed-weight models struggled to infer implicit standards from examples, choose broadly useful topics, design experiments that measured their stated questions, and interpret flawed results appropriately. Open-weight models lagged further behind, sometimes making basic factual or data-filtering errors that closed-weight models avoided. Fable 5.1 also tended to produce unnecessarily information-dense writing and graphics, while several models converged on similar ideas despite open-ended prompts. Epoch concludes that current models cannot yet automate its work end to end, although it plans to expand the benchmark as new models are released. The authors caution that the results are noisy because each model ran once, grading was partly subjective, tasks were specific to Epoch, and different agent harnesses may have affected performance.

A Mozilla report says the performance gap between leading US closed frontier models and the best Chinese open-weight models has narrowed to about 4.4 months. Moonshot AI’s Kimi K3 scored only three points below Anthropic’s Fable 5 on the Artificial Analysis Intelligence Index while costing about 30 percent as much. Mozilla CTO Raffi Krikorian argues that the choice between closed and open models should depend on the workload: closed systems still earn their premium for expert professional work, intensive retrieval, and long-context tasks, and they commonly include compliance support, operational assistance, and accountability. Open-weight models can be downloaded and run by others, although their training data, data pipelines, and training code are often not disclosed. The article cites DoorDash as an example of a company using Kimi for routine work while reserving a closed model for harder tasks. METR’s time-horizon framework suggests that the best closed model can reliably handle tasks about 1.7 times as long as the best open model: roughly 12-hour tasks versus seven-hour tasks in the example. The practical difference is concentrated in tasks lasting about eight to 12 hours; shorter work may be handled by either type, while neither is generally reliable beyond 12 hours. The report says this lead is temporary because open models are improving quickly. Benchmarking also depends on the software harness surrounding a model, so Vals AI used a neutral harness in Terminal-Bench 2.1. Under that setup, Z.ai’s GLM 5.2 scored within one point of Anthropic’s Claude Opus 4.7 and 4.8 while costing about five times less per completed task. Open models are gaining usage, with eight of OpenRouter’s ten most-used models by token volume in August 2026 offering open weights, but they still captured only 4 percent of revenue in an older Linux Foundation analysis. Krikorian warns that the current open-model ecosystem is heavily concentrated in China and calls for public compute, neutral foundations, broader openness, evaluation, and audit infrastructure to prevent any single country from setting global defaults.

Moonshot is conducting an internal review after security testing found that its Kimi K2.6 and K3 Swarm models could be persuaded to discuss making biological weapons and carrying out assassinations. Mindgard said the discovery was made in July through a jailbreak, a sequence of complex instructions intended to make an AI system ignore its safety guardrails. Its founder said that, once the jailbreak succeeded, the models could discuss harmful subjects freely and generate further malicious suggestions. Mindgard also said a jailbroken Kimi 2.6 might let hackers run code on the provider's computing resources and connect to the internet, potentially creating a platform for cyber-attacks. However, the firm has not demonstrated that the models' answers about harmful topics would work in practice. Mindgard notified Moonshot on 27 July and later published a blog post without disclosing the key exploitation details. Moonshot told the BBC that it welcomed third-party input, was discussing the findings with Mindgard, and had seen a high refusal rate for similar requests in its internal evaluations. The report has renewed debate over the safety implications of open-weight models, which can in theory be run on users' own infrastructure, and over how quickly regulation can respond to AI misuse.