Rails Presents AI Coding-Agent Benchmarks and Its Convention-Driven Approach

Summary

Ruby on Rails presents its framework as a strong environment for AI coding agents, arguing that convention over configuration gives agents a predictable map of names, folders, commands, and implementation patterns. It also says concise Ruby and Rails code can reduce token use, while decades of public Rails code provide training signals for common application components such as controllers, models, tests, jobs, and migrations. The page positions Rails as a full product framework that can let a solo builder take on broader work with an agent. Its first benchmark uses 20 feature-sized tickets for Fizzy, 37signals’ kanban application, and compares models at provider-default and maximum effort on accuracy, speed, mean tokens, and mean cost. GPT-6 Astramax recorded the highest listed accuracy at 53.3%, while GPT-6 Astramedium was faster at 9 minutes 11 seconds and averaged 58,000 tokens and $2.51 per run; the table also shows substantial trade-offs among accuracy, runtime, token use, and cost. Each model and effort level ran every ticket three times in September 2026, for 60 runs, with limits of 90 minutes, 400 steps, and $60 per run. Accuracy required passing both the application test suite and hidden ticket checks, and Rails cautions that differences of a few points may fall within run-to-run noise. A second evaluation covers 21 small, self-contained Writebook tasks, each activating one Rails API. Claude Opus 5 high and Claude Fable 5.1 high both reached 92.1% accuracy, while Fable had the higher API-recall figure at 41.3%; the results also report speed, tokens, and cost. Those runs used provider-default settings, 63 runs per model, and counted refusals as failures. Rails says the evaluation suite is open source and invites readers to inspect underlying results. Statements from David Heinemeier Hansson and other Rails users support the view that standardized, familiar code helps both models generate changes and people review them, although the page’s benchmark caveat means small score differences should not be treated as definitive rankings.