Hangar Harness Tests Compare AI Models and Coding Harnesses

Summary

Hangar Harness evaluates different model and harness combinations with the same /goal-mode task: build a self-contained HTML Three.js sci-fi hangar featuring hovering drones, warning lights, emissive runway strips, fog planes, a formation toggle, and a cinematic camera path. The table records duration, time to first token, input, output, reasoning and total tokens, cached-input percentage, tool calls, tool errors, and whether the result opened in a browser or passed screenshot checks. The GLM 5.3 Flash Max run with Codex Open was blocked despite 14 tool calls, while Luna and SOL with MaxCodexOpen completed in 9 and 10 minutes but did not both pass browser validation: Luna opened and passed screenshots, whereas SOL did neither. Astra 6.0 with MaxCodexOpen took 37 minutes, opened in a browser, but did not pass screenshot checks. The OMP runs for GLM and Qwen completed browser and screenshot checks, with GLM taking about 30 minutes and Qwen 41 minutes. OpenCode completed its GLM and Qwen runs in 20 and 8 minutes, and both passed the two checks. Qwen with DSH/PTC also passed both checks in 24 minutes, while DSH alone finished in 18 minutes but failed both. The author notes that cached input is included in input-token totals, and that DSH duration sums active time across two turns rather than the pause between them. Reasoning counts are unavailable for some harnesses, so the figures are not fully comparable.