ARC-Bench Finds Frozen JEPA World Models Misrank Actions

Summary

The paper introduces ARC-Bench, a no-leak, fixed-candidate protocol for testing whether frozen JEPA-style latent world models rank actions according to their true cost. These planners select an action by predicting its future embedding and choosing the candidate closest to a goal embedding, which assumes that latent distance preserves the correct ordering of actions. The authors apply the benchmark to officially released JEPA-WM checkpoints in navigation and manipulation-style control. In the manipulation audits, the highest-scored candidate is almost always suboptimal, and the same ranking inversion appears in maze environments. The defect remains when the visual encoder is changed from DINOv2 to video-pretrained V-JEPA 1 or V-JEPA 2 at ViT-L and ViT-G scale. Controls addressing provenance, undertraining, matched backbone budgets, and metric circularity rule out the paper's proposed trivial explanations. The study then identifies why standard closed-loop results can miss the problem: frequent replanning allows a controller to recover after a poor first choice. Reducing replanning frequency causes success to collapse in both a navigation and a manipulation domain, while PointMaze diagnostics show that episodes rescued by frequent replanning contain more severe first-plan ranking failures. The authors conclude that closed-loop success rates can overstate the action-rankability of frozen latent representations, and propose ARC-Bench as a direct audit for methods that adapt, amortize, or replan around latent-space planners.