OGPO Trains Vision-Language-Action Models to Recover from Deviations

Summary

Vision-language-action (VLA) models for robotic manipulation are commonly post-trained with trajectory-level supervised imitation, teaching them to reproduce expert actions step by step. The authors argue that this can bias policies toward demonstrated action paths and provide limited guidance when execution leaves those paths. They introduce OGPO, an Offline Goal-conditioned Policy Optimization framework that adaptively relabels demonstrations with semantic key states and builds goal-conditioned process rewards around reaching those states. This changes the post-training objective from copying an entire trajectory to learning whether a policy can reach important task-progress states. The framework is designed to improve completion from feasible states that were not present in the demonstrations, without additional online interaction. Experiments on LIBERO and MetaWorld show consistent gains over standard supervised fine-tuning. Tests on LIBERO-Plus and the authors’ LIBERO-ReAct benchmark, which evaluates mid-execution perturbations, further report stronger robustness under distribution shifts and better recovery from off-demonstration states.