Reach or Solve? Attributing Agentic Reinforcement-Learning Gains with Checkpoint Handoffs

Summary

This paper argues that endpoint success can misattribute the gains of reinforcement learning in language-model agents. In a closed-loop environment, an agent’s later observations depend partly on its earlier actions, so SFT and RL checkpoints may be evaluated from different states even on the same task. Endpoint success therefore combines two capabilities: reaching a favorable state and completing the task once there. Comparing only states reached by both policies does not solve the problem because it conditions on an outcome and, in the authors’ data, reverses the estimated effect. The paper introduces checkpoint handoff, which replays a state reached by one released checkpoint and gives the cloned state to another checkpoint without retraining. By crossing a REACHER role with a SOLVER role across SFT and RL checkpoints, the protocol decomposes endpoint gains into REACH, the frequency of arriving at a state a fixed number of actions from success, and SOLVE, the probability of finishing from an identical state. Across two benchmarks and two independently released pipelines, the reacher-by-solver interaction is positive in all five reported conditions: an RL-generated history benefits an RL solver more than the same history benefits an SFT solver. On ALFWorld, RL improves both components, and the SFT solver never succeeds on a state where the RL solver fails. Independent REACH and SOLVE gaps predict the aggregate interaction. The authors conclude that long-horizon evaluations can report arrival and completion alongside endpoint success, provided a checkpoint’s history can be replayed under another checkpoint.