MagicBeanStalk Targets the Zero-Advantage Trap in RLVR Data

Summary

The paper examines why simply expanding datasets of verifiable problems does not necessarily improve reinforcement learning with verifiable rewards (RLVR). For a fixed target policy and rollout budget, about 88% of problems in a large open-source math corpus were either solved in every rollout or failed in every rollout, providing little useful advantage signal. Filtering can concentrate the remaining problems, but it only discards difficult cases. The authors argue that useful synthetic problems must preserve the reasoning operations and compositions needed for those cases while being calibrated so that some rollouts succeed, some fail, and at least one correct solution fits within the training budget. Directly prompting a teacher to vary hard problems did not meet these requirements. They introduce MagicBeanStalk, which decomposes seed problems top-down and recursively grows families of substantively new problems while preserving their reasoning blueprints and calibrating them to the target policy and budget. Across eight math benchmarks, continual RLVR on MagicBeanStalk data raised gpt-oss-120b from 61.2 to 69.8 and DeepSeek v4Flash from 72.4 to 77.9 under a 16K-token budget. Training on the full 78K-source corpus produced only a 1.1-point gain, while its best filtered real-data subset gained 5.8 points. In finance, the method generated more than 14K verifiable problems from 340 solutions and improved over the base models by 6.9 points on SecQue and 2.4 points on FinanceBench.