Small Search Agents Benefit from Reward-Designed Reinforcement Learning

Summary

Reinforcement Learning with Verifiable Rewards (RLVR) has shown strong results on tasks with clear rewards, but its suitability for less clearly rewarded open-domain question answering has been less established for small models. This study applies a reason-over-search recipe to Qwen3.5-0.8B, training it with Group Relative Policy Optimization and an interleaved Wikipedia-search tool on the MuSiQue dataset. The researchers vary only the reward shape, testing three alternatives across three random seeds each, and evaluate every checkpoint on a held-out suite of seven question-answering benchmarks. The best run reaches 0.352 average exact match, compared with 0.092 for the untrained baseline, a 3.8-fold improvement. The result is achieved without distillation from a larger teacher in the training loop, addressing a limitation of earlier sub-billion-parameter demonstrations. Reward design materially affects performance: a Search-R1-style exact-match-only reward is the weakest of the three at every seed at the matched training horizon, including on exact match itself. The authors conclude that sparse exact-match rewards, commonly used in mathematics and coding RLVR, are a poor default for models of this size. Small-model RLVR can work in search-grounded QA, but its reward design requires dedicated study rather than simply scaling down recipes developed for larger models.