Putting the Value Back in RL: Better Test-Time Scaling by Unifying LLM Reasoners With Verifiers - Sareen, Kusha; Moss, Morgane M; Sordoni, Alessandro; Agarwal, Rishabh; Hosseini, Arian (COLM 2025, 2025)
Prevalent reinforcement learning (RL) methods for fine-tuning LLM reasoners, such as GRPO or Leave-one-out PPO, abandon the learned value function in favor of empirically estimated returns. This hinders test-time compute scaling that relies on using the value-function for verification. In this work, we propose RL that augments any "value-free" RL method by jointly training the LLM as both a reasoner and a generative verifier using RL-generated data, adding verification capabilities without significant overhead. Empirically, RL boosts MATH accuracy by over 20% with parallel sampling and enables efficient test-time compute scaling compared to the base RL method. RL also exhibits strong generalization capabilities for both easy-to-hard and out-of-domain tasks. Furthermore, RL achieves higher performance when jointly scaling parallel and sequential test-time compute with a long reasoning R1 model.