MASPO: Robust and Sample-Efficient LLM Reasoning via Unified Policy Optimization

ABSTRACT

Policy optimization for Large Language Models often suffers from gradient instability and reward signal unreliability, particularly in mathematical and verifiable reasoning tasks. We introduce MASPO, a framework that unifies gradient utilization, probability mass, and signal reliability into a single differentiable objective. MASPO consistently outperforms existing methods like GRPO across various model scales, leading to more robust and sample-efficient fine-tuning for autonomous agents. Our experiments demonstrate that MASPO-trained models exhibit a significant reduction in 'alignment drift' during multi-turn interactions.