Scaling Laws for Reward Model Overoptimization

ABSTRACT

When optimizing a policy against a learned reward model, the policy eventually exploits errors in the reward model, leading to a decline in the true underlying objective. This phenomenon, known as reward hacking or overoptimization (often framed as Goodhart's Law), is a major challenge for Reinforcement Learning from Human Feedback (RLHF). We study this empirically by optimizing a policy against a proxy reward model and measuring the true reward using an objective gold-standard model. We establish empirical scaling laws that predict the onset of overoptimization, demonstrating that the 'alignment tax' scales predictably based on the size of the reward model dataset and the amount of KL-divergence optimized.