Deep reinforcement learning from human preferences

ABSTRACT

For many complex real-world tasks, defining a mathematical reward function is difficult, leading to misaligned AI behavior when optimized. We explore a method for solving reinforcement learning tasks without a programmatic reward function, relying instead on human feedback. Our algorithm asks humans to compare two short video clips of the agent's behavior and select the preferred one. We train a reward predictor on these preferences and simultaneously train an RL policy to optimize this predicted reward. Our method successfully solves complex tasks in Atari and simulated robotics environments, such as teaching a simulated robot to do a backflip, using only a few thousand bits of human feedback.