Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision

ABSTRACT

As AI models become increasingly capable, we will eventually face the challenge of superalignment: how can humans supervise AI systems that are much smarter than them? To study this empirically today, we use an analogy where a smaller, weaker model (e.g., GPT-2) supervises a larger, stronger model (e.g., GPT-4). We demonstrate a phenomenon we call weak-to-strong generalization: when we fine-tune a strong model using labels generated by a weak model, the strong model consistently performs significantly better than its weak supervisor. By applying simple methods like an auxiliary confidence loss, we can elicit highly capable behavior from strong models using only heavily flawed, weak supervision.