Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

ABSTRACT

We demonstrate a striking phenomenon in the training dynamics of neural networks on small algorithmic datasets: networks that initially severely overfit the training data can, after continued training for long periods, suddenly undergo a phase transition and perfectly generalize to the validation set. We call this phenomenon 'grokking'. We study this behavior across various binary operations (e.g., modular arithmetic, polynomial evaluation) and analyze how data dataset size and weight decay interact with this phase transition. Our findings challenge standard intuitions about early stopping and overfitting, suggesting that neural networks may discover generalizing representations long after memorization occurs.