Scaling Laws for Neural Language Models

ABSTRACT

We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude. Other architectural details such as network width or depth have minimal effects within a wide range. These relationships allow us to determine the optimal allocation of a fixed compute budget, demonstrating that larger models are significantly more sample-efficient and should be trained on a relatively modest amount of data and stopped significantly before convergence.