Scaling Language Models: Methods, Analysis & Insights from Training Gopher

ABSTRACT

Language modelling provides a step towards intelligent communication systems by harnessing large datasets and expressive models. We provide an analysis of Transformer-based language model architectures and training datasets, including a 280 billion parameter model called Gopher. Gopher is evaluated on 152 diverse tasks, achieving state-of-the-art performance across the majority of them. More importantly, we provide a holistic analysis of the training process, exploring the limits of scale on reading comprehension, fact-checking, and toxic output mitigation. This extensive 118-page report details our findings on architectural design choices, dataset curation (MassiveText), and the emergent capabilities of models at the 200B+ parameter scale.