Preprint2022
Training Compute-Optimal Large Language Models
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya et al. · arXiv · 2022
Abstract
Retrains over four hundred models across sizes and data budgets to find where a fixed amount of compute is best spent, and concludes that the models of the day were substantially oversized for the data they saw. Parameters and training tokens should scale together, roughly in step. A far smaller model trained on much more data beat its larger predecessors.
Why it matters
The result that redirected the field from 'bigger' to 'better fed', and the reason the models that followed got smaller and cheaper to serve while getting better. If you read one scaling paper, read this one.
scaling lawspretrainingtrainingefficiency
Read the source
https://arxiv.org/abs/2203.15556