Compute Trends Across Three Eras of Machine Learning

TLDR

Compute, data, and algorithmic advances are the three fundamental factors that drive progress in modern Machine Learning (ML). The study investigates how compute has evolved by compiling a dataset of 123 milestone ML systems and defining three distinct eras of development. The authors curated a 3× larger compute dataset and identified a novel trend around 2015 to delineate the Pre Deep Learning, Deep Learning, and Large‑Scale eras. They report a Deep Learning Era compute doubling time of roughly six months and emphasize the rapidly increasing compute demands for training advanced ML systems.

Abstract

Compute, data, and algorithmic advances are the three fundamental factors that drive progress in modern Machine Learning (ML). In this paper we study trends in the most readily quantified factor - compute. We make three novel contributions: (1) we curate a dataset with the training compute of 123 milestone ML systems, 3× larger than previous such datasets. (2) We frame the trends in compute in in three eras - the Pre Deep Learning Era, the Deep Learning Era, and the Large-Scale Era, based on our identification of a novel trend emerging around 2015. (3) We find a Deep Learning Era compute doubling time of around 6 months, significantly longer than previous findings. Overall, our work highlights the fast-growing compute requirements for training advanced ML systems.