论文标题
计算三个机器学习时代的趋势
Compute Trends Across Three Eras of Machine Learning
论文作者
论文摘要
计算，数据和算法进步是指导现代机器学习进步（ML）的三个基本因素。在本文中，我们研究了最容易量化的因子 - 计算的趋势。我们表明，在2010年培训之前，根据摩尔的法律，培训的计算加剧了，大约每20个月加倍。自2010年代初深度学习的出现以来，训练计算的缩放量已经加速，大约每6个月加倍。 2015年底，随着公司开发了大规模的ML型号，培训计算中的需求更大，新趋势出现了。基于这些观察，我们将ML中计算的历史分为三个时代：深度学习时代，深度学习时代和大规模时代。总体而言，我们的工作突出了培训高级ML系统的快速增长计算要求。
Compute, data, and algorithmic advances are the three fundamental factors that guide the progress of modern Machine Learning (ML). In this paper we study trends in the most readily quantified factor - compute. We show that before 2010 training compute grew in line with Moore's law, doubling roughly every 20 months. Since the advent of Deep Learning in the early 2010s, the scaling of training compute has accelerated, doubling approximately every 6 months. In late 2015, a new trend emerged as firms developed large-scale ML models with 10 to 100-fold larger requirements in training compute. Based on these observations we split the history of compute in ML into three eras: the Pre Deep Learning Era, the Deep Learning Era and the Large-Scale Era. Overall, our work highlights the fast-growing compute requirements for training advanced ML systems.
