论文标题
通过回顾学习
Learning with Retrospection
论文作者
论文摘要
深层神经网络已成功部署在人工智能的各个领域，包括计算机视觉和自然语言处理。我们观察到，培训DNN的当前标准程序在过去的时期中丢弃了所有学习的信息，除了当前学习的权重。一个有趣的问题是：这个废弃的信息确实没有用吗？我们认为，废弃的信息可以使后续培训受益。在本文中，我们建议对回顾（LWR）进行学习，该学习利用过去时期的学习信息来指导随后的培训。 LWR是一个简单而有效的培训框架，可在不引入任何其他网络参数或推理成本的情况下改善DNN的准确性，校准和鲁棒性，仅在训练开销的情况下只有微不足道。在几个基准数据集上进行的广泛实验证明了LWR对培训DNN的优越性。
Deep neural networks have been successfully deployed in various domains of artificial intelligence, including computer vision and natural language processing. We observe that the current standard procedure for training DNNs discards all the learned information in the past epochs except the current learned weights. An interesting question is: is this discarded information indeed useless? We argue that the discarded information can benefit the subsequent training. In this paper, we propose learning with retrospection (LWR) which makes use of the learned information in the past epochs to guide the subsequent training. LWR is a simple yet effective training framework to improve accuracies, calibration, and robustness of DNNs without introducing any additional network parameters or inference cost, only with a negligible training overhead. Extensive experiments on several benchmark datasets demonstrate the superiority of LWR for training DNNs.
