论文标题
对新兴记忆技术的深度神经网络的低排名培训
Low-Rank Training of Deep Neural Networks for Emerging Memory Technology
论文作者
论文摘要
神经网络在解决困难决策任务方面的最新成功激励了将智能决策“优势”纳入智能决策。但是，由于记忆和计算局限性，这项工作传统上一直集中在神经网络推断上，而不是训练，尤其是在新兴的非挥发记忆系统中，在这些内存的内存系统上，写作的成本昂贵并降低了寿命。然而，在边缘训练的能力变得越来越重要，因为它可以实时适应设备漂移和环境变化，用户自定义以及跨设备的联合学习。在这项工作中，我们解决了在具有非易失性内存的边缘设备上训练的两个关键挑战：低写入密度和低辅助内存。我们提出了一个低级培训计划，该计划在保持计算效率的同时解决了这些挑战。然后，我们在几个适应性问题上演示了代表性卷积神经网络上的技术，在精确度和重量数量上，它都超过了标准SGD。
The recent success of neural networks for solving difficult decision tasks has incentivized incorporating smart decision making "at the edge." However, this work has traditionally focused on neural network inference, rather than training, due to memory and compute limitations, especially in emerging non-volatile memory systems, where writes are energetically costly and reduce lifespan. Yet, the ability to train at the edge is becoming increasingly important as it enables real-time adaptability to device drift and environmental variation, user customization, and federated learning across devices. In this work, we address two key challenges for training on edge devices with non-volatile memory: low write density and low auxiliary memory. We present a low-rank training scheme that addresses these challenges while maintaining computational efficiency. We then demonstrate the technique on a representative convolutional neural network across several adaptation problems, where it out-performs standard SGD both in accuracy and in number of weight writes.
