论文标题
SALR：清晰度感知的学习率调度程序，以改善概括
SALR: Sharpness-aware Learning Rate Scheduler for Improved Generalization
论文作者
论文摘要
为了改善深度学习的概括并自动化学习率调度的过程，我们提出了SALR：一种敏锐的学习率更新技术，旨在恢复平坦的最小化器。我们的方法根据损耗函数的局部清晰度动态更新基于梯度的优化器的学习率。这样一来，优化者就可以自动提高锋利山谷的学习率，以增加逃脱它们的机会。当通过各种算法在广泛的网络中采用各种算法时，我们证明了SALR的有效性。我们的实验表明，SALR改善了概括，收敛的速度更快，并将解决方案驱动到显着平坦的区域。
In an effort to improve generalization in deep learning and automate the process of learning rate scheduling, we propose SALR: a sharpness-aware learning rate update technique designed to recover flat minimizers. Our method dynamically updates the learning rate of gradient-based optimizers based on the local sharpness of the loss function. This allows optimizers to automatically increase learning rates at sharp valleys to increase the chance of escaping them. We demonstrate the effectiveness of SALR when adopted by various algorithms over a broad range of networks. Our experiments indicate that SALR improves generalization, converges faster, and drives solutions to significantly flatter regions.
