论文标题
注入噪声作为深度学习动力的探测
Noise Injection as a Probe of Deep Learning Dynamics
论文作者
论文摘要
我们提出了一种新方法，通过使用噪声注入节点（NINS）扰动系统，探测深神经网络（DNN）的学习机制。这些节点通过不更改优化算法的情况下，通过其他可优化的权重向现有的前馈网络体系结构注入不相关的噪声。我们发现该系统在训练过程中显示出不同的阶段，这是由注射噪声的规模决定的。我们首先得出网络动力学的表达式，并利用简单的线性模型作为测试用例。我们发现，在某些情况下，噪声节点的演变类似于不受干扰的损失的演变，因此表明有可能在将来使用NINS了解有关完整系统的更多信息。
We propose a new method to probe the learning mechanism of Deep Neural Networks (DNN) by perturbing the system using Noise Injection Nodes (NINs). These nodes inject uncorrelated noise via additional optimizable weights to existing feed-forward network architectures, without changing the optimization algorithm. We find that the system displays distinct phases during training, dictated by the scale of injected noise. We first derive expressions for the dynamics of the network and utilize a simple linear model as a test case. We find that in some cases, the evolution of the noise nodes is similar to that of the unperturbed loss, thus indicating the possibility of using NINs to learn more about the full system in the future.
