论文标题
神经切线内核超出无限宽度极限：深度和初始化的影响
Neural Tangent Kernel Beyond the Infinite-Width Limit: Effects of Depth and Initialization
论文作者
论文摘要
由于Jacot等人的著名结果，神经切线内核（NTK）被广泛用于分析过多兼容的神经网络。 （2018）：在无限宽度限制中，NTK在训练过程中是确定性和恒定的。但是，该结果无法解释深网的行为，因为如果深度和宽度同时无穷大，它通常不会保持。在本文中，我们研究了与宽度相当的深度连接的Relu网络的NTK。我们证明NTK性质显着取决于初始化时的深度与宽度比和参数的分布。实际上，我们的结果表明，在Poole等人中确定的超参数空间中这三个阶段的重要性。 （2016年）：订购，混乱和混乱的边缘（EOC）。我们在所有三个阶段中都在无限深度和宽度极限中得出NTK分散剂的精确表达式，并得出结论，NTK的可变性在EOC和混乱阶段呈指数增长，但在有序阶段中并非如此。我们还表明，深网的NTK只能在有序阶段训练期间保持稳定，并讨论NTK矩阵的结构在训练过程中如何变化。
Neural Tangent Kernel (NTK) is widely used to analyze overparametrized neural networks due to the famous result by Jacot et al. (2018): in the infinite-width limit, the NTK is deterministic and constant during training. However, this result cannot explain the behavior of deep networks, since it generally does not hold if depth and width tend to infinity simultaneously. In this paper, we study the NTK of fully-connected ReLU networks with depth comparable to width. We prove that the NTK properties depend significantly on the depth-to-width ratio and the distribution of parameters at initialization. In fact, our results indicate the importance of the three phases in the hyperparameter space identified in Poole et al. (2016): ordered, chaotic and the edge of chaos (EOC). We derive exact expressions for the NTK dispersion in the infinite-depth-and-width limit in all three phases and conclude that the NTK variability grows exponentially with depth at the EOC and in the chaotic phase but not in the ordered phase. We also show that the NTK of deep networks may stay constant during training only in the ordered phase and discuss how the structure of the NTK matrix changes during training.
