论文标题
关于组成深神经网络的线性函数的数量：朝着神经网络复杂性的精致定义
On the Number of Linear Functions Composing Deep Neural Network: Towards a Refined Definition of Neural Networks Complexity
论文作者
论文摘要
用分段线性激活测量深神经网络表达能力的经典方法是基于计算其最大线性区域数量的。这种复杂性度量与了解神经网络表达的一般特性非常相关，例如深度优于宽度。然而，在比较不同网络体系结构的表现力时，似乎有限。由于线性区域之间的对称冗余，考虑到置换不变网络时，这种缺乏变得特别突出。为了解决这个问题，我们提出了分段线性函数复杂性的精致定义：而不是直接计算线性区域的数量，而是首先在组成分段线性函数的线性函数之间引入等价关系，然后计算这些线性函数与该等价关系。我们的新复杂度度量可以清楚地区分上述两个模型，与经典度量一致，并且与深度成倍增加。
The classical approach to measure the expressive power of deep neural networks with piecewise linear activations is based on counting their maximum number of linear regions. This complexity measure is quite relevant to understand general properties of the expressivity of neural networks such as the benefit of depth over width. Nevertheless, it appears limited when it comes to comparing the expressivity of different network architectures. This lack becomes particularly prominent when considering permutation-invariant networks, due to the symmetrical redundancy among the linear regions. To tackle this, we propose a refined definition of piecewise linear function complexity: instead of counting the number of linear regions directly, we first introduce an equivalence relation among the linear functions composing a piecewise linear function and then count those linear functions relative to that equivalence relation. Our new complexity measure can clearly distinguish between the two aforementioned models, is consistent with the classical measure, and increases exponentially with depth.
