论文标题
一种不精确的增强拉格朗日算法，用于培训渗漏神经网络与小组稀疏
An Inexact Augmented Lagrangian Algorithm for Training Leaky ReLU Neural Network with Group Sparsity
论文作者
论文摘要
近年来，具有稀疏正规化项的泄漏的Relu网络已被广泛使用。但是，培训这样的网络会产生非平滑的非凸优化问题，并且缺乏确定性计算固定点的方法。在本文中，我们首先通过引入辅助变量和其他约束来解决原始优化问题中的多层复合项。我们显示新模型具有非空的解决方案集，其可行集合满足了Mangasarian-Fromovitz的约束资格。此外，我们显示了新模型与原始问题之间的关系。值得注意的是，我们提出了一种不精确的增强拉格朗日算法，用于解决新模型，并显示该算法与KKT点的收敛性。数值实验表明，与某些众所周知的算法相比，我们的算法在训练稀疏泄漏的恢复神经网络方面更有效。
The leaky ReLU network with a group sparse regularization term has been widely used in the recent years. However, training such a network yields a nonsmooth nonconvex optimization problem and there exists a lack of approaches to compute a stationary point deterministically. In this paper, we first resolve the multi-layer composite term in the original optimization problem by introducing auxiliary variables and additional constraints. We show the new model has a nonempty and bounded solution set and its feasible set satisfies the Mangasarian-Fromovitz constraint qualification. Moreover, we show the relationship between the new model and the original problem. Remarkably, we propose an inexact augmented Lagrangian algorithm for solving the new model and show the convergence of the algorithm to a KKT point. Numerical experiments demonstrate that our algorithm is more efficient for training sparse leaky ReLU neural networks than some well-known algorithms.
