论文标题
通过subsodular优化的数据有效的结构化修剪
Data-Efficient Structured Pruning via Submodular Optimization
论文作者
论文摘要
结构化修剪是压缩大型预训练的神经网络而不会显着影响其性能的有效方法。但是，大多数当前的结构化修剪方法没有提供任何性能保证，并且通常需要微调，这使得它们在有限数据制度中不适用。我们提出了一种基于子解体优化的原则性数据有效的结构化修剪方法。特别是，对于给定的层，我们选择神经元/通道以修剪下一个层的修剪和相应的新权重，从而最大程度地减少了修剪引起的下一层输入的变化。我们表明，这个选择问题是一个弱的下二个最大化问题，因此可以使用有效的贪婪算法证明它可以证明它。在合理的假设下，我们的方法可以保证在原始模型和修剪模型输出W.R.T之间具有指数降低的误差。它也是文献中仅使用有限数量的培训数据而没有标签的少数方法之一。我们的实验结果表明，我们的方法在有限数据制度中优于最先进的方法。
Structured pruning is an effective approach for compressing large pre-trained neural networks without significantly affecting their performance. However, most current structured pruning methods do not provide any performance guarantees, and often require fine-tuning, which makes them inapplicable in the limited-data regime. We propose a principled data-efficient structured pruning method based on submodular optimization. In particular, for a given layer, we select neurons/channels to prune and corresponding new weights for the next layer, that minimize the change in the next layer's input induced by pruning. We show that this selection problem is a weakly submodular maximization problem, thus it can be provably approximated using an efficient greedy algorithm. Our method is guaranteed to have an exponentially decreasing error between the original model and the pruned model outputs w.r.t the pruned size, under reasonable assumptions. It is also one of the few methods in the literature that uses only a limited-number of training data and no labels. Our experimental results demonstrate that our method outperforms state-of-the-art methods in the limited-data regime.
