论文标题
神经启发的深神经网络具有稀疏，强烈激活
Neuro-Inspired Deep Neural Networks with Sparse, Strong Activations
论文作者
论文摘要
深度神经网络（DNNS）的端到端培训在越来越多的应用程序中产生了最先进的性能，但它并不能洞悉提取的功能或控制所提取的功能。我们在这里报告了一种有希望的神经启发的方法，可以用更稀疏，更强的激活进行DNN。我们使用标准的随机梯度训练，以促进HEBBIAN（“ Fire在一起”，“电线”）更新，以补充端到端的判别成本功能，以及剩余神经元的反Hebbian更新。我们使用激活的分裂归一化（使用强产量抑制弱输出），以及隐式$ \ ell_2 $归一化神经元权重的标准化。 Experiments with standard image classification tasks on CIFAR-10 demonstrate that, relative to baseline end-to-end trained architectures, our proposed architecture (a) leads to sparser activations (with only a slight compromise on accuracy), (b) exhibits more robustness to noise (without being trained on noisy data), (c) exhibits more robustness to adversarial perturbations (without adversarial training).
While end-to-end training of Deep Neural Networks (DNNs) yields state of the art performance in an increasing array of applications, it does not provide insight into, or control over, the features being extracted. We report here on a promising neuro-inspired approach to DNNs with sparser and stronger activations. We use standard stochastic gradient training, supplementing the end-to-end discriminative cost function with layer-wise costs promoting Hebbian ("fire together," "wire together") updates for highly active neurons, and anti-Hebbian updates for the remaining neurons. Instead of batch norm, we use divisive normalization of activations (suppressing weak outputs using strong outputs), along with implicit $\ell_2$ normalization of neuronal weights. Experiments with standard image classification tasks on CIFAR-10 demonstrate that, relative to baseline end-to-end trained architectures, our proposed architecture (a) leads to sparser activations (with only a slight compromise on accuracy), (b) exhibits more robustness to noise (without being trained on noisy data), (c) exhibits more robustness to adversarial perturbations (without adversarial training).
