论文标题
现代Hopfield Networks出现的Boltzmann机器家族中的关注
Attention in a family of Boltzmann machines emerging from modern Hopfield networks
论文作者
论文摘要
Hopfield Networks和Boltzmann机器（BMS）是基于能量的基于能量的神经网络模型。关于现代Hopfield网络的最新研究扩大了能量功能的类别，并导致了包括注意模块在内的通用Hopfield网络的统一观点。在这封信中，我们使用相关的能量功能来考虑现代Hopfield网络的BM对应物，并从训练性的角度研究其显着特性。特别是，与注意模块相对应的能量函数自然引入了新型BM，我们称之为注意力BM（ATTNBM）。我们验证ATTNBM具有可拖动的可能性功能和某些特殊情况的梯度，并且易于训练。此外，我们揭示了ATTNBM与某些单层型号之间的隐藏连接，即Gaussian-Bernoulli限制了BM和DeNoising AutoCododer，其SoftMax单元来自Denoising Scories匹配。我们还研究了其他能量函数引入的BMS，并表明密集的关联记忆模型的能量函数使BMS属于指数族家族和谐。
Hopfield networks and Boltzmann machines (BMs) are fundamental energy-based neural network models. Recent studies on modern Hopfield networks have broaden the class of energy functions and led to a unified perspective on general Hopfield networks including an attention module. In this letter, we consider the BM counterparts of modern Hopfield networks using the associated energy functions, and study their salient properties from a trainability perspective. In particular, the energy function corresponding to the attention module naturally introduces a novel BM, which we refer to as the attentional BM (AttnBM). We verify that AttnBM has a tractable likelihood function and gradient for certain special cases and is easy to train. Moreover, we reveal the hidden connections between AttnBM and some single-layer models, namely the Gaussian--Bernoulli restricted BM and the denoising autoencoder with softmax units coming from denoising score matching. We also investigate BMs introduced by other energy functions and show that the energy function of dense associative memory models gives BMs belonging to Exponential Family Harmoniums.
