论文标题
代表学习的共同信息梯度估计
Mutual Information Gradient Estimation for Representation Learning
论文作者
论文摘要
相互信息（MI）在表示学习中起着重要作用。但是，不幸的是，MI在连续和高维环境中棘手。最近的进步建立了可易和可扩展的MI估计器，以发现有用的表示。但是，当MI大时，大多数现有方法无法提供低变化的MI的准确估计。我们认为，直接估计MI梯度比估计MI本身更具吸引力。为此，我们根据隐式分布的分数估计，建议使用代表学习的相互信息梯度估计器（MIGE）。 MIGE在高维和大型MI设置中表现出MI的紧密而平滑的梯度估计。我们将MIGE的应用扩展在基于Infomax和信息瓶颈方法的无监督学习中。实验结果表明，在学习有用表示方面的绩效显着提高。
Mutual Information (MI) plays an important role in representation learning. However, MI is unfortunately intractable in continuous and high-dimensional settings. Recent advances establish tractable and scalable MI estimators to discover useful representation. However, most of the existing methods are not capable of providing an accurate estimation of MI with low-variance when the MI is large. We argue that directly estimating the gradients of MI is more appealing for representation learning than estimating MI in itself. To this end, we propose the Mutual Information Gradient Estimator (MIGE) for representation learning based on the score estimation of implicit distributions. MIGE exhibits a tight and smooth gradient estimation of MI in the high-dimensional and large-MI settings. We expand the applications of MIGE in both unsupervised learning of deep representations based on InfoMax and the Information Bottleneck method. Experimental results have indicated significant performance improvement in learning useful representation.
