论文标题
通过副本交换随机梯度MCMC学习非凸线学习
Non-convex Learning via Replica Exchange Stochastic Gradient MCMC
论文作者
论文摘要
副本交换蒙特卡洛（REMC），也称为平行回火，是加速传统马尔可夫链蒙特卡洛（MCMC）算法的收敛性的重要技术。但是，这种方法需要基于完整数据集对能量函数进行评估，并且不可扩展到大数据。在迷你批次设置中，REMC的幼稚实现引入了较大的偏见，无法将其直接扩展到随机梯度MCMC（SGMCMC），这是用于模拟深神经网络（DNNS）的标准采样方法。在本文中，我们提出了一种自适应复制品交换SGMCMC（RESGMCMC），以自动纠正偏见并研究相应的特性。该分析意味着在随机环境中马尔可夫跳跃过程的数值离散化中，加速 - 准确性的权衡。从经验上讲，我们通过对各种设置进行广泛的实验来测试算法，并在监督的学习和半监督的学习任务中获得CIFAR10，CIFAR100和SVHN的最新结果。
Replica exchange Monte Carlo (reMC), also known as parallel tempering, is an important technique for accelerating the convergence of the conventional Markov Chain Monte Carlo (MCMC) algorithms. However, such a method requires the evaluation of the energy function based on the full dataset and is not scalable to big data. The naïve implementation of reMC in mini-batch settings introduces large biases, which cannot be directly extended to the stochastic gradient MCMC (SGMCMC), the standard sampling method for simulating from deep neural networks (DNNs). In this paper, we propose an adaptive replica exchange SGMCMC (reSGMCMC) to automatically correct the bias and study the corresponding properties. The analysis implies an acceleration-accuracy trade-off in the numerical discretization of a Markov jump process in a stochastic environment. Empirically, we test the algorithm through extensive experiments on various setups and obtain the state-of-the-art results on CIFAR10, CIFAR100, and SVHN in both supervised learning and semi-supervised learning tasks.
