论文标题
关于VAE模型的变异得分匹配的失败
On the failure of variational score matching for VAE models
论文作者
论文摘要
得分匹配（SM）是一种训练灵活概率模型的方便方法，通常比传统的最大样本（ML）方法更受欢迎。但是，这些模型比归一化模型不容易解释。因此，通常难以评估训练鲁棒性。我们对现有的变异SM目标进行了重要研究，显示了各种数据集和网络体系结构上的灾难性故障。当优化变异自动编码器（VAE）模型时，我们对目标的理论见解直接从它们的等效自动编码损失中出现。首先，我们表明，在Fisher自动编码器中，SM产生的模型要比最大样本差得多，而Fisher Divergence的近似推断可能会导致低密度的局部Optima。但是，有了重要的修改，该目标将减少到类似于证据下限（ELBO）的正规自动编码损失。该分析预测，经过修改的SM算法应与高斯VAE上的Elbo非常相似。然后，我们从文献中回顾了另外两个基于FD的目标，并表明它们减少到无法解释的自动编码损失，这可能导致性能差。该实验验证了我们的理论预测，并表明只有Elbo和基线客观可鲁棒地产生预期的结果，而先前提出的SM方法则没有。
Score matching (SM) is a convenient method for training flexible probabilistic models, which is often preferred over the traditional maximum-likelihood (ML) approach. However, these models are less interpretable than normalized models; as such, training robustness is in general difficult to assess. We present a critical study of existing variational SM objectives, showing catastrophic failure on a wide range of datasets and network architectures. Our theoretical insights on the objectives emerge directly from their equivalent autoencoding losses when optimizing variational autoencoder (VAE) models. First, we show that in the Fisher autoencoder, SM produces far worse models than maximum-likelihood, and approximate inference by Fisher divergence can lead to low-density local optima. However, with important modifications, this objective reduces to a regularized autoencoding loss that resembles the evidence lower bound (ELBO). This analysis predicts that the modified SM algorithm should behave very similarly to ELBO on Gaussian VAEs. We then review two other FD-based objectives from the literature and show that they reduce to uninterpretable autoencoding losses, likely leading to poor performance. The experiments verify our theoretical predictions and suggest that only ELBO and the baseline objective robustly produce expected results, while previously proposed SM methods do not.
