论文标题
gendice：固定值的广义离线估计
GenDICE: Generalized Offline Estimation of Stationary Values
论文作者
论文摘要
强化学习和蒙特卡洛方法中出现的一个重要问题是估计由马尔可夫链的固定分布定义的数量。在许多实际应用程序中，对基础过渡操作员的访问仅限于已经收集的固定数据集，而无需与可用环境的其他互动。我们表明，在这种挑战性的情况下，一致的估计仍然是可能的，并且在重要应用中仍然可以实现有效的估计。我们的方法是基于估算一个比率，该比率纠正了固定分布和经验分布之间的差异，该比率源自固定分布的基本属性，并基于基于变异差异最小化的约束重新构造。所得算法Gendice是直接有效的。我们证明了它在一般条件下的一致性，提供了错误分析，并在基准问题（包括离线Pagerank和Off-Policy政策评估）上表现出强烈的经验绩效。
An important problem that arises in reinforcement learning and Monte Carlo methods is estimating quantities defined by the stationary distribution of a Markov chain. In many real-world applications, access to the underlying transition operator is limited to a fixed set of data that has already been collected, without additional interaction with the environment being available. We show that consistent estimation remains possible in this challenging scenario, and that effective estimation can still be achieved in important applications. Our approach is based on estimating a ratio that corrects for the discrepancy between the stationary and empirical distributions, derived from fundamental properties of the stationary distribution, and exploiting constraint reformulations based on variational divergence minimization. The resulting algorithm, GenDICE, is straightforward and effective. We prove its consistency under general conditions, provide an error analysis, and demonstrate strong empirical performance on benchmark problems, including off-line PageRank and off-policy policy evaluation.
