论文标题
当因果关系不确定时，无模型和基于模型的政策评估
Model-Free and Model-Based Policy Evaluation when Causality is Uncertain
论文作者
论文摘要
When decision-makers can directly intervene, policy evaluation algorithms give valid causal estimates. In off-policy evaluation (OPE), there may exist unobserved variables that both impact the dynamics and are used by the unknown behavior policy. These "confounders" will introduce spurious correlations and naive estimates for a new policy will be biased.当每个时期都会绘制混杂因素时，我们会开发出最坏的情况，以评估对这些未观察到的混杂因素的敏感性。 We demonstrate that a model-based approach with robust MDPs gives sharper lower bounds by exploiting domain knowledge about the dynamics.最后，我们表明，当未观察到的混杂因素随着时间的流逝而持久，OPE变得更加困难，现有技术产生了极为保守的界限。
When decision-makers can directly intervene, policy evaluation algorithms give valid causal estimates. In off-policy evaluation (OPE), there may exist unobserved variables that both impact the dynamics and are used by the unknown behavior policy. These "confounders" will introduce spurious correlations and naive estimates for a new policy will be biased. We develop worst-case bounds to assess sensitivity to these unobserved confounders in finite horizons when confounders are drawn iid each period. We demonstrate that a model-based approach with robust MDPs gives sharper lower bounds by exploiting domain knowledge about the dynamics. Finally, we show that when unobserved confounders are persistent over time, OPE is far more difficult and existing techniques produce extremely conservative bounds.
