论文标题
PAC模仿和基于模型的批量学习马尔可夫决策过程
PAC Bounds for Imitation and Model-based Batch Learning of Contextual Markov Decision Processes
论文作者
论文摘要
我们考虑了与观察到的上下文描述符的批处理多任务增强学习的问题，这是由于其在个性化医疗治疗中的应用。特别是，我们研究了两种一般学习算法：直接政策学习（DPL），这是一种基于模仿学习的方法，它从专家轨迹中学习，基于模型的学习。首先，我们得出了DPL的样本复杂性界限，然后证明即使使用有限的模型类也可以从专家行动中进行基于模型的学习。在放宽了预期通过更大的国家行动空间覆盖基于模型的方法学习的条件之后，我们为基于模型的学习提供了带有有限模型类别的样本复杂性界限，这表明存在具有样本复杂性指数的模型类别的统计复杂性。然后，我们根据数据分布的浓度量度得出基于模型的学习的样本复杂性上限。我们的结果为在这种情况下对基于模型的学习的模仿学习提供了正式的理由。
We consider the problem of batch multi-task reinforcement learning with observed context descriptors, motivated by its application to personalized medical treatment. In particular, we study two general classes of learning algorithms: direct policy learning (DPL), an imitation-learning based approach which learns from expert trajectories, and model-based learning. First, we derive sample complexity bounds for DPL, and then show that model-based learning from expert actions can, even with a finite model class, be impossible. After relaxing the conditions under which the model-based approach is expected to learn by allowing for greater coverage of state-action space, we provide sample complexity bounds for model-based learning with finite model classes, showing that there exist model classes with sample complexity exponential in their statistical complexity. We then derive a sample complexity upper bound for model-based learning based on a measure of concentration of the data distribution. Our results give formal justification for imitation learning over model-based learning in this setting.
