论文标题
要测试机器理解，请先定义理解
To Test Machine Comprehension, Start by Defining Comprehension
论文作者
论文摘要
许多旨在测量机器阅读理解（MRC）的任务，通常集中于认为困难的问题类型。但是，很少有任务设计师从考虑哪些系统应该理解的是开始。在本文中，我们做出了两个关键贡献。首先，我们认为现有的方法不能充分定义理解。他们对测试的内容太不合时宜。其次，我们为广泛有用的文本（即简短的叙述）提供了理解的详细定义 - “理解模板”。然后，我们进行了一项实验，强烈建议现有系统在定义它时不符合叙事理解的任务。
Many tasks aim to measure machine reading comprehension (MRC), often focusing on question types presumed to be difficult. Rarely, however, do task designers start by considering what systems should in fact comprehend. In this paper we make two key contributions. First, we argue that existing approaches do not adequately define comprehension; they are too unsystematic about what content is tested. Second, we present a detailed definition of comprehension -- a "Template of Understanding" -- for a widely useful class of texts, namely short narratives. We then conduct an experiment that strongly suggests existing systems are not up to the task of narrative understanding as we define it.
