论文标题
可能的故事：在多种可能的情况下评估定位定位推理
Possible Stories: Evaluating Situated Commonsense Reasoning under Multiple Possible Scenarios
论文作者
论文摘要
相同上下文的可能后果可能会因我们所指的情况而异。但是，当前在自然语言处理中的研究并不集中于多种可能情况下的常识性推理。这项研究通过询问与候选人答案相同的结尾的多个问题，鉴于短篇小说文字，这是该任务的结构。我们由此产生的数据集，可能的故事，包括超过1.3k的故事文本的4.5k个问题。我们发现，即使是当前的强大审计语言模型也很难始终如一地回答问题，这强调了无监督环境中最高的准确性（60.2％）远远落后于人类准确性（92.5％）。通过与现有数据集的比较，我们观察到数据集中的问题包含答案选项中的最小注释伪像。此外，我们的数据集还包括需要反事实推理的示例，以及需要读者的反应和虚构信息的示例，这表明我们的数据集可以作为对未来常识性推理的未来研究的挑战性测试。
The possible consequences for the same context may vary depending on the situation we refer to. However, current studies in natural language processing do not focus on situated commonsense reasoning under multiple possible scenarios. This study frames this task by asking multiple questions with the same set of possible endings as candidate answers, given a short story text. Our resulting dataset, Possible Stories, consists of more than 4.5K questions over 1.3K story texts in English. We discover that even current strong pretrained language models struggle to answer the questions consistently, highlighting that the highest accuracy in an unsupervised setting (60.2%) is far behind human accuracy (92.5%). Through a comparison with existing datasets, we observe that the questions in our dataset contain minimal annotation artifacts in the answer options. In addition, our dataset includes examples that require counterfactual reasoning, as well as those requiring readers' reactions and fictional information, suggesting that our dataset can serve as a challenging testbed for future studies on situated commonsense reasoning.
