论文标题
一种基于分解的方法，用于评估和分析通知者分歧
A Decomposition-Based Approach for Evaluating and Analyzing Inter-Annotator Disagreement
论文作者
论文摘要
我们提出了一种新颖的方法，可以从概念上将现有注释分解为单独的级别，从而可以分别分析通道间的分析。我们提出了两种不同的策略来实现这种方法：一种理论上驱动的方法，其中研究人员基于对注释任务的先验知识定义了分解，以及一种基于探索的任务，其中许多可能的分解是对研究人员进行归纳计算并呈现给研究人员进行解释和评估的。利用最近构造的数据集作为叙事分析作为用例，我们应用了两种策略中的每一种，以证明我们在测试有关注释分歧的来源的假设中的潜力，并揭示了注释任务中的潜在结构和关系。最后，我们建议如何扩展和推广我们的方法，并将其用于其他目的。
We propose a novel method to conceptually decompose an existing annotation into separate levels, allowing the analysis of inter-annotators disagreement in each level separately. We suggest two distinct strategies in order to actualize this approach: a theoretically-driven one, in which the researcher defines a decomposition based on prior knowledge of the annotation task, and an exploration-based one, in which many possible decompositions are inductively computed and presented to the researcher for interpretation and evaluation. Utilizing a recently constructed dataset for narrative analysis as our use-case, we apply each of the two strategies to demonstrate the potential of our approach in testing hypotheses regarding the sources of annotation disagreements, as well as revealing latent structures and relations within the annotation task. We conclude by suggesting how to extend and generalize our approach, as well as use it for other purposes.
