论文标题
探索众包中微型杀剂间资格测试的有效性
Exploring Effectiveness of Inter-Microtask Qualification Tests in Crowdsourcing
论文作者
论文摘要
众包中的资格测试通常用于通过衡量执行微型释放的能力来预滤波工人。在为每种任务类型的每种任务类型创建资格测试时，被视为一种常见且合理的方式，这项研究调查了其工人过滤绩效，当相同的资格测试在多个任务中使用相同的资格测试。现实世界领域：四个组合性案例，其中资格测试和实际任务相同或彼此不同，以及其他两个案例，其中要求具有Masters资格的工人仅执行实际任务。实验结果证明了以下两个发现：I）i）被分配给了难以确定的较难的批准准确性的工人，没有对实际任务进行艰难的任务。 ii）具有Masters资格的工人在低难题的任务上得分更好，但不如通过高缺陷任务的资格测试的人准确。
Qualification tests in crowdsourcing are often used to pre-filter workers by measuring their ability in executing microtasks.While creating qualification tests for each task type is considered as a common and reasonable way, this study investigates into its worker-filtering performance when the same qualification test is used across multiple types of tasks.On Amazon Mechanical Turk, we tested the annotation accuracy in six different cases where tasks consisted of two different difficulty levels, arising from the identical real-world domain: four combinatory cases in which the qualification test and the actual task were the same or different from each other, as well as two other cases where workers with Masters Qualification were asked to perform the actual task only.The experimental results demonstrated the two following findings: i) Workers that were assigned to a difficult qualification test scored better annotation accuracy regardless of the difficulty of the actual task; ii) Workers with Masters Qualification scored better annotation accuracy on the low-difficulty task, but were not as accurate as those who passed a qualification test on the high-difficulty task.
