论文标题
是Akpans Trick还是Treat：在助理系统中揭示有用的偏见
Are Akpans Trick or Treat: Unveiling Helpful Biases in Assistant Systems
论文作者
论文摘要
信息寻求AI助理系统旨在及时回答用户对知识的疑问。但是，人类感知到的信息寻求信息助理系统的帮助及其公平性的含义均未探索。在本文中，我们研究了有用性的计算测量值。我们收集有关对话响应有助于的人类注释，开发自动帮助评估的模型，然后建议将对话系统的有益性水平用于不同的用户查询，以评估对话系统的公平性。在三种信息寻求信息的情况下，使用最先进的对话系统（包括Chatgpt）进行的实验表明，现有系统对来自高度发达国家的概念的问题比较不发达国家更有帮助，从而发现了当前信息寻求助理系统的潜在公平问题。
Information-seeking AI assistant systems aim to answer users' queries about knowledge in a timely manner. However, both the human-perceived helpfulness of information-seeking assistant systems and its fairness implication are under-explored. In this paper, we study computational measurements of helpfulness. We collect human annotations on the helpfulness of dialogue responses, develop models for automatic helpfulness evaluation, and then propose to use the helpfulness level of a dialogue system towards different user queries to gauge the fairness of a dialogue system. Experiments with state-of-the-art dialogue systems, including ChatGPT, under three information-seeking scenarios reveal that existing systems tend to be more helpful for questions regarding concepts from highly-developed countries than less-developed countries, uncovering potential fairness concerns underlying the current information-seeking assistant systems.
