论文标题
使用Stella框架在线信息检索评估
Online Information Retrieval Evaluation using the STELLA Framework
论文作者
论文摘要
让用户参与软件开发的早期阶段已成为一种普遍的策略，因为它使开发人员从一开始就可以考虑用户需求。一旦系统正在生产，随着更多信息可用，可以观察，评估和学习用户的新机会。从用户收集信息以不断评估其行为是商业软件的常见实践，而Cranfield范式仍然是学术界信息检索（IR）和推荐系统的首选选择。在这里，我们介绍了生活实验室Stella项目的基础架构，该项目旨在创建一个评估基础架构，允许实验系统与真实用户一起运行基于生产的基于Web的学术搜索系统。 Stella结合了用户互动和日志文件分析，以实现大规模的A/B实验进行学术搜索。
Involving users in early phases of software development has become a common strategy as it enables developers to consider user needs from the beginning. Once a system is in production, new opportunities to observe, evaluate and learn from users emerge as more information becomes available. Gathering information from users to continuously evaluate their behavior is a common practice for commercial software, while the Cranfield paradigm remains the preferred option for Information Retrieval (IR) and recommendation systems in the academic world. Here we introduce the Infrastructures for Living Labs STELLA project which aims to create an evaluation infrastructure allowing experimental systems to run along production web-based academic search systems with real users. STELLA combines user interactions and log files analyses to enable large-scale A/B experiments for academic search.
