论文标题
桩：多师学标记蒸馏的成对迭代逻辑合奏
PILE: Pairwise Iterative Logits Ensemble for Multi-Teacher Labeled Distillation
论文作者
论文摘要
预训练的语言模型已成为排名系统的关键部分，最近取得了令人印象深刻的效果。为了保持高性能的同时保持有效的计算，知识蒸馏被广泛使用。在本文中，我们重点介绍了排名模型的知识蒸馏中的两个关键问题：1）如何从多教老师那里合并知识； 2）如何在蒸馏过程中利用数据的标签信息。我们提出了一种称为成对迭代逻辑集合（堆）的统一算法，以同时解决这两个问题。 PILE合奏以迭代方式监督标签信息监督的多教老师逻辑，并在离线和在线实验中都取得了竞争性能。所提出的方法已部署在现实世界的商业搜索系统中。
Pre-trained language models have become a crucial part of ranking systems and achieved very impressive effects recently. To maintain high performance while keeping efficient computations, knowledge distillation is widely used. In this paper, we focus on two key questions in knowledge distillation for ranking models: 1) how to ensemble knowledge from multi-teacher; 2) how to utilize the label information of data in the distillation process. We propose a unified algorithm called Pairwise Iterative Logits Ensemble (PILE) to tackle these two questions simultaneously. PILE ensembles multi-teacher logits supervised by label information in an iterative way and achieved competitive performance in both offline and online experiments. The proposed method has been deployed in a real-world commercial search system.
