论文标题
迈向专家混合物的通用门控网络
Towards a Universal Gating Network for Mixtures of Experts
论文作者
论文摘要
从多个神经网络中的知识的结合和聚集可以以专家的混合形式观察到。但是，这种组合通常是使用在相同任务上训练的网络完成的，几乎没有提及异质预训练的网络的组合，尤其是在无数据的制度中。本文提出了多种无数据的方法，用于异质神经网络的组合，从简单输出logit统计数据到训练专业的控球网络等。门控网络根据生成的专家激活的性质决定特定的输入是否属于特定网络。该实验表明，包括通用门控方法在内的门控网络构成了最准确的方法，因此代表了在无数据方面使用专家的异构混合物应用的实用步骤。该项目的代码托管在https://github.com/cwkang1998/network-merging上。
The combination and aggregation of knowledge from multiple neural networks can be commonly seen in the form of mixtures of experts. However, such combinations are usually done using networks trained on the same tasks, with little mention of the combination of heterogeneous pre-trained networks, especially in the data-free regime. This paper proposes multiple data-free methods for the combination of heterogeneous neural networks, ranging from the utilization of simple output logit statistics, to training specialized gating networks. The gating networks decide whether specific inputs belong to specific networks based on the nature of the expert activations generated. The experiments revealed that the gating networks, including the universal gating approach, constituted the most accurate approach, and therefore represent a pragmatic step towards applications with heterogeneous mixtures of experts in a data-free regime. The code for this project is hosted on github at https://github.com/cwkang1998/network-merging.
