论文标题
开放式语言生成的采样算法的系统表征
A Systematic Characterization of Sampling Algorithms for Open-ended Language Generation
论文作者
论文摘要
这项工作研究了自动回归语言模型广泛采用的祖传抽样算法，在文献中并未对此进行广泛研究。我们使用质量多样性（Q-D）折衷来研究三种流行的采样算法（TOP-K，Nucleus和perked采样）。我们专注于开放式语言生成的任务。我们首先表明现有的采样算法具有相似的性能。仔细检查了由不同采样算法定义的转换后，我们确定了其中共享的三个关键特性：熵减少，订单保存和坡度保存。为了验证已确定属性的重要性，我们设计了两组新的采样算法：一个集合，其中每种算法满足所有三个属性，而每个算法违反了至少一个属性。我们将它们的性能与现有的采样算法进行比较，并发现违反所确定的属性可能导致性能急剧下降，如Q-D权衡衡量。另一方面，我们发现满足这些属性的采样算法集与现有采样算法相当。我们的数据和代码可在https://github.com/moinnadeem/characterizing-sampling-algoriths上找到
This work studies the widely adopted ancestral sampling algorithms for auto-regressive language models, which is not widely studied in the literature. We use the quality-diversity (Q-D) trade-off to investigate three popular sampling algorithms (top-k, nucleus and tempered sampling). We focus on the task of open-ended language generation. We first show that the existing sampling algorithms have similar performance. After carefully inspecting the transformations defined by different sampling algorithms, we identify three key properties that are shared among them: entropy reduction, order preservation, and slope preservation. To validate the importance of the identified properties, we design two sets of new sampling algorithms: one set in which each algorithm satisfies all three properties, and one set in which each algorithm violates at least one of the properties. We compare their performance with existing sampling algorithms, and find that violating the identified properties could lead to drastic performance degradation, as measured by the Q-D trade-off. On the other hand, we find that the set of sampling algorithms that satisfies these properties performs on par with the existing sampling algorithms. Our data and code are available at https://github.com/moinnadeem/characterizing-sampling-algorithms
