论文标题
COGS：基于语义解释的组成概括挑战
COGS: A Compositional Generalization Challenge Based on Semantic Interpretation
论文作者
论文摘要
自然语言的特征是组成：复杂表达的含义是由其组成部分的含义构成的。为了促进评估语言处理体系结构的组成能力，我们介绍了COGS，这是基于英语片段的语义解析数据集。 COG的评估部分包含多个系统差距，只能通过组成概括来解决。这些包括熟悉的句法结构的新组合，或熟悉的单词和熟悉结构的新组合。在使用变压器和LSTMS的实验中，我们发现COGS测试集的分布精度接近（96---99％），但是概括的准确性大大降低（16---35％），并且对随机种子显示高敏感性（$ \ \ \ \ \ \ \ \ pm $ 6---8％）。这些发现表明，当代标准NLP模型的组成概括能力受到限制，将COG定位为衡量进度的好方法。
Natural language is characterized by compositionality: the meaning of a complex expression is constructed from the meanings of its constituent parts. To facilitate the evaluation of the compositional abilities of language processing architectures, we introduce COGS, a semantic parsing dataset based on a fragment of English. The evaluation portion of COGS contains multiple systematic gaps that can only be addressed by compositional generalization; these include new combinations of familiar syntactic structures, or new combinations of familiar words and familiar structures. In experiments with Transformers and LSTMs, we found that in-distribution accuracy on the COGS test set was near-perfect (96--99%), but generalization accuracy was substantially lower (16--35%) and showed high sensitivity to random seed ($\pm$6--8%). These findings indicate that contemporary standard NLP models are limited in their compositional generalization capacity, and position COGS as a good way to measure progress.
