论文标题
神经语言模型是否显示出对句法形式主义的偏好？
Do Neural Language Models Show Preferences for Syntactic Formalisms?
论文作者
论文摘要
关于深神经语言模型的解释性的最新工作得出结论，自然语法的许多特性在其代表空间中编码。但是，这样的研究通常通过关注一种语言和单一语言形式主义而遭受有限的范围。在这项研究中，我们旨在调查语言模型捕获的句法结构的外表遵守表面句法或深度句法分析样式，以及这些模式是否在不同语言之间保持一致。我们将探针应用于在13种不同语言上训练的BERT和ELMO模型，以探测两种不同的句法注释样式：通用依赖关系（UD），优先考虑深层句法关系和表面句法的普遍依赖（SUD），专注于表面结构。我们发现，这两种模型都偏爱UD而不是SUD - 在语言和层次之间具有有趣的变化 - 并且这种偏好的强度与树状的差异相关。
Recent work on the interpretability of deep neural language models has concluded that many properties of natural language syntax are encoded in their representational spaces. However, such studies often suffer from limited scope by focusing on a single language and a single linguistic formalism. In this study, we aim to investigate the extent to which the semblance of syntactic structure captured by language models adheres to a surface-syntactic or deep syntactic style of analysis, and whether the patterns are consistent across different languages. We apply a probe for extracting directed dependency trees to BERT and ELMo models trained on 13 different languages, probing for two different syntactic annotation styles: Universal Dependencies (UD), prioritizing deep syntactic relations, and Surface-Syntactic Universal Dependencies (SUD), focusing on surface structure. We find that both models exhibit a preference for UD over SUD - with interesting variations across languages and layers - and that the strength of this preference is correlated with differences in tree shape.
