论文标题
探测自回归语言模型中的增量解析状态
Probing for Incremental Parse States in Autoregressive Language Models
论文作者
论文摘要
自回旋神经语言模型的下一个字预测对语法表现出显着的敏感性。这项工作评估了由于学习的能力维持增量句法结构的隐式表示，因此该行为在多大程度上产生的程度。我们将句法探测的工作扩展到增量设置，并提出了从自回归语言模型中提取不完整的句法结构（通过基于堆栈的解析器进行操作）的几种探针。我们发现，我们的探针可用于预测模型前缀上的模型偏好，并在因果关系上进行模型表示和引导模型行为。这表明隐式增量句法推论是自回归神经语言模型中的下一词预测的基础。
Next-word predictions from autoregressive neural language models show remarkable sensitivity to syntax. This work evaluates the extent to which this behavior arises as a result of a learned ability to maintain implicit representations of incremental syntactic structures. We extend work in syntactic probing to the incremental setting and present several probes for extracting incomplete syntactic structure (operationalized through parse states from a stack-based parser) from autoregressive language models. We find that our probes can be used to predict model preferences on ambiguous sentence prefixes and causally intervene on model representations and steer model behavior. This suggests implicit incremental syntactic inferences underlie next-word predictions in autoregressive neural language models.
