论文标题
Erica：通过对比学习改善预训练语言模型的实体和关系理解
ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning
论文作者
论文摘要
预训练的语言模型（PLM）在各种下游自然语言处理（NLP）任务上表现出卓越的性能。但是，传统的训练预训练目标并未在文本中明确对关系事实进行建模，这对于文本理解至关重要。为了解决这个问题，我们提出了一个新颖的对比学习框架埃里卡（Erica），以深入了解实体及其在文本中的关系。具体而言，我们定义了两个新颖的训练前任务，以更好地理解实体和关系：（1）区分可以通过给定的头部实体和关系来推断哪个尾巴实体的实体歧视任务； （2）区分两个关系是否在语义上是紧密的，涉及复杂的关系推理的关系。实验结果表明，埃里卡（Erica）可以在几种语言理解任务上改善典型的PLM（BERT和ROBERTA），包括关系提取，实体键入和问题答案，尤其是在低资源设置下。
Pre-trained Language Models (PLMs) have shown superior performance on various downstream Natural Language Processing (NLP) tasks. However, conventional pre-training objectives do not explicitly model relational facts in text, which are crucial for textual understanding. To address this issue, we propose a novel contrastive learning framework ERICA to obtain a deep understanding of the entities and their relations in text. Specifically, we define two novel pre-training tasks to better understand entities and relations: (1) the entity discrimination task to distinguish which tail entity can be inferred by the given head entity and relation; (2) the relation discrimination task to distinguish whether two relations are close or not semantically, which involves complex relational reasoning. Experimental results demonstrate that ERICA can improve typical PLMs (BERT and RoBERTa) on several language understanding tasks, including relation extraction, entity typing and question answering, especially under low-resource settings.
