论文标题

关于生物医学NLP任务的对抗示例

On Adversarial Examples for Biomedical NLP Tasks

论文作者

Araujo, Vladimir, Carvallo, Andres, Aspillaga, Carlos, Parra, Denis

论文摘要

预训练的单词嵌入的成功促使其在生物医学领域的任务中使用。 BERT语言模型在诸如命名实体识别(NER)和语义文本相似性(STS)等任务中的标准性能指标(STS)中显示出了显着的结果,这在NLP领域带来了重大进展。但是,目前尚不清楚这些系统在关键领域(例如法律或医疗)中是否可以很好地工作。因此,在这项工作中,我们在两个著名的医疗和STS数据集上提出了一个对抗评估方案。我们提出了两种类型的攻击,灵感来自人类自然拼写错误和错别字。我们还提出了另一种使用医学术语同义词的攻击。在这些对抗性设置下,模型的准确性大大下降,我们量化了这种性能损失的程度。我们还表明,我们可以通过训练对抗性例子来显着提高模型的鲁棒性。我们希望我们的工作能够激发使用对抗性示例来评估和开发具有鲁棒性的医疗任务的模型。

The success of pre-trained word embeddings has motivated its use in tasks in the biomedical domain. The BERT language model has shown remarkable results on standard performance metrics in tasks such as Named Entity Recognition (NER) and Semantic Textual Similarity (STS), which has brought significant progress in the field of NLP. However, it is unclear whether these systems work seemingly well in critical domains, such as legal or medical. For that reason, in this work, we propose an adversarial evaluation scheme on two well-known datasets for medical NER and STS. We propose two types of attacks inspired by natural spelling errors and typos made by humans. We also propose another type of attack that uses synonyms of medical terms. Under these adversarial settings, the accuracy of the models drops significantly, and we quantify the extent of this performance loss. We also show that we can significantly improve the robustness of the models by training them with adversarial examples. We hope our work will motivate the use of adversarial examples to evaluate and develop models with increased robustness for medical tasks.

扫码加入交流群

加入微信交流群

微信交流群二维码

扫码加入学术交流群,获取更多资源