论文标题
变压器和生物医学背景知识的表示
Transformers and the representation of biomedical background knowledge
论文作者
论文摘要
基于专业的变形金刚的模型（例如生物伯特和生物长）适用于基于公开可用的生物医学语料库的生物医学领域。因此，它们有可能编码大规模的生物学知识。我们研究了这些模型中生物学知识的编码和表示，及其支持癌症精度医学推断的潜在实用性 - 即，解释基因组改变的临床意义。我们比较了不同变压器基线的性能；我们使用探测来确定针对不同实体的编码的一致性；我们使用聚类方法来比较和对比基因，变异，药物和疾病的嵌入的内部特性。我们表明，这些模型确实确实编码了生物学知识，尽管其中一些模型在针对特定任务的微调中丢失了。最后，我们分析了模型在数据集中的偏见和失衡方面的行为。
Specialised transformers-based models (such as BioBERT and BioMegatron) are adapted for the biomedical domain based on publicly available biomedical corpora. As such, they have the potential to encode large-scale biological knowledge. We investigate the encoding and representation of biological knowledge in these models, and its potential utility to support inference in cancer precision medicine - namely, the interpretation of the clinical significance of genomic alterations. We compare the performance of different transformer baselines; we use probing to determine the consistency of encodings for distinct entities; and we use clustering methods to compare and contrast the internal properties of the embeddings for genes, variants, drugs and diseases. We show that these models do indeed encode biological knowledge, although some of this is lost in fine-tuning for specific tasks. Finally, we analyse how the models behave with regard to biases and imbalances in the dataset.
