论文标题
富集的预训练的变压器用于关节插槽填充和意图检测
Enriched Pre-trained Transformers for Joint Slot Filling and Intent Detection
论文作者
论文摘要
检测用户的意图并在话语中找到相应的插槽是自然语言理解中的重要任务。他们的相互联系的性质使他们的联合建模成为培训此类模型的标准部分。此外，数据稀缺性和专业词汇提出了其他挑战。最近，在培训的语言模型中的进步，即Elmo和Bert等上下文化模型，通过利用训练非常大型模型的潜力并在特定于任务的数据集中进行微调的几步来彻底改变了该领域。在这里，我们利用了这样的模型，即伯特和罗伯塔，我们在其中设计了一种新颖的建筑。此外，我们提出了一个意图汇总注意机制，并通过融合意图分布，单词特征和令牌表示来加强插槽填充任务。标准数据集的实验结果表明，我们的模型表现优于当前的非伯特状态以及一些基于BERT的基线。
Detecting the user's intent and finding the corresponding slots among the utterance's words are important tasks in natural language understanding. Their interconnected nature makes their joint modeling a standard part of training such models. Moreover, data scarceness and specialized vocabularies pose additional challenges. Recently, the advances in pre-trained language models, namely contextualized models such as ELMo and BERT have revolutionized the field by tapping the potential of training very large models with just a few steps of fine-tuning on a task-specific dataset. Here, we leverage such models, namely BERT and RoBERTa, and we design a novel architecture on top of them. Moreover, we propose an intent pooling attention mechanism, and we reinforce the slot filling task by fusing intent distributions, word features, and token representations. The experimental results on standard datasets show that our model outperforms both the current non-BERT state of the art as well as some stronger BERT-based baselines.
