论文标题
重要的不仅很重要：小语言模型也很少。
It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners
论文作者
论文摘要
当缩放到数百十亿个参数时，诸如GPT-3（Brown等，2020）之类的验证语言模型实现了显着的很少的表现。但是，培训和应用如此大型模型需要大量的计算，从而产生了较大的碳足迹，并使研究人员和从业人员很难使用它们。我们表明，与GPT-3相似的性能可以通过“更绿”的语言模型获得，因为它们的参数计数较小几个数量级。这是通过将文本输入转换为包含任务描述的披肩问题，并结合基于梯度的优化来实现的；利用未标记的数据可以进一步改进。我们通过小语言模型确定成功自然语言理解所需的关键因素。
When scaled to hundreds of billions of parameters, pretrained language models such as GPT-3 (Brown et al., 2020) achieve remarkable few-shot performance. However, enormous amounts of compute are required for training and applying such big models, resulting in a large carbon footprint and making it difficult for researchers and practitioners to use them. We show that performance similar to GPT-3 can be obtained with language models that are much "greener" in that their parameter count is several orders of magnitude smaller. This is achieved by converting textual inputs into cloze questions that contain a task description, combined with gradient-based optimization; exploiting unlabeled data gives further improvements. We identify key factors required for successful natural language understanding with small language models.
