论文标题
通过对样品排列的优化进行全局对比度批处理采样
Global Contrastive Batch Sampling via Optimization on Sample Permutations
论文作者
论文摘要
对比度学习最近在各种任务中取得了最新的表现。许多对比学习方法都使用挖掘的硬否负面因素使批次在训练过程中更具信息性，但是这些方法的效率低下，因为它们的时期长度与开采的负面因素成正比，并且需要频繁更新最近的邻居指数或最近批次的采矿。在这项工作中，我们为硬采矿，全球对比批量抽样（GCB）提供了替代方案，这是对批处理分配问题的有效近似，该问题使全球和培训损失之间的差距上限，$ \ MATHCAL {l}^{l}^{global}^{global}^{global}^{global} - \ nathcal {l}^l}^{l}^{l}^{train} $，在对比中。通过实验，我们发现GCB可以改善句子嵌入和代码搜索任务中的最新性能。此外，GCB易于实现，因为它仅需要几行代码，不能维护外部数据结构，例如最近的邻居指数，比最小的硬性负面挖掘方法更有效地计算效率，并且对训练的模型没有任何更改。
Contrastive Learning has recently achieved state-of-the-art performance in a wide range of tasks. Many contrastive learning approaches use mined hard negatives to make batches more informative during training but these approaches are inefficient as they increase epoch length proportional to the number of mined negatives and require frequent updates of nearest neighbor indices or mining from recent batches. In this work, we provide an alternative to hard negative mining, Global Contrastive Batch Sampling (GCBS), an efficient approximation to the batch assignment problem that upper bounds the gap between the global and training losses, $\mathcal{L}^{Global} - \mathcal{L}^{Train}$, in contrastive learning settings. Through experimentation we find GCBS improves state-of-the-art performance in sentence embedding and code-search tasks. Additionally, GCBS is easy to implement as it requires only a few additional lines of code, does not maintain external data structures such as nearest neighbor indices, is more computationally efficient than the most minimal hard negative mining approaches, and makes no changes to the model being trained.
