论文标题
是什么塑造了自我监督学习的损失格局？
What shapes the loss landscape of self-supervised learning?
论文作者
论文摘要
预防表征的完整和维度崩溃最近已成为自我监视学习（SSL）的设计原则。但是，我们的理论理解仍然存在：这些崩溃何时发生？有哪些机制和原因？我们通过得出和彻底分析SSL损失景观的可分析理论来回答这些问题。在这一理论中，我们确定了维度崩溃的原因，并研究了归一化和偏见的效果。最后，我们利用分析理论提供的可解释性来了解维度崩溃如何有益，以及影响SSL对数据不平衡的鲁棒性的原因。
Prevention of complete and dimensional collapse of representations has recently become a design principle for self-supervised learning (SSL). However, questions remain in our theoretical understanding: When do those collapses occur? What are the mechanisms and causes? We answer these questions by deriving and thoroughly analyzing an analytically tractable theory of SSL loss landscapes. In this theory, we identify the causes of the dimensional collapse and study the effect of normalization and bias. Finally, we leverage the interpretability afforded by the analytical theory to understand how dimensional collapse can be beneficial and what affects the robustness of SSL against data imbalance.
