论文标题
局部随机二元优化，基于动量的方差降低
Local Stochastic Bilevel Optimization with Momentum-Based Variance Reduction
论文作者
论文摘要
Bilevel优化最近通过新的有效算法目睹了显着的进步，并已应用于许多机器学习任务，例如数据清洁，很少的学习学习和神经体系结构搜索。但是，很少有人注意解决分布式设置下的二重性问题。联合学习（FL）是一个新兴的范式，它可以解决机器学习任务，而不是分布式的数据。由于异质性和沟通瓶颈，FL问题要解决问题。但是，目前尚不清楚这些挑战将如何影响双光线优化算法的收敛性。在本文中，我们研究了联合的双层优化问题。具体来说，我们首先提出了基于确定性梯度的算法FedBio，并且我们表明它需要$ O（ε^{ - 2}）$迭代数量才能达到$ε$ - 稳定点。然后，我们建议FEDBIOACC在随机场景下使用基于动量的方差减少技术加速FedBio。我们显示FedBioAcc的复杂性为$ O（ε^{ - 1.5}）$。最后，我们通过重要的公平联合学习任务来验证我们提出的算法。更具体地说，我们定义了一个基于双重的集体集体fl目标。与数值实验中的其他基线相比，我们的算法表现出优越的性能。
Bilevel Optimization has witnessed notable progress recently with new emerging efficient algorithms and has been applied to many machine learning tasks such as data cleaning, few-shot learning, and neural architecture search. However, little attention has been paid to solve the bilevel problems under distributed setting. Federated learning (FL) is an emerging paradigm which solves machine learning tasks over distributed-located data. FL problems are challenging to solve due to the heterogeneity and communication bottleneck. However, it is unclear how these challenges will affect the convergence of Bilevel Optimization algorithms. In this paper, we study Federated Bilevel Optimization problems. Specifically, we first propose the FedBiO, a deterministic gradient-based algorithm and we show it requires $O(ε^{-2})$ number of iterations to reach an $ε$-stationary point. Then we propose FedBiOAcc to accelerate FedBiO with the momentum-based variance-reduction technique under the stochastic scenario. We show FedBiOAcc has complexity of $O(ε^{-1.5})$. Finally, we validate our proposed algorithms via the important Fair Federated Learning task. More specifically, we define a bilevel-based group fair FL objective. Our algorithms show superior performances compared to other baselines in numerical experiments.
