论文标题
改进的信息理论概括范围分布式学习和联合学习
Improved Information Theoretic Generalization Bounds for Distributed and Federated Learning
论文作者
论文摘要
我们将信息理论界限考虑到网络环境中统计学习问题的预期概括错误。在这种情况下，有$ k $节点，每个节点都有自己的独立数据集，每个节点的模型都必须汇总为最终的集中式模型。我们认为模型的简单平均以及更复杂的多轮算法。我们对各种问题的预期概括误差（例如Bregman Divergence或Lipschitz连续损失的问题）给出了上限，这表明$ 1/K $对节点数量的依赖性提高。这些“每个节点”界限是根据训练数据集和每个节点的训练权重之间的共同信息，因此在描述每个节点上具有通信或隐私约束固有的概括属性。
We consider information-theoretic bounds on expected generalization error for statistical learning problems in a networked setting. In this setting, there are $K$ nodes, each with its own independent dataset, and the models from each node have to be aggregated into a final centralized model. We consider both simple averaging of the models as well as more complicated multi-round algorithms. We give upper bounds on the expected generalization error for a variety of problems, such as those with Bregman divergence or Lipschitz continuous losses, that demonstrate an improved dependence of $1/K$ on the number of nodes. These "per node" bounds are in terms of the mutual information between the training dataset and the trained weights at each node, and are therefore useful in describing the generalization properties inherent to having communication or privacy constraints at each node.
