论文标题
Supernet如何帮助神经体系结构搜索？
How Does Supernet Help in Neural Architecture Search?
论文作者
论文摘要
重量共享，作为加快建筑绩效估算的方法，人们广泛关注。重量共享不是分别训练每个体系结构，而是建立一个将所有架构作为其子模型组装的超网。但是，由于超级核优化和NAS的目标之间的差距，关于NAS过程是否实际上受益于重量分享的争论。为了进一步了解重量共享对NAS的影响，我们对五个搜索空间进行了全面分析，包括NAS-Bench-101，Nas-Bench-201，Darts-Cifar10，Darts-PTB和ProxylessNA。我们发现，重量共享在某些搜索空间上效果很好，但在其他搜索空间上失败了。向前迈出一步，我们进一步确定了这种现象和体重分享能力的偏见。预计我们的工作将激发未来的NAS研究人员更好地利用体重共享的力量。
Weight sharing, as an approach to speed up architecture performance estimation has received wide attention. Instead of training each architecture separately, weight sharing builds a supernet that assembles all the architectures as its submodels. However, there has been debate over whether the NAS process actually benefits from weight sharing, due to the gap between supernet optimization and the objective of NAS. To further understand the effect of weight sharing on NAS, we conduct a comprehensive analysis on five search spaces, including NAS-Bench-101, NAS-Bench-201, DARTS-CIFAR10, DARTS-PTB, and ProxylessNAS. We find that weight sharing works well on some search spaces but fails on others. Taking a step forward, we further identified biases accounting for such phenomenon and the capacity of weight sharing. Our work is expected to inspire future NAS researchers to better leverage the power of weight sharing.
