论文标题
梯度增强决策树的高阶优化
High-Order Optimization of Gradient Boosted Decision Trees
论文作者
论文摘要
梯度增强决策树（GBDTS）是用于建模离散或表格数据的主要机器学习算法。与具有数百万个可训练参数的神经网络不同，GBDTS以加性方式优化损失功能，并具有每个叶子的单个可训练参数，这使得易于应用损失函数的高阶优化。在本文中，我们基于数值优化理论介绍了GBDTS的高阶优化，该理论使我们能够基于给定损耗函数的高阶导数构造树。在实验中，我们表明高阶优化具有更快的均值收敛速度，从而导致运行时间减少。我们的解决方案很容易并行化，并在GPU上运行，而代码上的开销很小。最后，我们讨论了未来的潜在改进，例如自动分化任意损失函数以及GBDT与神经网络的组合。
Gradient Boosted Decision Trees (GBDTs) are dominant machine learning algorithms for modeling discrete or tabular data. Unlike neural networks with millions of trainable parameters, GBDTs optimize loss function in an additive manner and have a single trainable parameter per leaf, which makes it easy to apply high-order optimization of the loss function. In this paper, we introduce high-order optimization for GBDTs based on numerical optimization theory which allows us to construct trees based on high-order derivatives of a given loss function. In the experiments, we show that high-order optimization has faster per-iteration convergence that leads to reduced running time. Our solution can be easily parallelized and run on GPUs with little overhead on the code. Finally, we discuss future potential improvements such as automatic differentiation of arbitrary loss function and combination of GBDTs with neural networks.
