XGBoost is a machine learning algorithm based on boosting decision trees. It builds decision trees sequentially to optimize a specified loss function. At each step, it fits a decision tree to the residuals of the previous tree to minimize the loss. It uses regularization to control overfitting by penalizing complex trees with many leaves and high weights. To determine the best split at each node, it calculates the loss reduction from splitting the node versus keeping it whole. The split that maximizes loss reduction is selected.

1. 1

2. XGBoost
XGBoost is the machine learning method based on
“boosting algorithm”
This method uses the many decision tree
We don’t explain “decision tree”, please refer
following URL:
https://en.wikipedia.org/wiki/Decision_tree
2

3. What is ”Boosting”?
3

4. Boosting algorithm(1)
For a given dataset with n examples and m
features, the explanatory variables 𝒙𝑖 are defined :
Define the 𝑖 − 𝑡ℎ objective variable 𝑦𝑖; i = 1,2, ⋯ , 𝑛
4
𝒙𝑖 = 𝑥𝑖1, 𝑥𝑖2, ⋯ , 𝑥𝑖𝑚

5. Boosting algorithm(2)
Define the output of 𝑡 − 𝑡ℎ decision tree: 𝑦𝑖
(𝑡)
The error 𝜖𝑖
(1)
between first decision tree’s output and
objective variable 𝑦𝑖 is following:
5
𝜖𝑖
(1)
= 𝑦𝑖
(1)
− 𝑦𝑖

6. Boosting algorithm(3)
The second decision tree predict 𝜖𝑖
(1)
Define the second decision tree’s output 𝜖𝑖
(2)
The predicted value 𝑦𝑖
(𝟐)
is following:
6
𝑦𝑖
2
= 𝑦𝑖
1
+ 𝜖𝑖
2

7. Boosting algorithm(4)
The error 𝜖𝑖
(2)
between predicted value using two
decision tree and objective value is following:
7
𝜖𝑖
(2)
= 𝑦𝑖 − 𝑦𝑖
2
= 𝑦𝑖 − 𝑦𝑖
1
+ 𝜖𝑖
(2)

8. Boosting algorithm(5)
Define the third decision tree’s predicted value 𝜖𝑖
(2)
The predicted value 𝑦𝑖
(3)
using three decision trees is:
8
𝑦𝑖
3
= 𝑦𝑖
2
+ 𝜖𝑖
3
= 𝑦𝑖
1
+ 𝜖𝑖
2
+ 𝜖𝑖
3

9. Boosting algorithm(6)
Construct a new model using the information of the
model learned so far
“Boosting”
9
* It is not Boosting algorithm to make error as objective variable

10. What is “XGBoost”?
10

11. XGBoost
XGBoost has been shown to give state-of-the-art
results on many standard classification benchmarks
More than half of the methods won by the Kaggle
competition use XGBoost
11

12. XGBoost algorithm(1)
Define the 𝑘 − 𝑡ℎ decision tree: 𝑓𝑘
The predicted value when boosting K times is as
follows:
12
y𝑖 =
𝑘=1
𝐾
𝑓𝑘 𝑥𝑖

13. XGBoost algorithm(2)
Define the loss function:
Our purpose is minimize the following objective
13
𝑙 𝑦𝑖, 𝑦𝑖
ℒ 𝜙 =
𝑖=1
𝐼
𝑙 𝑦𝑖, 𝑦𝑖

14. XGBoost algorithm(3)
Deformation of formula
14
min
𝑓𝑡
ℒ 𝑓𝑡 = min
𝑓𝑡 𝑖=1
𝐼
𝑙 𝑦𝑖, 𝑦𝑖
(𝑡)
= min
𝑓𝑡 𝑖=1
𝐼
𝑙 𝑦𝑖,
𝑘=1
𝑡
𝑓𝑘 𝑥𝑖
= min
𝑓𝑡 𝑖=1
𝐼
𝑙 𝑦𝑖, 𝑦𝑖
(𝑡−1)
+ 𝑓𝑡 𝑥𝑖

15. XGBoost algorithm(4)
Define the “penalizes function”:
𝛾 and 𝜆 is the hyper parameters
𝑇 is number of tree node
𝑤 is the vector of the nodes
15
𝛺 𝑓 = 𝛾𝑇 +
1
2
𝜆 𝒘 2

16. XGBoost algorithm(5)
If we add the “penalizes function” to loss, it helps to
smooth the final learnt weight to avoid over-fitting
So, Our new purpose is minimize the
following objective function ℒ 𝜙 :
16
ℒ 𝜙 =
𝑖=1
𝐼
𝑙 𝑦𝑖, 𝑦𝑖 +
𝑘=1
𝐾
𝛺 𝑓𝑘

17. XGBoost algorithm(6)
Minimizing ℒ 𝜙 is same to minimizing all ℒ(𝑡)
:
17
min
𝑓𝑡
ℒ(𝑡) = min
𝑓𝑡 𝑖=1
𝐼
𝑙 𝑦𝑖, 𝑦𝑖
(𝑡−1)
+ 𝑓𝑡 𝑥𝑖 + Ω 𝑓𝑡

18. XGBoost algorithm(7)
Second-order approximation can be used
to quickly optimize the objective :
18
ℒ(𝑡) =
𝑖=1
𝐼
𝑙 𝑦𝑖, 𝑦𝑖
(𝑡−1)
+ 𝑓𝑡 𝑥𝑖 + Ω 𝑓𝑡
Taylor expansion
ℒ(𝑡) ≅
𝑖=1
𝐼
𝑙 𝑦𝑖, 𝑦𝑖
(𝑡−1)
+ 𝑔𝑖 𝑓𝑡 𝑥𝑖 +
1
2
ℎ𝑖 𝑓𝑡
2
𝑥𝑖 + Ω 𝑓𝑡
𝑔𝑖 = 𝜕 𝑦 𝑡−1 𝑙 𝑦𝑖, 𝑦 𝑡−1 ℎ𝑖 = 𝜕 𝑦 𝑡−1
2
𝑙 𝑦𝑖, 𝑦 𝑡−1

19. XGBoost algorithm(8)
We can remove the constant terms to obtain
the following simplified objective at step 𝑡:
19
ℒ(𝑡)
=
𝑖=1
𝐼
𝑔𝑖 𝑓𝑡 𝑥𝑖 +
1
2
ℎ𝑖 𝑓𝑡
2
𝑥𝑖 + Ω 𝑓𝑡
ℒ(𝑡)
=
𝑖=1
𝐼
𝑙 𝑦𝑖, 𝑦𝑖
(𝑡−1)
+ 𝑔𝑖 𝑓𝑡 𝑥𝑖 +
1
2
ℎ𝑖 𝑓𝑡
2
𝑥𝑖 + Ω 𝑓𝑡

20. XGBoost algorithm(9)
Define 𝐼𝑗 as the instance set of leaf j
20
Leaf 1
Leaf 2
Leaf 3Leaf 4
𝐼4

21. XGBoost algorithm(11)
Deformation of formula
21
min
𝑓𝑡
ℒ(𝑡)
= min
𝑓𝑡
𝑖=1
𝐼
𝑔𝑖 𝑓𝑡 𝑥𝑖 +
1
2
ℎ𝑖 𝑓𝑡
2
𝑥𝑖 + Ω 𝑓𝑡
= min
𝑓𝑡
𝑖=1
𝐼
𝑔𝑖 𝑓𝑡 𝑥𝑖 +
1
2
ℎ𝑖 𝑓𝑡
2
𝑥𝑖 + 𝛾𝑇 +
1
2
𝜆 𝑤 2
= min
𝑓𝑡
𝑗=1
𝑇
𝑖∈𝐼 𝑗
𝑔𝑖 𝑤𝑗 +
1
2
𝑖∈𝐼 𝑗
ℎ𝑖 + 𝜆 𝑤𝑗
2
+ 𝛾𝑇
Quadratic function of w

22. XGBoost algorithm(12)
We can solve the quadratic function ℒ(𝑡)
on 𝑤𝑗
22
𝑤ℎ𝑒𝑟𝑒
𝑑ℒ(𝑡)
𝑑𝑤𝑗
= 0
𝑤𝑗 = −
𝑖∈𝐼 𝑗
𝑔𝑖
𝑖∈𝐼 𝑗
ℎ𝑖 + 𝜆

23. XGBoost algorithm(13)
Remember 𝑔𝑖 and ℎ𝑖 is the inclination of loss function
and they can calculate with the output of (𝑡 − 1)𝑡ℎ
tree and 𝑦𝑖
So, we can calculate 𝑤𝑗 and minimizing ℒ 𝜙
23
𝑔𝑖 = 𝜕 𝑦 𝑡−1 𝑙 𝑦𝑖, 𝑦 𝑡−1
ℎ𝑖 = 𝜕 𝑦 𝑡−1
2
𝑙 𝑦𝑖, 𝑦 𝑡−1

24. How to split the node
24

25. How to split the node
I told you how to minimize the loss.
One of the key problems in tree learning is to find the
best split.
25

26. XGBoost algorithm(14)
Substitute 𝑤𝑗 for ℒ(𝑡)
:
In this equation, if
𝑖∈𝐼 𝑗
𝑔 𝑖
2
𝑖∈𝐼 𝑗
ℎ 𝑖+𝜆
in each node become bigger,
ℒ(𝑡)
become smaller.
26
ℒ(𝑡) = −
1
2
𝑗=1
𝑇
𝑖∈𝐼 𝑗
𝑔𝑖
2
𝑖∈𝐼 𝑗
ℎ𝑖 + 𝜆
+ 𝛾𝑇

27. XGBoost algorithm(15)
Compare the
𝑖∈𝐼 𝑗
𝑔 𝑖
2
𝑖∈𝐼 𝑗
ℎ 𝑖+𝜆
in before split node and after split
node
Define the objective function before
split node : ℒ 𝑏𝑒𝑓𝑜𝑟𝑒
(𝑡)
Define the objective function after
split node : ℒ 𝑎𝑓𝑡𝑒𝑟
(𝑡)
27

28. XGBoost algorithm(16)
ℒ 𝑏𝑒𝑓𝑜𝑟𝑒
(𝑡)
= −
1
2 𝑗≠𝑠
𝑇
𝑖∈𝐼 𝑗
𝑔 𝑖
2
𝑖∈𝐼 𝑗
ℎ 𝑖+𝜆
−
1
2
𝑖∈𝐼 𝑠
𝑔 𝑖
2
𝑖∈𝐼 𝑠
ℎ 𝑖+𝜆
+ 𝛾𝑇
ℒ 𝑎𝑓𝑡𝑒𝑟
(𝑡)
= −
1
2 𝑗≠𝑠
𝑇
𝑖∈𝐼 𝑗
𝑔 𝑖
2
𝑖∈𝐼 𝑗
ℎ 𝑖+𝜆
−
1
2
𝑖∈𝐼 𝐿
𝑔 𝑖
2
𝑖∈𝐼 𝐿
ℎ 𝑖+𝜆
−
1
2
𝑖∈𝐼 𝑅
𝑔 𝑖
2
𝑖∈𝐼 𝑅
ℎ 𝑖+𝜆
+ 𝛾𝑇
28
𝐼𝑠
𝐼𝐿 𝐼 𝑅
before
after

29. XGBoost algorithm(17)
After maximizing ℒ 𝑏𝑒𝑓𝑜𝑟𝑒
(𝑡)
− ℒ 𝑎𝑓𝑡𝑒𝑟
(𝑡)
we can get the
minimizing ℒ(𝑡)
29