This document summarizes gradient boosting algorithms XGBoost and LightGBM. It covers decision trees, overfitting, regularization, feature engineering, parameter tuning, evaluation metrics, and comparisons between XGBoost and LightGBM. Key aspects discussed include XGBoost and LightGBM's tolerance of outliers, non-standardized features, collinear features, and NaN values. Parameter tuning, using RandomizedSearchCV and GridSearchCV, and ensembling models to optimize multiple metrics are also covered.

1. Gradient Boosting in Practice:
XGBoost and LightGBM
Gabriel Cypriano

2. source: http://blog.kaggle.com/2017/01/23/a-kaggle-master-explains-gradient-boosting

3. Decision Trees
review
source: https://intelligentjava.wordpress.com/2015/04/28/machine-learning-decision-tree

4. Overfitting review
source: https://en.wikipedia.org/wiki/Overfitting

5. Overfitting with Decision Trees

6. Gradient Boosting
source: https://blog.bigml.com/2017/03/14/introduction-to-boosted-trees

7. Regularization
review
source: https://www.r-bloggers.com/an-attempt-to-understand-boosting-algorithms

8. Regularization review
Ridge (L2) Lasso (L1)

9. XGBoost

10. Feature
Engineering
● OK with outliers
● OK with non-standardized
features
● OK with collinear features
● OK with NaN values
or lack thereof

11. Feature
Engineering
● Got NaN’s?
○ set them to -999
○ set missing = -999
or lack thereof

12. XGBoost
Parameter
Tuning
● n_estimators
● max_depth
● learning_rate
● reg_lambda
● reg_alpha
● subsample
● colsample_bytree
● gamma
yes, it’s combinatorial

13. XGBoost Parameter Tuning
RandomizedSearchCV and GridSearchCV to the rescue.

14. XGBoost Parameter Tuning
How not to do grid search (3 * 2 * 15 * 3 = 270 models):

15. XGBoost
Parameter Tuning
1 * 1 * 3 * 5 * 3 = 45 models

16. Evaluation Metric

17. XGBoost
Ensembles
VotingClassifier with
voting='soft' for
combining multiple
XGBoost models and
optimizing for multiple
metrics.
ensemble xgb_auc xgb_precision xgb_log_loss
AUC 0.84 0.84 0.76 0.84
Precision 0.71 0.44 0.94 0.71
Log Loss 0.42 0.49 0.48 0.38

18. LightGBM

19. LightGBM
and its advantages
● OK with NaN values
● OK with categorical features
● Faster training than XGBoost
● Often better results

20. Trees: Categorical
features vs
One-Hot-Encoded
features
source:
https://medium.com/data-design/visiting-catego
rical-features-and-encoding-in-decision-trees-534
00fa65931

21. Tree Growth strategies
XGBoost:
LightGBM:
source: https://www.analyticsvidhya.com/blog/2017/06/which-algorithm-takes-the-crown-light-gbm-vs-xgboost

22. The good
● Often yields good results
● Reduced need for feature
engineering
● Fast to train a single model
● Good choice if all you have is 1
shot at the problem
● GPU support
● Scikit-learn API
● Great to ensemble and optimize for
multiple metrics
The bad
● Too many parameters
● Slow to tune parameters
● GPU config can be tough (try
Docker)
● No GPU support on
scikit-learn API (XGBoost)

23. Gracias!
gabrielcs.me
vagas.creditas.com.br
somostera.com