xdcd

1. Ensemble Learning: Bagging,
Boosting & Stacking
A practical, teaching-ready deck (with
code snippets)

2. Why Ensemble Learning?
• Combine multiple models to improve accuracy
and robustness
• Reduce variance (averaging), sometimes bias
(boosting)
• Stronger generalization on tabular data;
competitive baseline in practice
• Natural parallelization (bagging) & strong off-
the-shelf performance (RF)

3. Bias–Variance Perspective (High-
Level)
• Prediction error = Bias² + Variance +
Irreducible noise
• Bagging ↓ variance by averaging unstable
learners (e.g., trees)
• Boosting can ↓ bias by sequentially correcting
residuals
• Diversity among base learners is key for gains

4. Ensemble Taxonomy
• Homogeneous vs. Heterogeneous (same vs.
different base models)
• Parallel (Bagging/Random Forest) vs.
Sequential (Boosting)
• Voting/Averaging (hard vs. soft)
• Stacking/Blending with a meta-learner

5. Bagging: Bootstrap Aggregating
• Train B models on bootstrap samples
(sampling with replacement)
• Each model is high-variance (e.g., deep tree)
→ averaging stabilizes
• Out-of-Bag (OOB) estimation: ~37% data not
seen by a given tree
• Key knobs: n_estimators (B), base_learner
complexity, max_samples

6. Random Forests (RF)
• Bagging + Random Subspace: each split
considers random subset of features
• De-correlates trees; often best default for
tabular data
• Classification: majority vote; Regression:
average
• Tune: n_estimators, max_features,
max_depth, min_samples_leaf, class_weight

7. Extremely Randomized Trees
(ExtraTrees)
• Randomized thresholds + random feature
subsets at each split
• Even more de-correlation; often faster
• May increase bias slightly but reduce variance
further

8. Boosting: Core Idea
• Train weak learners sequentially; each focuses
on previous errors
• Stage-wise additive modeling: f_{t}(x) = f_{t-1}
(x) + η * h_t(x)
• Requires careful regularization (learning rate,
depth, subsampling)

9. AdaBoost (Binary Classification)
• Re-weights samples; harder examples get
higher weight
• Weak learner: typically shallow decision trees
(stumps)
• Final prediction: weighted vote of weak
learners
• Sensitive to noise/outliers; strong on clean,
small-to-medium data

10. Gradient Boosting (GBDT/GBM)
• Fit new tree to negative gradient of loss
(residuals)
• Key hyperparameters: n_estimators,
learning_rate, max_depth (or max_leaves),
subsample
• Use early stopping with validation set to
prevent overfit
• Variants: XGBoost, LightGBM, CatBoost

11. XGBoost vs. LightGBM vs. CatBoost
(At a Glance)
• XGBoost: robust regularization, shrinkage,
column subsampling, wide ecosystem
• LightGBM: leaf-wise growth with depth limits;
fast on large, sparse datasets
• CatBoost: native categorical handling, ordered
boosting (reduces target leakage)
• Pick based on data size, sparsity, categorical
richness, and latency needs

12. Stacking (Meta-Learning)
• Level-0: diverse base models; Level-1: meta-
learner uses out-of-fold predictions
• Cross-validation is crucial to avoid leakage
• Use simple meta-learners first (logistic/linear)
to avoid overfitting
• Blending: holdout set for meta-features
(simpler, less data-efficient)

13. Voting & Averaging
• Hard voting: majority class label
• Soft voting: average predicted probabilities
(requires calibrated models)
• Weighted voting: weight by validation
performance or domain knowledge

14. Imbalanced Data Strategies
• Use class_weight='balanced'
(RF/AdaBoost/etc.) or sampling strategies
• Optimize thresholds using PR curves; use
AUPRC for evaluation
• Consider Balanced Random Forest,
EasyEnsemble, or focal loss (boosting variants)

15. Interpretability & Diagnostics
• Global: permutation importance, minimal
depth, gain statistics
• Local: SHAP values, tree path analysis,
counterfactuals
• Check calibration (reliability curves) for
probability outputs

16. Practical Tips
• Start with RF as a baseline for tabular
problems
• For boosting: tune learning_rate and trees
with early stopping
• Use OOB (bagging/RF) for quick model
iteration
• Cross-validate across time splits for temporal
data (avoid leakage)

17. Key Hyperparameters (Cheat
Sheet)
• RF: n_estimators↑, max_features (sqrt/log2),
min_samples_leaf (1–10)
• GBM: learning_rate (0.01–0.1), n_estimators
(100–1000+), max_depth (3–10), subsample
(0.6–0.9)
• AdaBoost: n_estimators (50–500),
learning_rate (0.01–1.0)
• Stacking: base diversity ↑, meta-learner
regularization (ridge/logistic)

18. Code: Bagging & Random Forest
(scikit-learn)
• from sklearn.ensemble import BaggingClassifier, RandomForestClassifier
from sklearn.model_selection import train_test_split, cross_val_score
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,
random_state=42)
bag = BaggingClassifier(n_estimators=200, max_samples=0.8, n_jobs=-1,
random_state=42)
rf = RandomForestClassifier(n_estimators=400, max_features='sqrt',
oob_score=True,
n_jobs=-1, random_state=42)
for model in [bag, rf]:
scores = cross_val_score(model, X_train, y_train, cv=5, n_jobs=-1)
print(model.__class__.__name__, scores.mean(), scores.std())
rf.fit(X_train, y_train); print('OOB:', rf.oob_score_)

19. Code: AdaBoost &
GradientBoosting (scikit-learn)
• from sklearn.ensemble import AdaBoostClassifier,
GradientBoostingClassifier
ada = AdaBoostClassifier(n_estimators=300, learning_rate=0.1,
random_state=42)
gb = GradientBoostingClassifier(n_estimators=500, learning_rate=0.05,
max_depth=3,
subsample=0.8, random_state=42)
ada.fit(X_train, y_train)
gb.fit(X_train, y_train)
print('AdaBoost test acc:', ada.score(X_test, y_test))
print('GB test acc:', gb.score(X_test, y_test))

20. Code: Stacking & Voting (scikit-
learn)
• from sklearn.ensemble import StackingClassifier, VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
base_learners = [
('rf', RandomForestClassifier(n_estimators=300, random_state=42)),
('svc', SVC(probability=True, kernel='rbf', C=2.0, gamma='scale',
random_state=42))
]
meta = LogisticRegression(max_iter=1000)
stack = StackingClassifier(estimators=base_learners,
final_estimator=meta, cv=5, n_jobs=-1)
vote = VotingClassifier(estimators=base_learners, voting='soft',
n_jobs=-1)
stack.fit(X_train, y_train)
vote.fit(X_train, y_train)
print('Stack acc:', stack.score(X_test, y_test))
print('Vote acc:', vote.score(X_test, y_test))

21. When to Use What
• RF: strong baseline for mixed/tabular data;
low tuning cost
• GBM (XGB/LGBM/CatBoost): when you need
top accuracy and can tune carefully
• Bagging (generic): unstable base learner &
small data → variance reduction
• Stacking: when diverse models each capture
different structure; ensure robust CV

22. Common Pitfalls & How to Avoid
• Data leakage in stacking/blending → use out-
of-fold predictions
• Overfitting with too-deep trees in boosting →
use small max_depth + regularization
• Poor probability calibration in RF/Boosting →
use calibration on validation set
• Distribution shift → evaluate with time-aware
or group-aware splits

23. Mini Case Sketch (Credit Risk)
• Goal: predict default; imbalanced (5%
positive)
• Baseline RF with class_weight='balanced' →
tune max_features
• Compare with LightGBM + early stopping;
evaluate AUPRC
• Stack RF + SVC + LR-meta for final model;
check calibration

24. References & Further Reading
• Breiman, L. (1996) Bagging Predictors; (2001)
Random Forests
• Freund & Schapire (1997) AdaBoost
• Friedman (2001) Greedy Function
Approximation (GBM)
• Chen & Guestrin (2016) XGBoost; Ke et al.
(2017) LightGBM; Dorogush et al. (2018)
CatBoost