SlideShare a Scribd company logo
1 of 32
Download to read offline
A Deep Dive Into
Gradient Boosting
Daniel Kohlsdorf
2
Why Gradient Boosting?
3
1. Outlier Filtering For Job Recommendations
2. Reranking Job Recommendations
3. Classifying Profiles:
1. Willingness To Change Job
2. Discipline
Since Then …
4
Decision Trees, Gradient Boosting and Application(s)
5
Decision Trees
6
Decision Trees
7
Goal
1. Partition input space
2. Pure class distribution in each partition
8
Decision Trees: Guillotine cuts
9
Decision Trees: Guillotine cuts
10
Decision Trees: Guillotine cuts
11
Finding The Best Split
age
0
12,5
25
37,5
50
Spongebob CrimeShow
crime show fans
sponge bob fans
Finding The Best Split
crime show fans
sponge bob fans
age
0
25
50
75
100
Spongebob CrimeShow
0
17,5
35
52,5
70
Spongebob CrimeShow
Finding The Best Split
crime show fans
sponge bob fans
age
0
25
50
75
100
Spongebob CrimeShow
0
25
50
75
100
Spongebob CrimeShow
14
• Choose the best split based on class distribution impurity
• Common measure is entropy.
• Choose the split that minimizes impurity the most:
Finding the best split
0
12,5
25
37,5
50
Spongebob CrimeShow
0
25
50
75
100
Spongebob
0
25
50
75
100
CrimeShow
15
Greedily Constructing A Decision Tree
16
Greedily Constructing A Decision Tree
17
Greedily Constructing A Decision Tree
18
Greedily Constructing A Decision Tree
19
Gradient Boosting
One Tree Is Not Enough
20
1. Weighted combination of weak learners
2. Prediction is based on comitee votes
3. Boosting:
1. Train ensemble one weak learner at the time
2. Focus new learners on wrongly predicted examples
Ensemble Methods
21
1. Learn a regressor
2. Compute the error residual (Gradient in deep learning)
3. Then build a new model to predict that residual
Gradient Boosting
22
Gradient Boosting
Our model: For each datapoint return 0
23
Gradient Boosting
24
Gradient Boosting
25
Building a Decision Tree from Gradients
crime show fans
sponge bob fans
age
We have a gradient for each training example
Return pooled gradients instead of class
26
Building a Decision Tree from Gradients
crime show fans
sponge bob fans
age
Impurity is the mangnitude of the pooled gradient
27
1. RMSE [Prediction]
2. Sigmoid [Binary Classification]
3. Softmax [Multiclass Classification]
4. Ranking Loss [Ranking]
Not just regression
28
Gradient Boosting
@Xing
29
Outlier Filtering
30
31
32
Thanks
daniel.kohlsdorf@xing.com
http://daniel-at-world.blogspot.com

More Related Content

Similar to PyData Hamburg - Gradient Boosting

Liberating Structures 2 with blended f2f/online participation at #sfaddis
Liberating Structures 2 with blended f2f/online participation at #sfaddisLiberating Structures 2 with blended f2f/online participation at #sfaddis
Liberating Structures 2 with blended f2f/online participation at #sfaddisEuforic Services
 
Successful Project Presentation
Successful Project PresentationSuccessful Project Presentation
Successful Project PresentationColin Sullivan
 
Theory Of Constraints - Agile Tour 2013 Craig Strong & Daryn Holmes
Theory Of Constraints - Agile Tour 2013 Craig Strong &  Daryn HolmesTheory Of Constraints - Agile Tour 2013 Craig Strong &  Daryn Holmes
Theory Of Constraints - Agile Tour 2013 Craig Strong & Daryn Holmesstrongandagile.co.uk
 
4.1 solution interviews.pptx
4.1 solution interviews.pptx4.1 solution interviews.pptx
4.1 solution interviews.pptxhsstartup
 
Webinar: Remote Learning - How to Brainstorm Lean Six Sigma Improvement Ideas
Webinar: Remote Learning - How to Brainstorm Lean Six Sigma Improvement IdeasWebinar: Remote Learning - How to Brainstorm Lean Six Sigma Improvement Ideas
Webinar: Remote Learning - How to Brainstorm Lean Six Sigma Improvement IdeasGoLeanSixSigma.com
 
Extreme editing
Extreme editingExtreme editing
Extreme editingLeah Guren
 
Question generating process (2)
Question generating process (2)Question generating process (2)
Question generating process (2)15waddelli1
 
Question generating process
Question generating processQuestion generating process
Question generating processCodyloklamcheung
 
Question Generating Process
Question Generating Process Question Generating Process
Question Generating Process ak3069
 
Question generating process
Question generating processQuestion generating process
Question generating processsorbieku
 
Agile DC Lead Time
Agile DC Lead TimeAgile DC Lead Time
Agile DC Lead Timeazheglov
 
How To (Not) Open Source - Javazone, Oslo 2014
How To (Not) Open Source - Javazone, Oslo 2014How To (Not) Open Source - Javazone, Oslo 2014
How To (Not) Open Source - Javazone, Oslo 2014gdusbabek
 
'Top Challenges We Face In IT Today To Hinder Our Pursuit Of Quality' by Lloy...
'Top Challenges We Face In IT Today To Hinder Our Pursuit Of Quality' by Lloy...'Top Challenges We Face In IT Today To Hinder Our Pursuit Of Quality' by Lloy...
'Top Challenges We Face In IT Today To Hinder Our Pursuit Of Quality' by Lloy...TEST Huddle
 
Global Day of Coderetreat Munich 2018
Global Day of Coderetreat Munich 2018Global Day of Coderetreat Munich 2018
Global Day of Coderetreat Munich 2018David Völkel
 
Cynthia Lee ITEM 2018
Cynthia Lee ITEM 2018Cynthia Lee ITEM 2018
Cynthia Lee ITEM 2018ITEM
 

Similar to PyData Hamburg - Gradient Boosting (16)

Liberating Structures 2 with blended f2f/online participation at #sfaddis
Liberating Structures 2 with blended f2f/online participation at #sfaddisLiberating Structures 2 with blended f2f/online participation at #sfaddis
Liberating Structures 2 with blended f2f/online participation at #sfaddis
 
Successful Project Presentation
Successful Project PresentationSuccessful Project Presentation
Successful Project Presentation
 
Theory Of Constraints - Agile Tour 2013 Craig Strong & Daryn Holmes
Theory Of Constraints - Agile Tour 2013 Craig Strong &  Daryn HolmesTheory Of Constraints - Agile Tour 2013 Craig Strong &  Daryn Holmes
Theory Of Constraints - Agile Tour 2013 Craig Strong & Daryn Holmes
 
4.1 solution interviews.pptx
4.1 solution interviews.pptx4.1 solution interviews.pptx
4.1 solution interviews.pptx
 
Kanban for Business
Kanban for BusinessKanban for Business
Kanban for Business
 
Webinar: Remote Learning - How to Brainstorm Lean Six Sigma Improvement Ideas
Webinar: Remote Learning - How to Brainstorm Lean Six Sigma Improvement IdeasWebinar: Remote Learning - How to Brainstorm Lean Six Sigma Improvement Ideas
Webinar: Remote Learning - How to Brainstorm Lean Six Sigma Improvement Ideas
 
Extreme editing
Extreme editingExtreme editing
Extreme editing
 
Question generating process (2)
Question generating process (2)Question generating process (2)
Question generating process (2)
 
Question generating process
Question generating processQuestion generating process
Question generating process
 
Question Generating Process
Question Generating Process Question Generating Process
Question Generating Process
 
Question generating process
Question generating processQuestion generating process
Question generating process
 
Agile DC Lead Time
Agile DC Lead TimeAgile DC Lead Time
Agile DC Lead Time
 
How To (Not) Open Source - Javazone, Oslo 2014
How To (Not) Open Source - Javazone, Oslo 2014How To (Not) Open Source - Javazone, Oslo 2014
How To (Not) Open Source - Javazone, Oslo 2014
 
'Top Challenges We Face In IT Today To Hinder Our Pursuit Of Quality' by Lloy...
'Top Challenges We Face In IT Today To Hinder Our Pursuit Of Quality' by Lloy...'Top Challenges We Face In IT Today To Hinder Our Pursuit Of Quality' by Lloy...
'Top Challenges We Face In IT Today To Hinder Our Pursuit Of Quality' by Lloy...
 
Global Day of Coderetreat Munich 2018
Global Day of Coderetreat Munich 2018Global Day of Coderetreat Munich 2018
Global Day of Coderetreat Munich 2018
 
Cynthia Lee ITEM 2018
Cynthia Lee ITEM 2018Cynthia Lee ITEM 2018
Cynthia Lee ITEM 2018
 

Recently uploaded

Rithik Kumar Singh codealpha pythohn.pdf
Rithik Kumar Singh codealpha pythohn.pdfRithik Kumar Singh codealpha pythohn.pdf
Rithik Kumar Singh codealpha pythohn.pdfrahulyadav957181
 
Digital Marketing Plan, how digital marketing works
Digital Marketing Plan, how digital marketing worksDigital Marketing Plan, how digital marketing works
Digital Marketing Plan, how digital marketing worksdeepakthakur548787
 
Principles and Practices of Data Visualization
Principles and Practices of Data VisualizationPrinciples and Practices of Data Visualization
Principles and Practices of Data VisualizationKianJazayeri1
 
English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdf
English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdfEnglish-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdf
English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdfblazblazml
 
Role of Consumer Insights in business transformation
Role of Consumer Insights in business transformationRole of Consumer Insights in business transformation
Role of Consumer Insights in business transformationAnnie Melnic
 
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...Jack Cole
 
modul pembelajaran robotic Workshop _ by Slidesgo.pptx
modul pembelajaran robotic Workshop _ by Slidesgo.pptxmodul pembelajaran robotic Workshop _ by Slidesgo.pptx
modul pembelajaran robotic Workshop _ by Slidesgo.pptxaleedritatuxx
 
World Economic Forum Metaverse Ecosystem By Utpal Chakraborty.pdf
World Economic Forum Metaverse Ecosystem By Utpal Chakraborty.pdfWorld Economic Forum Metaverse Ecosystem By Utpal Chakraborty.pdf
World Economic Forum Metaverse Ecosystem By Utpal Chakraborty.pdfsimulationsindia
 
Bank Loan Approval Analysis: A Comprehensive Data Analysis Project
Bank Loan Approval Analysis: A Comprehensive Data Analysis ProjectBank Loan Approval Analysis: A Comprehensive Data Analysis Project
Bank Loan Approval Analysis: A Comprehensive Data Analysis ProjectBoston Institute of Analytics
 
Digital Indonesia Report 2024 by We Are Social .pdf
Digital Indonesia Report 2024 by We Are Social .pdfDigital Indonesia Report 2024 by We Are Social .pdf
Digital Indonesia Report 2024 by We Are Social .pdfNicoChristianSunaryo
 
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...Boston Institute of Analytics
 
Statistics For Management by Richard I. Levin 8ed.pdf
Statistics For Management by Richard I. Levin 8ed.pdfStatistics For Management by Richard I. Levin 8ed.pdf
Statistics For Management by Richard I. Levin 8ed.pdfnikeshsingh56
 
IBEF report on the Insurance market in India
IBEF report on the Insurance market in IndiaIBEF report on the Insurance market in India
IBEF report on the Insurance market in IndiaManalVerma4
 
knowledge representation in artificial intelligence
knowledge representation in artificial intelligenceknowledge representation in artificial intelligence
knowledge representation in artificial intelligencePriyadharshiniG41
 
Data Factory in Microsoft Fabric (MsBIP #82)
Data Factory in Microsoft Fabric (MsBIP #82)Data Factory in Microsoft Fabric (MsBIP #82)
Data Factory in Microsoft Fabric (MsBIP #82)Cathrine Wilhelmsen
 
Introduction to Mongo DB-open-­‐source, high-­‐performance, document-­‐orient...
Introduction to Mongo DB-open-­‐source, high-­‐performance, document-­‐orient...Introduction to Mongo DB-open-­‐source, high-­‐performance, document-­‐orient...
Introduction to Mongo DB-open-­‐source, high-­‐performance, document-­‐orient...boychatmate1
 
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...Boston Institute of Analytics
 
What To Do For World Nature Conservation Day by Slidesgo.pptx
What To Do For World Nature Conservation Day by Slidesgo.pptxWhat To Do For World Nature Conservation Day by Slidesgo.pptx
What To Do For World Nature Conservation Day by Slidesgo.pptxSimranPal17
 

Recently uploaded (20)

Rithik Kumar Singh codealpha pythohn.pdf
Rithik Kumar Singh codealpha pythohn.pdfRithik Kumar Singh codealpha pythohn.pdf
Rithik Kumar Singh codealpha pythohn.pdf
 
Digital Marketing Plan, how digital marketing works
Digital Marketing Plan, how digital marketing worksDigital Marketing Plan, how digital marketing works
Digital Marketing Plan, how digital marketing works
 
Principles and Practices of Data Visualization
Principles and Practices of Data VisualizationPrinciples and Practices of Data Visualization
Principles and Practices of Data Visualization
 
English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdf
English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdfEnglish-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdf
English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdf
 
Role of Consumer Insights in business transformation
Role of Consumer Insights in business transformationRole of Consumer Insights in business transformation
Role of Consumer Insights in business transformation
 
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
 
modul pembelajaran robotic Workshop _ by Slidesgo.pptx
modul pembelajaran robotic Workshop _ by Slidesgo.pptxmodul pembelajaran robotic Workshop _ by Slidesgo.pptx
modul pembelajaran robotic Workshop _ by Slidesgo.pptx
 
World Economic Forum Metaverse Ecosystem By Utpal Chakraborty.pdf
World Economic Forum Metaverse Ecosystem By Utpal Chakraborty.pdfWorld Economic Forum Metaverse Ecosystem By Utpal Chakraborty.pdf
World Economic Forum Metaverse Ecosystem By Utpal Chakraborty.pdf
 
Bank Loan Approval Analysis: A Comprehensive Data Analysis Project
Bank Loan Approval Analysis: A Comprehensive Data Analysis ProjectBank Loan Approval Analysis: A Comprehensive Data Analysis Project
Bank Loan Approval Analysis: A Comprehensive Data Analysis Project
 
Digital Indonesia Report 2024 by We Are Social .pdf
Digital Indonesia Report 2024 by We Are Social .pdfDigital Indonesia Report 2024 by We Are Social .pdf
Digital Indonesia Report 2024 by We Are Social .pdf
 
Data Analysis Project: Stroke Prediction
Data Analysis Project: Stroke PredictionData Analysis Project: Stroke Prediction
Data Analysis Project: Stroke Prediction
 
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...
 
Statistics For Management by Richard I. Levin 8ed.pdf
Statistics For Management by Richard I. Levin 8ed.pdfStatistics For Management by Richard I. Levin 8ed.pdf
Statistics For Management by Richard I. Levin 8ed.pdf
 
IBEF report on the Insurance market in India
IBEF report on the Insurance market in IndiaIBEF report on the Insurance market in India
IBEF report on the Insurance market in India
 
2023 Survey Shows Dip in High School E-Cigarette Use
2023 Survey Shows Dip in High School E-Cigarette Use2023 Survey Shows Dip in High School E-Cigarette Use
2023 Survey Shows Dip in High School E-Cigarette Use
 
knowledge representation in artificial intelligence
knowledge representation in artificial intelligenceknowledge representation in artificial intelligence
knowledge representation in artificial intelligence
 
Data Factory in Microsoft Fabric (MsBIP #82)
Data Factory in Microsoft Fabric (MsBIP #82)Data Factory in Microsoft Fabric (MsBIP #82)
Data Factory in Microsoft Fabric (MsBIP #82)
 
Introduction to Mongo DB-open-­‐source, high-­‐performance, document-­‐orient...
Introduction to Mongo DB-open-­‐source, high-­‐performance, document-­‐orient...Introduction to Mongo DB-open-­‐source, high-­‐performance, document-­‐orient...
Introduction to Mongo DB-open-­‐source, high-­‐performance, document-­‐orient...
 
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
 
What To Do For World Nature Conservation Day by Slidesgo.pptx
What To Do For World Nature Conservation Day by Slidesgo.pptxWhat To Do For World Nature Conservation Day by Slidesgo.pptx
What To Do For World Nature Conservation Day by Slidesgo.pptx
 

PyData Hamburg - Gradient Boosting