SlideShare a Scribd company logo
1 of 32
Download to read offline
PresciientTraining
The Zen of Predictive Modelling
Eugene Dubossarsky
eugene@presciient.com
+61414573322
@cargomoose
What This Talk Isn’t About
But worth mentioning anyway:
R and The Sydney Users of R Forum
Analyst First
My Courses
Sydney Users of R Forum
• Just 1 shy of 500 members
• Regular meetups
• Study groups: introduction to R, “Machine
Learning for Hackers”, “Elements of Statistical
Learning”
R
• Do a Google image search for “ggplot2”
• Look for “r4stats”, “popularity”
• Join SURF
• Download R and start using it.
Analyst First
• Strategic, Cultural, Organisational, Human issues in
analytics
• Making analytics work in organisations
• Focus on the Human side of analytics
• International : Aust, NZ, Singapore, US, Japan, India, Hong
Kong
• analystfirst.com – see “core principles” and “what is analyst
first” ?
My Analytics Training Courses
• Predictive Modelling, Data Mining, R, Forensic
Analytics, Visualisation, Forecasting training courses
• Sydney, Melbourne, Canberra, Singapore
• Public and in-house
• Pre-prepared or customised
• Informal coaching/mentoring
• Strategy, Review, Advice and Assistance with Analytics Capability
Development in your organisation
The Zen of Predictive Modelling
PredictiveModels
• The Most Important Part of My “Predictive Modelling and Data Mining Course”
• What every user of predictive modelling should know
• What every manager and owner of predictive modelling capability must know
• “Open Secrets” known to the masters
The Zen of Predictive Modelling
PredictiveModels
• To save people time
• To see the forest for the trees
• To real value out of predictive analytics
The Right Point of View
PredictiveModels
Which is unlike the other two?
• Kohonen neural network
• Backpropagation neural network
• CART decision tree
The Right Point of View
PredictiveModels
Which is unlike the other two?
• CART decision tree
• Random Forest
• Support Vector Machine
The Right Point of View
PredictiveModels
Which is unlike the other two?
• Backpropagation Neural Network
• Linear Model
• CART Decision Tree
The Right Point of View
PredictiveModels
• Out Of Sample Accuracy
• Robustness (Out of Time Accuracy)
• Interpretability
• Implementability
The Right Point of View
PredictiveModels
• Out Of Sample Accuracy
• Robustness (Out of Time Accuracy)
• Interpretability
• Implementability
The Right Point of View
PredictiveModels
• Out Of Sample Accuracy
• Robustness (Out of Time Accuracy)
• Interpretability
• Implementability
The Right Point of View
PredictiveModels
Why build predictive models ?
• Insights
• Operational prediction
• “What-if” analysis
What Do All Predictive Models Have in Common ?
PredictiveModels
All Predictive Models:
• Have a training set of predictors and outcomes
• Probably have a cross-validation and test set of predictors and outcomes too.
• Are “fit” (optimsied) to minimise an error function between their actual and target
outcomes
• Are probably cross-validated to control overfitting on an out-of-sample data set
• Provide information on the relationship between the predictors and outcomes in
the data
• Can be used to score new data (make new predictions)
• Can be deployed in IT systems
• Can be interrogated for insights
• Are only as accurate as the data allows
• Provide a (fairly) accurate estimate of how well they will predict on new data
What Do All Predictive Model Insights Have in Common ?
PredictiveModels
All Predictive Models:
• Have variable importance measures (a number of which can be applied to any
model)
• Allow plotting predictors vs outcomes
• Have variable accuracy measures
• Can be resampled for more robust measures of accuracy
What Do All Predictive Model Predictions Have in Common?
PredictiveModels
All Predictive Models:
• Make predictions that are numeric : estimates of amount for regression, and
probability for classification
• All predictions are applications of the underlying model structure and parameters
(formula) to new predictor data sets
• All predictions are deterministic. Once a model is fitted, the predictions for a given
record will be the same every time. (Though the prediction may be a distribution
rather than a fixed point. Also, note that model fitting itself may be random – some
models may differ slightly each time they are fitted to the same data set)
How Do Predictive Model Families Differ?
PredictiveModels
• Classification vs Regression (most families can do both)
• Predictive accuracy vs insights
• Predictive accuracy vs stability
• Deterministic fitting vs randomised fitting
• Specific insights
• Structure and complexity
• Model assumptions (linear models, neural nets)
• Model structure (trees vs additive models vs SVM vs Neural Nets etc)
• The kinds of insights models provide
• Tendency to overfit (most, but not all)
• Dependence on metrics
• Sensitivity to missing values and categorical variables
Becoming a Master of Modelling Kung Fu
PredictiveModels
• Predictive models should be thought of as a “black box” initially, with the
characteristics that all models have in common recognised
• The focus should be on the data, not the model.
• Focusing on the specific characteristics of the model is important when: deciding on
the degree of accuracy desired, and the kinds of insights desired.
• It is good to start by working with one highly accurate, simple to use method
(randomForest is a good choice) and one or two highly interpretable models (rpart
decision trees and (generalised) linear models are good here.
• In fact, you can go a long way with just randomForest alone.
Becoming a Master of Modelling Kung Fu
PredictiveModels
• Master an adequate tool.
• Empty your mind of the tool . It is an illusion.
• Meditate on the data.
Meditating on Data
PredictiveModels
• Start with a highly accurate, nonparametric model you are comfortable with.
• The accuracy of a highly accuarate method is close to the theoretical limit of
accuracy possible on the data. World class experts may get closer, but not a whole
lot closer.
• So once you build the model, forget about the specific family you used. It is just a
tool.
• Each predictor may provide a unique amount of predictability to the model.
Measure it.
• Each predictor may be masked by other predictors. Be careful.
• Check relationships between data and strongest predictors
Meditating on Data
PredictiveModels
• There are at least 3 ways that a predictor can be important. They are not the same:
• What is the unique contribution of the predictor to the accuracy of the model
?
• What is the individual predictive power of the predictor alone ?
• How vital is the predictor to the structure of a particular model ?
• The first two are about the data, the third is more about the specific model. Which
is more important ?
Meditating on Data
PredictiveModels
• There are at least 3 ways that a predictor can be important. They are not the same:
• What is the unique contribution of the predictor to the accuracy of the model
?
• What is the individual predictive power of the predictor alone ?
• How vital is the predictor to the structure of a particular model ?
• The first two are about the data, the third is more about the specific model. Which
is more important ?
The Predictive Modelling Master’s Data Meditation
PredictiveModels
• Start with a highly accurate, nonparametric model you are comfortable with.
• The accuracy of a highly accuarate method is close to the theoretical limit of
accuracy possible on the data. World class experts may get closer, but not a whole
lot closer.
• So once you build the model, forget about the specific family you used. It is just a
tool.
• Measure model accuracy on out-of-sample data. Pay attention to any imbalances in
class or data subset accuracy.
• Measure model stability if necessary (it almost always is)
• Measure the importance of all variables, using the three main techniques.
• Measure again, holding some of the main predictors constant
• Measure (visualise) the effects of each predictor
• Build an interpretable model to help tell the story
The Master Sharpens the Sword : Getting More Accuracy
PredictiveModels
• There is never enough data
• Some model accuracy can result from trying other model families. Usually not
much, and not the best use of time, though for some reason the favourite activity of
new data miners.
• Some more model accuracy can result from tweaking model parameters. This is
perhaps less of a waste of time, but still not the ideal focus.
• The most dramatic improvement in model accuracy comes from new predictors.
• New predictors may be entirely new data sets, or complex new transformations of
existing data.
• A large, multi-tabular data set may well have information that has not been
captured in the data.
• The most common information of this type involves relations between individual
records. (eg. Time series windows, geographic neighbourhoods or social network
statistics per record)
Illusions On the Path
PredictiveModels
• Colossal wastes of time can include
• Trying to find the “right” model family
• Getting stuck in data preprocessing trying to get all the predictors “right”
• Trying to figure out what the targets should be (usually a sign that the business
problem is not well understood)
• Trying to “improve” the model without defining what that means
The Sun Tzu of Modelling: Be Prepared
PredictiveModels
• Know what you are modelling and for what purpose.
• Know what your target variable is. You may have more than one.
• Do not hesitate, model with what you have, and add more predictors later.
• Messy data is better than no data
• Use the right error measures
• Know the connection between the model and your business
• Evaluate, interrogate the model accordingly
• Always question the business value of the analysis
• Always be ready to suggest the business use of the analysis
• Don’t assume that the client understands what to do with the model
Strategy and Tactics
PredictiveModels
• Why are you (re)building the model?
• If Strategic: what is going to be done with the insights ? By whom ?
• If Operational: what are the key metrics – accuracy, value, deployability?
Questions ?
PredictiveModels

More Related Content

What's hot

Module 1 introduction to machine learning
Module 1  introduction to machine learningModule 1  introduction to machine learning
Module 1 introduction to machine learningSara Hooker
 
H2O World - Top 10 Data Science Pitfalls - Mark Landry
H2O World - Top 10 Data Science Pitfalls - Mark LandryH2O World - Top 10 Data Science Pitfalls - Mark Landry
H2O World - Top 10 Data Science Pitfalls - Mark LandrySri Ambati
 
6 Modelling Purposes
6 Modelling Purposes6 Modelling Purposes
6 Modelling PurposesBruce Edmonds
 
Artificial Intelligence Approaches
Artificial Intelligence  ApproachesArtificial Intelligence  Approaches
Artificial Intelligence ApproachesJincy Nelson
 
Decision support systems
Decision support systemsDecision support systems
Decision support systemsMR Z
 
Barga Data Science lecture 5
Barga Data Science lecture 5Barga Data Science lecture 5
Barga Data Science lecture 5Roger Barga
 
Analytical Skills Tools and Attitudes 2013 Survey lavastorm analytics
Analytical Skills Tools and Attitudes 2013 Survey   lavastorm analyticsAnalytical Skills Tools and Attitudes 2013 Survey   lavastorm analytics
Analytical Skills Tools and Attitudes 2013 Survey lavastorm analyticsjjoseph100
 
Creativity and Curiosity - The Trial and Error of Data Science
Creativity and Curiosity - The Trial and Error of Data ScienceCreativity and Curiosity - The Trial and Error of Data Science
Creativity and Curiosity - The Trial and Error of Data ScienceDamianMingle
 
MLSEV Virtual. State of the Art in ML
MLSEV Virtual. State of the Art in MLMLSEV Virtual. State of the Art in ML
MLSEV Virtual. State of the Art in MLBigML, Inc
 
RESEARCH in software engineering
RESEARCH in software engineeringRESEARCH in software engineering
RESEARCH in software engineeringIvano Malavolta
 
Module 4: Model Selection and Evaluation
Module 4: Model Selection and EvaluationModule 4: Model Selection and Evaluation
Module 4: Model Selection and EvaluationSara Hooker
 
20211229120253D6323_PERT 06_ Ensemble Learning.pptx
20211229120253D6323_PERT 06_ Ensemble Learning.pptx20211229120253D6323_PERT 06_ Ensemble Learning.pptx
20211229120253D6323_PERT 06_ Ensemble Learning.pptxRaflyRizky2
 
Presentation research- chapter 10-11 istiqlal
Presentation research- chapter 10-11 istiqlalPresentation research- chapter 10-11 istiqlal
Presentation research- chapter 10-11 istiqlalIstiqlalEid
 
Machine learning it is time...
Machine learning it is time...Machine learning it is time...
Machine learning it is time...Sandip Chatterjee
 
Understanding your data with Bayesian networks (in Python) by Bartek Wilczyns...
Understanding your data with Bayesian networks (in Python) by Bartek Wilczyns...Understanding your data with Bayesian networks (in Python) by Bartek Wilczyns...
Understanding your data with Bayesian networks (in Python) by Bartek Wilczyns...PyData
 

What's hot (20)

Managing machine learning
Managing machine learningManaging machine learning
Managing machine learning
 
Module 1 introduction to machine learning
Module 1  introduction to machine learningModule 1  introduction to machine learning
Module 1 introduction to machine learning
 
H2O World - Top 10 Data Science Pitfalls - Mark Landry
H2O World - Top 10 Data Science Pitfalls - Mark LandryH2O World - Top 10 Data Science Pitfalls - Mark Landry
H2O World - Top 10 Data Science Pitfalls - Mark Landry
 
6 Modelling Purposes
6 Modelling Purposes6 Modelling Purposes
6 Modelling Purposes
 
Artificial Intelligence Approaches
Artificial Intelligence  ApproachesArtificial Intelligence  Approaches
Artificial Intelligence Approaches
 
Rm tutorial
Rm tutorialRm tutorial
Rm tutorial
 
Decision support systems
Decision support systemsDecision support systems
Decision support systems
 
910 plenary Elder
910 plenary Elder910 plenary Elder
910 plenary Elder
 
Barga Data Science lecture 5
Barga Data Science lecture 5Barga Data Science lecture 5
Barga Data Science lecture 5
 
Analytical Skills Tools and Attitudes 2013 Survey lavastorm analytics
Analytical Skills Tools and Attitudes 2013 Survey   lavastorm analyticsAnalytical Skills Tools and Attitudes 2013 Survey   lavastorm analytics
Analytical Skills Tools and Attitudes 2013 Survey lavastorm analytics
 
Creativity and Curiosity - The Trial and Error of Data Science
Creativity and Curiosity - The Trial and Error of Data ScienceCreativity and Curiosity - The Trial and Error of Data Science
Creativity and Curiosity - The Trial and Error of Data Science
 
MLSEV Virtual. State of the Art in ML
MLSEV Virtual. State of the Art in MLMLSEV Virtual. State of the Art in ML
MLSEV Virtual. State of the Art in ML
 
RESEARCH in software engineering
RESEARCH in software engineeringRESEARCH in software engineering
RESEARCH in software engineering
 
Module 4: Model Selection and Evaluation
Module 4: Model Selection and EvaluationModule 4: Model Selection and Evaluation
Module 4: Model Selection and Evaluation
 
20211229120253D6323_PERT 06_ Ensemble Learning.pptx
20211229120253D6323_PERT 06_ Ensemble Learning.pptx20211229120253D6323_PERT 06_ Ensemble Learning.pptx
20211229120253D6323_PERT 06_ Ensemble Learning.pptx
 
Learning to learn Model Behavior: How to use "human-in-the-loop" to explain d...
Learning to learn Model Behavior: How to use "human-in-the-loop" to explain d...Learning to learn Model Behavior: How to use "human-in-the-loop" to explain d...
Learning to learn Model Behavior: How to use "human-in-the-loop" to explain d...
 
Presentation research- chapter 10-11 istiqlal
Presentation research- chapter 10-11 istiqlalPresentation research- chapter 10-11 istiqlal
Presentation research- chapter 10-11 istiqlal
 
Machine Learning
Machine LearningMachine Learning
Machine Learning
 
Machine learning it is time...
Machine learning it is time...Machine learning it is time...
Machine learning it is time...
 
Understanding your data with Bayesian networks (in Python) by Bartek Wilczyns...
Understanding your data with Bayesian networks (in Python) by Bartek Wilczyns...Understanding your data with Bayesian networks (in Python) by Bartek Wilczyns...
Understanding your data with Bayesian networks (in Python) by Bartek Wilczyns...
 

Viewers also liked

Best Practices In Predictive Analytics
Best Practices In Predictive AnalyticsBest Practices In Predictive Analytics
Best Practices In Predictive AnalyticsCapgemini
 
Seismi Case Study | Oracle Mining Event | Santiago de Chile | 15 March 2012
Seismi Case Study | Oracle Mining Event | Santiago de Chile | 15 March 2012Seismi Case Study | Oracle Mining Event | Santiago de Chile | 15 March 2012
Seismi Case Study | Oracle Mining Event | Santiago de Chile | 15 March 2012Seismi Limited
 
5 Benefits of Predictive Analytics for E-Commerce
5 Benefits of Predictive Analytics for E-Commerce5 Benefits of Predictive Analytics for E-Commerce
5 Benefits of Predictive Analytics for E-CommerceEdureka!
 
Creating Your First Predictive Model In Python
Creating Your First Predictive Model In PythonCreating Your First Predictive Model In Python
Creating Your First Predictive Model In PythonRobert Dempsey
 
Webinar: The Whys and Hows of Predictive Modelling
Webinar: The Whys and Hows of Predictive Modelling Webinar: The Whys and Hows of Predictive Modelling
Webinar: The Whys and Hows of Predictive Modelling Edureka!
 
Presentation Churn Management
Presentation Churn ManagementPresentation Churn Management
Presentation Churn Managementfarhanmajeed
 
Introduction to Machine Learning (case studies)
Introduction to Machine Learning (case studies)Introduction to Machine Learning (case studies)
Introduction to Machine Learning (case studies)Dmitry Efimov
 
churn prediction in telecom
churn prediction in telecom churn prediction in telecom
churn prediction in telecom Hong Bui Van
 
Artificial Intelligence, Predictive Modelling and Chatbots: Applications in P...
Artificial Intelligence, Predictive Modelling and Chatbots: Applications in P...Artificial Intelligence, Predictive Modelling and Chatbots: Applications in P...
Artificial Intelligence, Predictive Modelling and Chatbots: Applications in P...Nick Brown
 
Amazon Machine Learning Case Study: Predicting Customer Churn
Amazon Machine Learning Case Study: Predicting Customer ChurnAmazon Machine Learning Case Study: Predicting Customer Churn
Amazon Machine Learning Case Study: Predicting Customer ChurnAmazon Web Services
 
AWS re:Invent 2016: Predicting Customer Churn with Amazon Machine Learning (M...
AWS re:Invent 2016: Predicting Customer Churn with Amazon Machine Learning (M...AWS re:Invent 2016: Predicting Customer Churn with Amazon Machine Learning (M...
AWS re:Invent 2016: Predicting Customer Churn with Amazon Machine Learning (M...Amazon Web Services
 
Data analytics telecom churn final ppt
Data analytics telecom churn final ppt Data analytics telecom churn final ppt
Data analytics telecom churn final ppt Gunvansh Khanna
 
Recommender Systems (Machine Learning Summer School 2014 @ CMU)
Recommender Systems (Machine Learning Summer School 2014 @ CMU)Recommender Systems (Machine Learning Summer School 2014 @ CMU)
Recommender Systems (Machine Learning Summer School 2014 @ CMU)Xavier Amatriain
 

Viewers also liked (17)

Best Practices In Predictive Analytics
Best Practices In Predictive AnalyticsBest Practices In Predictive Analytics
Best Practices In Predictive Analytics
 
Seismi Case Study | Oracle Mining Event | Santiago de Chile | 15 March 2012
Seismi Case Study | Oracle Mining Event | Santiago de Chile | 15 March 2012Seismi Case Study | Oracle Mining Event | Santiago de Chile | 15 March 2012
Seismi Case Study | Oracle Mining Event | Santiago de Chile | 15 March 2012
 
5 Benefits of Predictive Analytics for E-Commerce
5 Benefits of Predictive Analytics for E-Commerce5 Benefits of Predictive Analytics for E-Commerce
5 Benefits of Predictive Analytics for E-Commerce
 
Creating Your First Predictive Model In Python
Creating Your First Predictive Model In PythonCreating Your First Predictive Model In Python
Creating Your First Predictive Model In Python
 
Webinar: The Whys and Hows of Predictive Modelling
Webinar: The Whys and Hows of Predictive Modelling Webinar: The Whys and Hows of Predictive Modelling
Webinar: The Whys and Hows of Predictive Modelling
 
Presentation Churn Management
Presentation Churn ManagementPresentation Churn Management
Presentation Churn Management
 
Introduction to Machine Learning (case studies)
Introduction to Machine Learning (case studies)Introduction to Machine Learning (case studies)
Introduction to Machine Learning (case studies)
 
Predictive analytics for E-commerce
Predictive analytics for E-commerce Predictive analytics for E-commerce
Predictive analytics for E-commerce
 
churn prediction in telecom
churn prediction in telecom churn prediction in telecom
churn prediction in telecom
 
Artificial Intelligence, Predictive Modelling and Chatbots: Applications in P...
Artificial Intelligence, Predictive Modelling and Chatbots: Applications in P...Artificial Intelligence, Predictive Modelling and Chatbots: Applications in P...
Artificial Intelligence, Predictive Modelling and Chatbots: Applications in P...
 
Amazon Machine Learning Case Study: Predicting Customer Churn
Amazon Machine Learning Case Study: Predicting Customer ChurnAmazon Machine Learning Case Study: Predicting Customer Churn
Amazon Machine Learning Case Study: Predicting Customer Churn
 
AWS re:Invent 2016: Predicting Customer Churn with Amazon Machine Learning (M...
AWS re:Invent 2016: Predicting Customer Churn with Amazon Machine Learning (M...AWS re:Invent 2016: Predicting Customer Churn with Amazon Machine Learning (M...
AWS re:Invent 2016: Predicting Customer Churn with Amazon Machine Learning (M...
 
Churn management
Churn managementChurn management
Churn management
 
Data analytics telecom churn final ppt
Data analytics telecom churn final ppt Data analytics telecom churn final ppt
Data analytics telecom churn final ppt
 
Churn Predictive Modelling
Churn Predictive ModellingChurn Predictive Modelling
Churn Predictive Modelling
 
Recommender Systems (Machine Learning Summer School 2014 @ CMU)
Recommender Systems (Machine Learning Summer School 2014 @ CMU)Recommender Systems (Machine Learning Summer School 2014 @ CMU)
Recommender Systems (Machine Learning Summer School 2014 @ CMU)
 
Big data ppt
Big data pptBig data ppt
Big data ppt
 

Similar to The zen of predictive modelling

Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez ...
Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez ...Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez ...
Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez ...tboubez
 
Hacking Predictive Modeling - RoadSec 2018
Hacking Predictive Modeling - RoadSec 2018Hacking Predictive Modeling - RoadSec 2018
Hacking Predictive Modeling - RoadSec 2018HJ van Veen
 
MACHINE LEARNING INTRODUCTION DIFFERENCE BETWEEN SUOERVISED , UNSUPERVISED AN...
MACHINE LEARNING INTRODUCTION DIFFERENCE BETWEEN SUOERVISED , UNSUPERVISED AN...MACHINE LEARNING INTRODUCTION DIFFERENCE BETWEEN SUOERVISED , UNSUPERVISED AN...
MACHINE LEARNING INTRODUCTION DIFFERENCE BETWEEN SUOERVISED , UNSUPERVISED AN...DurgaDevi310087
 
Modelling and evaluation
Modelling and evaluationModelling and evaluation
Modelling and evaluationeShikshak
 
Introduction to Data Science
Introduction to Data ScienceIntroduction to Data Science
Introduction to Data ScienceNiko Vuokko
 
Informs presentation new ppt
Informs presentation new pptInforms presentation new ppt
Informs presentation new pptSalford Systems
 
Improving AI Development - Dave Litwiller - Jan 11 2022 - Public
Improving AI Development - Dave Litwiller - Jan 11 2022 - PublicImproving AI Development - Dave Litwiller - Jan 11 2022 - Public
Improving AI Development - Dave Litwiller - Jan 11 2022 - PublicDave Litwiller
 
Statistical Learning and Model Selection module 2.pptx
Statistical Learning and Model Selection module 2.pptxStatistical Learning and Model Selection module 2.pptx
Statistical Learning and Model Selection module 2.pptxnagarajan740445
 
Michael Bolton - Heuristics: Solving Problems Rapidly
Michael Bolton - Heuristics: Solving Problems RapidlyMichael Bolton - Heuristics: Solving Problems Rapidly
Michael Bolton - Heuristics: Solving Problems RapidlyTEST Huddle
 
Barga Data Science lecture 2
Barga Data Science lecture 2Barga Data Science lecture 2
Barga Data Science lecture 2Roger Barga
 
Data Science 101
Data Science 101Data Science 101
Data Science 101ideatoipo
 
Statistics in the age of data science, issues you can not ignore
Statistics in the age of data science, issues you can not ignoreStatistics in the age of data science, issues you can not ignore
Statistics in the age of data science, issues you can not ignoreTuri, Inc.
 
Mixed Effects Models - Random Intercepts
Mixed Effects Models - Random InterceptsMixed Effects Models - Random Intercepts
Mixed Effects Models - Random InterceptsScott Fraundorf
 
Multi task learning stepping away from narrow expert models 7.11.18
Multi task learning stepping away from narrow expert models 7.11.18Multi task learning stepping away from narrow expert models 7.11.18
Multi task learning stepping away from narrow expert models 7.11.18Cloudera, Inc.
 

Similar to The zen of predictive modelling (20)

Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez ...
Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez ...Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez ...
Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez ...
 
Hacking Predictive Modeling - RoadSec 2018
Hacking Predictive Modeling - RoadSec 2018Hacking Predictive Modeling - RoadSec 2018
Hacking Predictive Modeling - RoadSec 2018
 
Introduction to Machine Learning
Introduction to Machine LearningIntroduction to Machine Learning
Introduction to Machine Learning
 
MACHINE LEARNING INTRODUCTION DIFFERENCE BETWEEN SUOERVISED , UNSUPERVISED AN...
MACHINE LEARNING INTRODUCTION DIFFERENCE BETWEEN SUOERVISED , UNSUPERVISED AN...MACHINE LEARNING INTRODUCTION DIFFERENCE BETWEEN SUOERVISED , UNSUPERVISED AN...
MACHINE LEARNING INTRODUCTION DIFFERENCE BETWEEN SUOERVISED , UNSUPERVISED AN...
 
Modelling and evaluation
Modelling and evaluationModelling and evaluation
Modelling and evaluation
 
Introduction to Data Science
Introduction to Data ScienceIntroduction to Data Science
Introduction to Data Science
 
Informs presentation new ppt
Informs presentation new pptInforms presentation new ppt
Informs presentation new ppt
 
Improving AI Development - Dave Litwiller - Jan 11 2022 - Public
Improving AI Development - Dave Litwiller - Jan 11 2022 - PublicImproving AI Development - Dave Litwiller - Jan 11 2022 - Public
Improving AI Development - Dave Litwiller - Jan 11 2022 - Public
 
Statistical Learning and Model Selection module 2.pptx
Statistical Learning and Model Selection module 2.pptxStatistical Learning and Model Selection module 2.pptx
Statistical Learning and Model Selection module 2.pptx
 
Michael Bolton - Heuristics: Solving Problems Rapidly
Michael Bolton - Heuristics: Solving Problems RapidlyMichael Bolton - Heuristics: Solving Problems Rapidly
Michael Bolton - Heuristics: Solving Problems Rapidly
 
Barga Data Science lecture 2
Barga Data Science lecture 2Barga Data Science lecture 2
Barga Data Science lecture 2
 
Data Analysis, Intepretation
Data Analysis, IntepretationData Analysis, Intepretation
Data Analysis, Intepretation
 
Data Science 101
Data Science 101Data Science 101
Data Science 101
 
Statistics in the age of data science, issues you can not ignore
Statistics in the age of data science, issues you can not ignoreStatistics in the age of data science, issues you can not ignore
Statistics in the age of data science, issues you can not ignore
 
Mixed Effects Models - Random Intercepts
Mixed Effects Models - Random InterceptsMixed Effects Models - Random Intercepts
Mixed Effects Models - Random Intercepts
 
lec1.ppt
lec1.pptlec1.ppt
lec1.ppt
 
Lecture 1
Lecture 1Lecture 1
Lecture 1
 
Multi task learning stepping away from narrow expert models 7.11.18
Multi task learning stepping away from narrow expert models 7.11.18Multi task learning stepping away from narrow expert models 7.11.18
Multi task learning stepping away from narrow expert models 7.11.18
 
Data Analysis
Data AnalysisData Analysis
Data Analysis
 
Ml2 production
Ml2 productionMl2 production
Ml2 production
 

Recently uploaded

MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotesMuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotesManik S Magar
 
Connecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfConnecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfNeo4j
 
React Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkReact Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkPixlogix Infotech
 
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...Wes McKinney
 
Landscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfLandscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfAarwolf Industries LLC
 
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...amber724300
 
React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...Karmanjay Verma
 
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...panagenda
 
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security ObservabilityGlenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security Observabilityitnewsafrica
 
WomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyoneWomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyoneUiPathCommunity
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality AssuranceInflectra
 
Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Hiroshi SHIBATA
 
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Strongerpanagenda
 
QMMS Lesson 2 - Using MS Excel Formula.pdf
QMMS Lesson 2 - Using MS Excel Formula.pdfQMMS Lesson 2 - Using MS Excel Formula.pdf
QMMS Lesson 2 - Using MS Excel Formula.pdfROWELL MARQUINA
 
A Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxA Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxAna-Maria Mihalceanu
 
A Framework for Development in the AI Age
A Framework for Development in the AI AgeA Framework for Development in the AI Age
A Framework for Development in the AI AgeCprime
 
Generative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfGenerative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfIngrid Airi González
 
Deliver Latency Free Customer Experience
Deliver Latency Free Customer ExperienceDeliver Latency Free Customer Experience
Deliver Latency Free Customer ExperienceOpsTree solutions
 
Decarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityDecarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityIES VE
 
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentEmixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentPim van der Noll
 

Recently uploaded (20)

MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotesMuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
 
Connecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfConnecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdf
 
React Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkReact Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App Framework
 
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
 
Landscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfLandscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdf
 
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
 
React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...
 
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
 
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security ObservabilityGlenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
 
WomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyoneWomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyone
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
 
Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024
 
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
 
QMMS Lesson 2 - Using MS Excel Formula.pdf
QMMS Lesson 2 - Using MS Excel Formula.pdfQMMS Lesson 2 - Using MS Excel Formula.pdf
QMMS Lesson 2 - Using MS Excel Formula.pdf
 
A Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxA Glance At The Java Performance Toolbox
A Glance At The Java Performance Toolbox
 
A Framework for Development in the AI Age
A Framework for Development in the AI AgeA Framework for Development in the AI Age
A Framework for Development in the AI Age
 
Generative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfGenerative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdf
 
Deliver Latency Free Customer Experience
Deliver Latency Free Customer ExperienceDeliver Latency Free Customer Experience
Deliver Latency Free Customer Experience
 
Decarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityDecarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a reality
 
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentEmixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
 

The zen of predictive modelling

  • 1. PresciientTraining The Zen of Predictive Modelling Eugene Dubossarsky eugene@presciient.com +61414573322 @cargomoose
  • 2. What This Talk Isn’t About But worth mentioning anyway: R and The Sydney Users of R Forum Analyst First My Courses
  • 3. Sydney Users of R Forum • Just 1 shy of 500 members • Regular meetups • Study groups: introduction to R, “Machine Learning for Hackers”, “Elements of Statistical Learning”
  • 4.
  • 5. R • Do a Google image search for “ggplot2” • Look for “r4stats”, “popularity” • Join SURF • Download R and start using it.
  • 6.
  • 7. Analyst First • Strategic, Cultural, Organisational, Human issues in analytics • Making analytics work in organisations • Focus on the Human side of analytics • International : Aust, NZ, Singapore, US, Japan, India, Hong Kong • analystfirst.com – see “core principles” and “what is analyst first” ?
  • 8. My Analytics Training Courses • Predictive Modelling, Data Mining, R, Forensic Analytics, Visualisation, Forecasting training courses • Sydney, Melbourne, Canberra, Singapore • Public and in-house • Pre-prepared or customised • Informal coaching/mentoring • Strategy, Review, Advice and Assistance with Analytics Capability Development in your organisation
  • 9. The Zen of Predictive Modelling PredictiveModels • The Most Important Part of My “Predictive Modelling and Data Mining Course” • What every user of predictive modelling should know • What every manager and owner of predictive modelling capability must know • “Open Secrets” known to the masters
  • 10. The Zen of Predictive Modelling PredictiveModels • To save people time • To see the forest for the trees • To real value out of predictive analytics
  • 11. The Right Point of View PredictiveModels Which is unlike the other two? • Kohonen neural network • Backpropagation neural network • CART decision tree
  • 12. The Right Point of View PredictiveModels Which is unlike the other two? • CART decision tree • Random Forest • Support Vector Machine
  • 13. The Right Point of View PredictiveModels Which is unlike the other two? • Backpropagation Neural Network • Linear Model • CART Decision Tree
  • 14. The Right Point of View PredictiveModels • Out Of Sample Accuracy • Robustness (Out of Time Accuracy) • Interpretability • Implementability
  • 15. The Right Point of View PredictiveModels • Out Of Sample Accuracy • Robustness (Out of Time Accuracy) • Interpretability • Implementability
  • 16. The Right Point of View PredictiveModels • Out Of Sample Accuracy • Robustness (Out of Time Accuracy) • Interpretability • Implementability
  • 17. The Right Point of View PredictiveModels Why build predictive models ? • Insights • Operational prediction • “What-if” analysis
  • 18. What Do All Predictive Models Have in Common ? PredictiveModels All Predictive Models: • Have a training set of predictors and outcomes • Probably have a cross-validation and test set of predictors and outcomes too. • Are “fit” (optimsied) to minimise an error function between their actual and target outcomes • Are probably cross-validated to control overfitting on an out-of-sample data set • Provide information on the relationship between the predictors and outcomes in the data • Can be used to score new data (make new predictions) • Can be deployed in IT systems • Can be interrogated for insights • Are only as accurate as the data allows • Provide a (fairly) accurate estimate of how well they will predict on new data
  • 19. What Do All Predictive Model Insights Have in Common ? PredictiveModels All Predictive Models: • Have variable importance measures (a number of which can be applied to any model) • Allow plotting predictors vs outcomes • Have variable accuracy measures • Can be resampled for more robust measures of accuracy
  • 20. What Do All Predictive Model Predictions Have in Common? PredictiveModels All Predictive Models: • Make predictions that are numeric : estimates of amount for regression, and probability for classification • All predictions are applications of the underlying model structure and parameters (formula) to new predictor data sets • All predictions are deterministic. Once a model is fitted, the predictions for a given record will be the same every time. (Though the prediction may be a distribution rather than a fixed point. Also, note that model fitting itself may be random – some models may differ slightly each time they are fitted to the same data set)
  • 21. How Do Predictive Model Families Differ? PredictiveModels • Classification vs Regression (most families can do both) • Predictive accuracy vs insights • Predictive accuracy vs stability • Deterministic fitting vs randomised fitting • Specific insights • Structure and complexity • Model assumptions (linear models, neural nets) • Model structure (trees vs additive models vs SVM vs Neural Nets etc) • The kinds of insights models provide • Tendency to overfit (most, but not all) • Dependence on metrics • Sensitivity to missing values and categorical variables
  • 22. Becoming a Master of Modelling Kung Fu PredictiveModels • Predictive models should be thought of as a “black box” initially, with the characteristics that all models have in common recognised • The focus should be on the data, not the model. • Focusing on the specific characteristics of the model is important when: deciding on the degree of accuracy desired, and the kinds of insights desired. • It is good to start by working with one highly accurate, simple to use method (randomForest is a good choice) and one or two highly interpretable models (rpart decision trees and (generalised) linear models are good here. • In fact, you can go a long way with just randomForest alone.
  • 23. Becoming a Master of Modelling Kung Fu PredictiveModels • Master an adequate tool. • Empty your mind of the tool . It is an illusion. • Meditate on the data.
  • 24. Meditating on Data PredictiveModels • Start with a highly accurate, nonparametric model you are comfortable with. • The accuracy of a highly accuarate method is close to the theoretical limit of accuracy possible on the data. World class experts may get closer, but not a whole lot closer. • So once you build the model, forget about the specific family you used. It is just a tool. • Each predictor may provide a unique amount of predictability to the model. Measure it. • Each predictor may be masked by other predictors. Be careful. • Check relationships between data and strongest predictors
  • 25. Meditating on Data PredictiveModels • There are at least 3 ways that a predictor can be important. They are not the same: • What is the unique contribution of the predictor to the accuracy of the model ? • What is the individual predictive power of the predictor alone ? • How vital is the predictor to the structure of a particular model ? • The first two are about the data, the third is more about the specific model. Which is more important ?
  • 26. Meditating on Data PredictiveModels • There are at least 3 ways that a predictor can be important. They are not the same: • What is the unique contribution of the predictor to the accuracy of the model ? • What is the individual predictive power of the predictor alone ? • How vital is the predictor to the structure of a particular model ? • The first two are about the data, the third is more about the specific model. Which is more important ?
  • 27. The Predictive Modelling Master’s Data Meditation PredictiveModels • Start with a highly accurate, nonparametric model you are comfortable with. • The accuracy of a highly accuarate method is close to the theoretical limit of accuracy possible on the data. World class experts may get closer, but not a whole lot closer. • So once you build the model, forget about the specific family you used. It is just a tool. • Measure model accuracy on out-of-sample data. Pay attention to any imbalances in class or data subset accuracy. • Measure model stability if necessary (it almost always is) • Measure the importance of all variables, using the three main techniques. • Measure again, holding some of the main predictors constant • Measure (visualise) the effects of each predictor • Build an interpretable model to help tell the story
  • 28. The Master Sharpens the Sword : Getting More Accuracy PredictiveModels • There is never enough data • Some model accuracy can result from trying other model families. Usually not much, and not the best use of time, though for some reason the favourite activity of new data miners. • Some more model accuracy can result from tweaking model parameters. This is perhaps less of a waste of time, but still not the ideal focus. • The most dramatic improvement in model accuracy comes from new predictors. • New predictors may be entirely new data sets, or complex new transformations of existing data. • A large, multi-tabular data set may well have information that has not been captured in the data. • The most common information of this type involves relations between individual records. (eg. Time series windows, geographic neighbourhoods or social network statistics per record)
  • 29. Illusions On the Path PredictiveModels • Colossal wastes of time can include • Trying to find the “right” model family • Getting stuck in data preprocessing trying to get all the predictors “right” • Trying to figure out what the targets should be (usually a sign that the business problem is not well understood) • Trying to “improve” the model without defining what that means
  • 30. The Sun Tzu of Modelling: Be Prepared PredictiveModels • Know what you are modelling and for what purpose. • Know what your target variable is. You may have more than one. • Do not hesitate, model with what you have, and add more predictors later. • Messy data is better than no data • Use the right error measures • Know the connection between the model and your business • Evaluate, interrogate the model accordingly • Always question the business value of the analysis • Always be ready to suggest the business use of the analysis • Don’t assume that the client understands what to do with the model
  • 31. Strategy and Tactics PredictiveModels • Why are you (re)building the model? • If Strategic: what is going to be done with the insights ? By whom ? • If Operational: what are the key metrics – accuracy, value, deployability?