A talk given by Eugene Dubossarsky on predictive analytics at the Big Data Analytics meetup in Sydney this month. The talk is available at http://www.youtube.com/watch?v=aG16YSFgtLY

1. PresciientTraining
The Zen of Predictive Modelling
Eugene Dubossarsky
eugene@presciient.com
+61414573322
@cargomoose

2. What This Talk Isn’t About
But worth mentioning anyway:
R and The Sydney Users of R Forum
Analyst First
My Courses

3. Sydney Users of R Forum
• Just 1 shy of 500 members
• Regular meetups
• Study groups: introduction to R, “Machine
Learning for Hackers”, “Elements of Statistical
Learning”

5. R
• Do a Google image search for “ggplot2”
• Look for “r4stats”, “popularity”
• Join SURF
• Download R and start using it.

7. Analyst First
• Strategic, Cultural, Organisational, Human issues in
analytics
• Making analytics work in organisations
• Focus on the Human side of analytics
• International : Aust, NZ, Singapore, US, Japan, India, Hong
Kong
• analystfirst.com – see “core principles” and “what is analyst
first” ?

8. My Analytics Training Courses
• Predictive Modelling, Data Mining, R, Forensic
Analytics, Visualisation, Forecasting training courses
• Sydney, Melbourne, Canberra, Singapore
• Public and in-house
• Pre-prepared or customised
• Informal coaching/mentoring
• Strategy, Review, Advice and Assistance with Analytics Capability
Development in your organisation

9. The Zen of Predictive Modelling
PredictiveModels
• The Most Important Part of My “Predictive Modelling and Data Mining Course”
• What every user of predictive modelling should know
• What every manager and owner of predictive modelling capability must know
• “Open Secrets” known to the masters

10. The Zen of Predictive Modelling
PredictiveModels
• To save people time
• To see the forest for the trees
• To real value out of predictive analytics

11. The Right Point of View
PredictiveModels
Which is unlike the other two?
• Kohonen neural network
• Backpropagation neural network
• CART decision tree

12. The Right Point of View
PredictiveModels
Which is unlike the other two?
• CART decision tree
• Random Forest
• Support Vector Machine

13. The Right Point of View
PredictiveModels
Which is unlike the other two?
• Backpropagation Neural Network
• Linear Model
• CART Decision Tree

14. The Right Point of View
PredictiveModels
• Out Of Sample Accuracy
• Robustness (Out of Time Accuracy)
• Interpretability
• Implementability

15. The Right Point of View
PredictiveModels
• Out Of Sample Accuracy
• Robustness (Out of Time Accuracy)
• Interpretability
• Implementability

16. The Right Point of View
PredictiveModels
• Out Of Sample Accuracy
• Robustness (Out of Time Accuracy)
• Interpretability
• Implementability

17. The Right Point of View
PredictiveModels
Why build predictive models ?
• Insights
• Operational prediction
• “What-if” analysis

18. What Do All Predictive Models Have in Common ?
PredictiveModels
All Predictive Models:
• Have a training set of predictors and outcomes
• Probably have a cross-validation and test set of predictors and outcomes too.
• Are “fit” (optimsied) to minimise an error function between their actual and target
outcomes
• Are probably cross-validated to control overfitting on an out-of-sample data set
• Provide information on the relationship between the predictors and outcomes in
the data
• Can be used to score new data (make new predictions)
• Can be deployed in IT systems
• Can be interrogated for insights
• Are only as accurate as the data allows
• Provide a (fairly) accurate estimate of how well they will predict on new data

19. What Do All Predictive Model Insights Have in Common ?
PredictiveModels
All Predictive Models:
• Have variable importance measures (a number of which can be applied to any
model)
• Allow plotting predictors vs outcomes
• Have variable accuracy measures
• Can be resampled for more robust measures of accuracy

20. What Do All Predictive Model Predictions Have in Common?
PredictiveModels
All Predictive Models:
• Make predictions that are numeric : estimates of amount for regression, and
probability for classification
• All predictions are applications of the underlying model structure and parameters
(formula) to new predictor data sets
• All predictions are deterministic. Once a model is fitted, the predictions for a given
record will be the same every time. (Though the prediction may be a distribution
rather than a fixed point. Also, note that model fitting itself may be random – some
models may differ slightly each time they are fitted to the same data set)

21. How Do Predictive Model Families Differ?
PredictiveModels
• Classification vs Regression (most families can do both)
• Predictive accuracy vs insights
• Predictive accuracy vs stability
• Deterministic fitting vs randomised fitting
• Specific insights
• Structure and complexity
• Model assumptions (linear models, neural nets)
• Model structure (trees vs additive models vs SVM vs Neural Nets etc)
• The kinds of insights models provide
• Tendency to overfit (most, but not all)
• Dependence on metrics
• Sensitivity to missing values and categorical variables

22. Becoming a Master of Modelling Kung Fu
PredictiveModels
• Predictive models should be thought of as a “black box” initially, with the
characteristics that all models have in common recognised
• The focus should be on the data, not the model.
• Focusing on the specific characteristics of the model is important when: deciding on
the degree of accuracy desired, and the kinds of insights desired.
• It is good to start by working with one highly accurate, simple to use method
(randomForest is a good choice) and one or two highly interpretable models (rpart
decision trees and (generalised) linear models are good here.
• In fact, you can go a long way with just randomForest alone.

23. Becoming a Master of Modelling Kung Fu
PredictiveModels
• Master an adequate tool.
• Empty your mind of the tool . It is an illusion.
• Meditate on the data.

24. Meditating on Data
PredictiveModels
• Start with a highly accurate, nonparametric model you are comfortable with.
• The accuracy of a highly accuarate method is close to the theoretical limit of
accuracy possible on the data. World class experts may get closer, but not a whole
lot closer.
• So once you build the model, forget about the specific family you used. It is just a
tool.
• Each predictor may provide a unique amount of predictability to the model.
Measure it.
• Each predictor may be masked by other predictors. Be careful.
• Check relationships between data and strongest predictors

25. Meditating on Data
PredictiveModels
• There are at least 3 ways that a predictor can be important. They are not the same:
• What is the unique contribution of the predictor to the accuracy of the model
?
• What is the individual predictive power of the predictor alone ?
• How vital is the predictor to the structure of a particular model ?
• The first two are about the data, the third is more about the specific model. Which
is more important ?

26. Meditating on Data
PredictiveModels
• There are at least 3 ways that a predictor can be important. They are not the same:
• What is the unique contribution of the predictor to the accuracy of the model
?
• What is the individual predictive power of the predictor alone ?
• How vital is the predictor to the structure of a particular model ?
• The first two are about the data, the third is more about the specific model. Which
is more important ?

27. The Predictive Modelling Master’s Data Meditation
PredictiveModels
• Start with a highly accurate, nonparametric model you are comfortable with.
• The accuracy of a highly accuarate method is close to the theoretical limit of
accuracy possible on the data. World class experts may get closer, but not a whole
lot closer.
• So once you build the model, forget about the specific family you used. It is just a
tool.
• Measure model accuracy on out-of-sample data. Pay attention to any imbalances in
class or data subset accuracy.
• Measure model stability if necessary (it almost always is)
• Measure the importance of all variables, using the three main techniques.
• Measure again, holding some of the main predictors constant
• Measure (visualise) the effects of each predictor
• Build an interpretable model to help tell the story

28. The Master Sharpens the Sword : Getting More Accuracy
PredictiveModels
• There is never enough data
• Some model accuracy can result from trying other model families. Usually not
much, and not the best use of time, though for some reason the favourite activity of
new data miners.
• Some more model accuracy can result from tweaking model parameters. This is
perhaps less of a waste of time, but still not the ideal focus.
• The most dramatic improvement in model accuracy comes from new predictors.
• New predictors may be entirely new data sets, or complex new transformations of
existing data.
• A large, multi-tabular data set may well have information that has not been
captured in the data.
• The most common information of this type involves relations between individual
records. (eg. Time series windows, geographic neighbourhoods or social network
statistics per record)

29. Illusions On the Path
PredictiveModels
• Colossal wastes of time can include
• Trying to find the “right” model family
• Getting stuck in data preprocessing trying to get all the predictors “right”
• Trying to figure out what the targets should be (usually a sign that the business
problem is not well understood)
• Trying to “improve” the model without defining what that means

30. The Sun Tzu of Modelling: Be Prepared
PredictiveModels
• Know what you are modelling and for what purpose.
• Know what your target variable is. You may have more than one.
• Do not hesitate, model with what you have, and add more predictors later.
• Messy data is better than no data
• Use the right error measures
• Know the connection between the model and your business
• Evaluate, interrogate the model accordingly
• Always question the business value of the analysis
• Always be ready to suggest the business use of the analysis
• Don’t assume that the client understands what to do with the model

31. Strategy and Tactics
PredictiveModels
• Why are you (re)building the model?
• If Strategic: what is going to be done with the insights ? By whom ?
• If Operational: what are the key metrics – accuracy, value, deployability?

32. Questions ?
PredictiveModels