The zen of predictive modelling

1,143 views

Published on

A talk given by Eugene Dubossarsky on predictive analytics at the Big Data Analytics meetup in Sydney this month. The talk is available at http://www.youtube.com/watch?v=aG16YSFgtLY

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,143
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
35
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

The zen of predictive modelling

  1. 1. PresciientTrainingThe Zen of Predictive ModellingEugene Dubossarskyeugene@presciient.com+61414573322@cargomoose
  2. 2. What This Talk Isn’t AboutBut worth mentioning anyway:R and The Sydney Users of R ForumAnalyst FirstMy Courses
  3. 3. Sydney Users of R Forum• Just 1 shy of 500 members• Regular meetups• Study groups: introduction to R, “MachineLearning for Hackers”, “Elements of StatisticalLearning”
  4. 4. R• Do a Google image search for “ggplot2”• Look for “r4stats”, “popularity”• Join SURF• Download R and start using it.
  5. 5. Analyst First• Strategic, Cultural, Organisational, Human issues inanalytics• Making analytics work in organisations• Focus on the Human side of analytics• International : Aust, NZ, Singapore, US, Japan, India, HongKong• analystfirst.com – see “core principles” and “what is analystfirst” ?
  6. 6. My Analytics Training Courses• Predictive Modelling, Data Mining, R, ForensicAnalytics, Visualisation, Forecasting training courses• Sydney, Melbourne, Canberra, Singapore• Public and in-house• Pre-prepared or customised• Informal coaching/mentoring• Strategy, Review, Advice and Assistance with Analytics CapabilityDevelopment in your organisation
  7. 7. The Zen of Predictive ModellingPredictiveModels• The Most Important Part of My “Predictive Modelling and Data Mining Course”• What every user of predictive modelling should know• What every manager and owner of predictive modelling capability must know• “Open Secrets” known to the masters
  8. 8. The Zen of Predictive ModellingPredictiveModels• To save people time• To see the forest for the trees• To real value out of predictive analytics
  9. 9. The Right Point of ViewPredictiveModelsWhich is unlike the other two?• Kohonen neural network• Backpropagation neural network• CART decision tree
  10. 10. The Right Point of ViewPredictiveModelsWhich is unlike the other two?• CART decision tree• Random Forest• Support Vector Machine
  11. 11. The Right Point of ViewPredictiveModelsWhich is unlike the other two?• Backpropagation Neural Network• Linear Model• CART Decision Tree
  12. 12. The Right Point of ViewPredictiveModels• Out Of Sample Accuracy• Robustness (Out of Time Accuracy)• Interpretability• Implementability
  13. 13. The Right Point of ViewPredictiveModels• Out Of Sample Accuracy• Robustness (Out of Time Accuracy)• Interpretability• Implementability
  14. 14. The Right Point of ViewPredictiveModels• Out Of Sample Accuracy• Robustness (Out of Time Accuracy)• Interpretability• Implementability
  15. 15. The Right Point of ViewPredictiveModelsWhy build predictive models ?• Insights• Operational prediction• “What-if” analysis
  16. 16. What Do All Predictive Models Have in Common ?PredictiveModelsAll Predictive Models:• Have a training set of predictors and outcomes• Probably have a cross-validation and test set of predictors and outcomes too.• Are “fit” (optimsied) to minimise an error function between their actual and targetoutcomes• Are probably cross-validated to control overfitting on an out-of-sample data set• Provide information on the relationship between the predictors and outcomes inthe data• Can be used to score new data (make new predictions)• Can be deployed in IT systems• Can be interrogated for insights• Are only as accurate as the data allows• Provide a (fairly) accurate estimate of how well they will predict on new data
  17. 17. What Do All Predictive Model Insights Have in Common ?PredictiveModelsAll Predictive Models:• Have variable importance measures (a number of which can be applied to anymodel)• Allow plotting predictors vs outcomes• Have variable accuracy measures• Can be resampled for more robust measures of accuracy
  18. 18. What Do All Predictive Model Predictions Have in Common?PredictiveModelsAll Predictive Models:• Make predictions that are numeric : estimates of amount for regression, andprobability for classification• All predictions are applications of the underlying model structure and parameters(formula) to new predictor data sets• All predictions are deterministic. Once a model is fitted, the predictions for a givenrecord will be the same every time. (Though the prediction may be a distributionrather than a fixed point. Also, note that model fitting itself may be random – somemodels may differ slightly each time they are fitted to the same data set)
  19. 19. How Do Predictive Model Families Differ?PredictiveModels• Classification vs Regression (most families can do both)• Predictive accuracy vs insights• Predictive accuracy vs stability• Deterministic fitting vs randomised fitting• Specific insights• Structure and complexity• Model assumptions (linear models, neural nets)• Model structure (trees vs additive models vs SVM vs Neural Nets etc)• The kinds of insights models provide• Tendency to overfit (most, but not all)• Dependence on metrics• Sensitivity to missing values and categorical variables
  20. 20. Becoming a Master of Modelling Kung FuPredictiveModels• Predictive models should be thought of as a “black box” initially, with thecharacteristics that all models have in common recognised• The focus should be on the data, not the model.• Focusing on the specific characteristics of the model is important when: deciding onthe degree of accuracy desired, and the kinds of insights desired.• It is good to start by working with one highly accurate, simple to use method(randomForest is a good choice) and one or two highly interpretable models (rpartdecision trees and (generalised) linear models are good here.• In fact, you can go a long way with just randomForest alone.
  21. 21. Becoming a Master of Modelling Kung FuPredictiveModels• Master an adequate tool.• Empty your mind of the tool . It is an illusion.• Meditate on the data.
  22. 22. Meditating on DataPredictiveModels• Start with a highly accurate, nonparametric model you are comfortable with.• The accuracy of a highly accuarate method is close to the theoretical limit ofaccuracy possible on the data. World class experts may get closer, but not a wholelot closer.• So once you build the model, forget about the specific family you used. It is just atool.• Each predictor may provide a unique amount of predictability to the model.Measure it.• Each predictor may be masked by other predictors. Be careful.• Check relationships between data and strongest predictors
  23. 23. Meditating on DataPredictiveModels• There are at least 3 ways that a predictor can be important. They are not the same:• What is the unique contribution of the predictor to the accuracy of the model?• What is the individual predictive power of the predictor alone ?• How vital is the predictor to the structure of a particular model ?• The first two are about the data, the third is more about the specific model. Whichis more important ?
  24. 24. Meditating on DataPredictiveModels• There are at least 3 ways that a predictor can be important. They are not the same:• What is the unique contribution of the predictor to the accuracy of the model?• What is the individual predictive power of the predictor alone ?• How vital is the predictor to the structure of a particular model ?• The first two are about the data, the third is more about the specific model. Whichis more important ?
  25. 25. The Predictive Modelling Master’s Data MeditationPredictiveModels• Start with a highly accurate, nonparametric model you are comfortable with.• The accuracy of a highly accuarate method is close to the theoretical limit ofaccuracy possible on the data. World class experts may get closer, but not a wholelot closer.• So once you build the model, forget about the specific family you used. It is just atool.• Measure model accuracy on out-of-sample data. Pay attention to any imbalances inclass or data subset accuracy.• Measure model stability if necessary (it almost always is)• Measure the importance of all variables, using the three main techniques.• Measure again, holding some of the main predictors constant• Measure (visualise) the effects of each predictor• Build an interpretable model to help tell the story
  26. 26. The Master Sharpens the Sword : Getting More AccuracyPredictiveModels• There is never enough data• Some model accuracy can result from trying other model families. Usually notmuch, and not the best use of time, though for some reason the favourite activity ofnew data miners.• Some more model accuracy can result from tweaking model parameters. This isperhaps less of a waste of time, but still not the ideal focus.• The most dramatic improvement in model accuracy comes from new predictors.• New predictors may be entirely new data sets, or complex new transformations ofexisting data.• A large, multi-tabular data set may well have information that has not beencaptured in the data.• The most common information of this type involves relations between individualrecords. (eg. Time series windows, geographic neighbourhoods or social networkstatistics per record)
  27. 27. Illusions On the PathPredictiveModels• Colossal wastes of time can include• Trying to find the “right” model family• Getting stuck in data preprocessing trying to get all the predictors “right”• Trying to figure out what the targets should be (usually a sign that the businessproblem is not well understood)• Trying to “improve” the model without defining what that means
  28. 28. The Sun Tzu of Modelling: Be PreparedPredictiveModels• Know what you are modelling and for what purpose.• Know what your target variable is. You may have more than one.• Do not hesitate, model with what you have, and add more predictors later.• Messy data is better than no data• Use the right error measures• Know the connection between the model and your business• Evaluate, interrogate the model accordingly• Always question the business value of the analysis• Always be ready to suggest the business use of the analysis• Don’t assume that the client understands what to do with the model
  29. 29. Strategy and TacticsPredictiveModels• Why are you (re)building the model?• If Strategic: what is going to be done with the insights ? By whom ?• If Operational: what are the key metrics – accuracy, value, deployability?
  30. 30. Questions ?PredictiveModels

×