Model Visualisation
Data science is a process of abstraction. In order to explain or to predict a real phenomena, the process starts with acquiring and refining the data. It then moves between the three layers of abstraction: transformations (data abstraction), visualizations (visual abstraction), and modeling (symbolic abstraction). All three layers of abstraction together build a truer (or closer) representation of the real phenomena. Data visualization (data-vis) helps us to understand the portrait and the shape of the data. The science of data-vis for exploratory data analysis is well developed for both static graphics (scatter-plot matrices, glyph-based approaches, geometric transforms like parallel coordinates) and interactive graphics (layering, brushing and linking, projections and tours). (For more information, see Amit Kapoor’s Strata + Hadoop World Singapore talk, Visualizing Multidimensional Data.) Though visualization is used in data science to understand the shape of the data, it’s not widely used for statistical models, which are evaluated based on numerical summaries. Amit Kapoor demonstrates extending visualization to the statistical model (model-vis), which aids in understanding the shape of the model, the impact of parameters and input data on the model, the fit of the model, and where it can be improved. Model visualization can help us to understand the shape of the model and compare it to the shape of the data. It allows us to see the fit of the model and understand where the fit can be improved. It also allows us to better understand the parameters in the model and how the model changes when the parameters change as well as how the parameters changes when the input data changes. The science and tools for model-vis are still very underdeveloped. Amit looks at practical examples of doing model-vis in regression (linear, lasso), classification (logistic, trees, LDA), and clustering (hierarchical) problems that can help us better understand the model. This includes exploring model-vis approaches that: Visualize the model in data space as opposed to data in model space - Visualize the entire space of models - Visualize the same model with varying tuning parameters - Visualize the same model with different input datasets - Visualize the process of model fitting as opposed to final result Integrating these approaches for model-vis as a part of model evaluation strengthens a data scientist’s understanding of the model and leads to better model building, complementing data-vis for fitting better models as well as communicating the insight from the data science process.
Recommended
More Related Content
Viewers also liked
Viewers also liked (12)
Similar to Model Visualisation
Similar to Model Visualisation (20)
More from Amit Kapoor
More from Amit Kapoor (18)
Recently uploaded
Recently uploaded (20)
Model Visualisation
- 2. Story ___ “We don’t see things as they are, we see them as we are.” — Anais Nin
- 3. The Blind Men & the Elephant ___ “And so these men of Indostan Disputed loud and long, Each in his own opinion Exceeding stiff and strong, Though each was partly in the right, And all were in the wrong.” — John Godfrey Saxe
- 4. The Elephant: Data ___ “Data is just a clue to the end truth” — Josh Smith
- 5. The Men: Building Models ___ "All models are wrong, but some are useful" — George Box
- 6. Ladder of Abstraction ___ Data Abstraction Visual Abstraction Model Abstraction
- 8. Why Build Models? ___ First Level of Ignorance "I know, what I don't know"
- 9. Why Visualise Models? ___ Second Level of Ignorance "I don't know, what I don't know"
- 10. Machine Learning (ML) Speak ___ Data Transformation Visual Exploration Model Building
- 11. ML Pipeline ___ Data Transformation — — — — Model Building (Tidy Data) ! ! ! Visual Exploration (Data-Vis)
- 12. ML Pipeline++ ___ Data Transformation — — — — Model Building (Tidy Data) (Tidy Model) ! ! ! ! ! ! Visual Exploration Model Exploration (Data-Vis) (Model-Vis)
- 13. Model-Vis Key Concept ___ Use visualisation to aid the transition of implicit knowledge in the data and your head to explicit knowledge in the model.
- 14. Model-Vis Approach ___ [0] Visualise the data space [1] Visualise the predictions in the data space [2] Visualise the errors in model fitting [3] Visualise with different model parameters [4] Visualise with different input datasets [5] Visualise the entire model space [6] Visualise the entire feature space [7] Visualise the many models together
- 15. Model-Vis Examples ___ Regression: Small Classification: Large p Regression: Large n
- 16. Model-Vis Examples ___ Cars (n < 50, p = 4) Digits (n ~ 5K, p = 785) Taxi (n ~ 10M, p = 20)
- 17. Regression: Small ___ Cars dataset - price vs kmpl Scraped from comparison website Refined & tidied up Base version for petrol cars Price < 1,000K, n = 42
- 18. brand model price kmpl type bhp Tata Nano 199 23.9 Hatchback 38 Suzuki Alto800 248 22.7 Hatchback 47 Hyundai EON 302 21.1 Hatchback 55 Nissan Datsun 312 20.6 Hatchback 67 ... ... ... ... ... ... Suzuki Ciaz 725 20.7 Sedan 91 Skoda Rapid 756 15.0 Sedan 104 Hyundai Verna 774 17.4 Sedan 106 VW Vento 785 16.1 Sedan 104
- 19. [0] Visualise the data space
- 20. [1] Visualise the predictions in data space
- 21. [2] Visualise the errors in model fitting
- 22. [3] Visualise with different model parameters
- 23. [4] Visualise with different input datasets
- 24. [5] Visualise the entire model space
- 25. [6] Visualise the entire feature space
- 26. [7] Visualise the many models together
- 27. Model-Vis Approach ___ [0] Visualise the data space [1] Visualise the predictions in the data space [2] Visualise the errors in model fitting [3] Visualise with different model parameters [4] Visualise with different input datasets [5] Visualise the entire model space [6] Visualise the entire feature space [7] Visualise the many models together
- 28. Model-Vis & ML Approach ___ [0] DATA VIS: the data space [1] PREDICTION: the predictions in the data space [2] VALIDATION: the errors in model fitting [3] TUNING: with different model parameters [4] BOOTSTRAP: with different input datasets [5] ENSEMBLE: the entire model space [6] FEATURE ENGG: the entire feature space [7] N-MODELS: the many models together
- 29. Move through Layers ___ Iterative, not linear Up and Down, not lateral Complementary, not exclusive
- 30. p/n/N Model-Vis challenge ___ p -- High dimensional data n -- Large and big data N -- Multiple models
- 31. Classification: 2 Class ___ MNIST - digit recognition Reduced to 2-class: 1 and 2 p = 784, 28 x 28 gray pixel map n > 5000
- 32. MNIST dataset: Examples of number 1 and 2
- 33. Visualise the data space
- 34. Identify the features - Symmetry & Intensity
- 35. Visualise the reduced feature space
- 36. Visualise the predictions in data space
- 37. Visualise the predictions boundaries
- 38. Visualise the errors in model fitting
- 39. Visualise with different model parameters
- 40. Easy to visualise errors in data space
- 41. How to scale for large p? ___ Curse of dimensionality Mesh approach computationally expensive Need to use projections
- 42. For entire feature space - PCA projection
- 43. Map the error on the projection
- 44. Cannot use any projection e.g. t-SNE
- 45. High-p Boundary Classifiers ___ Github: highdimensional-decision-boundary-plot
- 46. Regression: Large n ___ NYC Taxi Trip Data n ~ 10M (in just one month) p = 20, geo location (drop & pick up), fare breakup, passenger no. etc.
- 47. Data-Vis Issue ___ Plotting is hard e.g. alpha Sampling (~1%) may be effective Require careful tuning parameters e.g. overweighting unusual values
- 48. Binning Helps ___ “Bin - Summarize - Smooth: A framework for visualising big data” - Hadley Wickam Package in R: 'BigVis' (2013) Recent Interactive implementation in Python Package in Python: 'Datashader' (2016)
- 49. Vis Data Space ___ Plot the probability of getting a tip Start to see the patterns in the visualisation
- 50. Vis Predictions ___ Predict the probability of getting tip Simple Linear Model - drop coords, passenger count, time and day of week
- 51. Vis Errors ___ Visualise the errors in tip probability distribution
- 52. N-Models Challenge ___ Model Explosion Entire Model Space + Add Tuning Models + Add Bootstrap Models + Add Ensemble Models + Add Cross-Validation Models
- 53. N-Models Challenge ___ Keep track of prediction & errors Keep track of model output parameters
- 54. Tidy Model ___ Augment predictions & errors to dataset Create output parameters data frame Visualise like Tidy Data
- 55. Managing N-Models ___ "Managing Many Models in R" by Hadley Wickham "Broom Package in R by David Robinson
- 56. p/n/N Model-Vis challenge ___ p -- High dimensional data n -- Large and big data N -- Multiple models
- 57. p/n/N Model-Vis approach ___ p -- use Projections n -- use Binning or Sampling N -- use Tidy Model
- 58. Model-Vis ___ Similar challenges to Data-Vis More an Art, than a Science Essential in ML Model Pipeline Both to Explain or to Predict Scope for easier tooling
- 59. Model-Vis ___ Slides and Code http://modelvis.amitkaps.com Mini-Site and Explanation (by End of 2016)