Downloaded 24 times
Uploaded bymrphilroth
Data Visualization for Machine Learning
I talk about lessons I've learned maintaining an internal data visualization tool called Bit Inspector meant to aid the building of MalwareScore
Data Visualization for Machine Learning
- 1. VIZ4ML THE ROLE OFDATA VISUALIZATION IN IMPROVING MACHINE LEARNING MODELS BSidesLV 2017 Phil Roth
- 2. 2 Or…. Screenshots of thisinternal visualization tool I built to test MalwareScore along with lessons I learned building it
- 3. Phil Roth @mrphilroth proth@endgame.com PhD inphysics using ML Radar Imager Data Scientist at Endgame 3 whoami
- 4.
- 5. MalwareScore 5 MalwareScore is amachine learning first solution built for detecting and preventing malware.
- 6. MalwareScore Static features Deployed tocustomer machines Available at VirusTotal 6 https://www.virustotal.com/
- 7. Bit Inspector 7 Bit Inspectoris an internal tool for communicating progress, soliciting feedback, and identifying errors related to MalwareScore.
- 8. Bit Inspector 8 Built with Flask(http://flask.pocoo.org/) D3.js (https://d3js.org/) matplotlib (https://matplotlib.org/) seaborn (https://seaborn.pydata.org/) Connects to multiple internal data and processing resources
- 9.
- 10.
- 11.
- 12. ROC Curve 12 Area UnderReceiver Operating Characteristic Curve Created by plotting the true positive rate (TPR) against the false positive rate (FPR) at various threshold settings.
- 13. ROC Curve 13 import seabornas sns import matplotlib.pyplot as plt from sklearn.metrics import roc_curve y_true = metricsdf.y y_pred = metricsdf.y_pred_holdout fpr_plot, tpr_plot, _ = roc_curve(y_true, y_pred) plt.plot(fpr_plot, tpr_plot, lw=2, color="k")
- 14. Confusion Matrix 14 A tablewhere columns represent the predicted class and rows represent the actual class.
- 15. Confusion Matrix 15 import seabornas sns import matplotlib.pyplot as plt from sklearn.metrics import confusion_matrix y_true = metricsdf.y y_pred = metricsdf.y_pred_holdout matrix = confusion_matrix(y_true, y_pred > threshold) fig = sns.heatmap(matrix, annot=True, fmt="d") fig.invert_yaxis() plt.xlabel("Predicted") plt.ylabel("Actual")
- 16. Role of DataVisualization
- 17. Feature Experimentation 17 Byte Histogram SlidingWindow Byte Entropy 0 3 0 0 0 4 0 0 0 255 255 0 0 184 0 0 0 0 0 0 0 64 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 216 0 0 77 90 144 14 31 186 14 0 180 9 205 33 184 1 76 205 33 84 104 105 115 32 112 114 111 103 114 97 109 32 99 97 110 110 111 116 32 98 101 32 117 110 32 105 110 32 68 79 83 32 109 111 100 101 46 13 13 10 36 0 0 0 0 0 0 0 49 184 132 58 117 217 234 105 117 217 234 105 11 217 234 105 182 214 181 105 119 217 234 105 117 217 235 105 238 217 234 105 182 214 183 105 100 217 234 105 33 250 218 105 127 217 234 105 178 223 236 105 116 217 234 105 82 105 99 104 117 21 234 105 0 2 https://arxiv.org/pdf/1508.03096.pdf
- 18. Model Performance 18 Communicating resultsand performance
- 19. Finding Problems 19 There’s arudimentary system for gathering feedback and using it in future model trainings
- 20. Tracking Solutions 20 For eachmodel, problem areas are broken out and analyzed separately
- 21.
- 22.
- 23. Explainability 23 Can this visualizationbe understood on its own? Annotations Explanations in readable prose https://xkcd.com/1732/ …
- 24. Trustworthiness 24 https://www.economist.com/blogs/dailychart/2010/11/us_human_development_state Can you trustthe source of this visualization? Consistent styling Data sources listed Logos
- 25.
- 26.
- 27. Yourself 27 The purpose ofthe visualization is to convince yourself you’ve done something useful. Explainability Trustworthiness Beauty Less Time More Time
- 28. Yourself 28 “The first principleis that you must not fool yourself – and you are the easiest person to fool.” Richard Feynman
- 29. Yourself 29 Try something Visualize andinspect the results Looks wrong Think critically Woohoo! I’m done! Model Building Process How might I be fooling myself?
- 30. Data Science Team 30 Purposeis to communicate what you’ve done and get feedback you didn’t consider Add context (data sources, model parameters) Explainability Trustworthiness Beauty Less Time More Time
- 31. Domain Experts 31 Explainability Trustworthiness Beauty LessTime More Time Same. But now the context is domain specific. For me, hashes, PE header information, links to VirusTotal, etc…
- 32. Managers and Executives 32 Explainability Trustworthiness BeautyLess Time More Time Purpose is to communicate progress and current performance.
- 33.
- 34.
- 35.
- 36. Python Plotting 36 http://pythonplot.com/ Comparison ofplotting syntax between: pandas matplotlib plotnine ggplot2 (R) altair (planned)
- 37. Jupyter Notebooks 37 http://jupyter.org/ Excellent forexploratory data analysis Changes can be made and the results will update as fast as the code can run
- 38. Kibana 38 https://www.elastic.co/products/kibana Allows for rapidly buildingconstantly updating dashboards Works best when querying against data that’s in ElasticSearch Internal tool by @laborious_dtg
- 39. D3js 39 https://d3js.org/ Javascript Probably requires data translation Largetime commitment. Payoff is the customization possibilities (and thus trustworthiness/beauty).
- 40.
- 41. Yellowbrick 41 # Instantiate thevisualizer visualizer = Rank2D(features=features, algorithm='covariance') visualizer.fit(X, y) # Fit the data to the visualizer visualizer.transform(X) # Transform the data visualizer.poof() # Draw/show/poof the data The idea is to have prebaked model evaluation visualizations that adhere to the scikit-learn API
- 42. Facets 42 https://github.com/pair-code/facets It’s early, butso far this looks like the best method for truly responsive Exploratory Data Analysis that I’ve seen
- 43. Facets 43 time for ademo?
- 44.