H2O tutorial at Analyx, Poznan

1. Introduction to Machine Learning
with H2O and Python
Jo-fai (Joe) Chow
Data Scientist
joe@h2o.ai
@matlabulous
H2O Tutorial at Analyx
20th April, 2017

2. Slides and Code Examples:
bit.ly/joe_h2o_tutorials
2

3. About Me
• Civil (Water) Engineer
• 2010 – 2015
• Consultant (UK)
• Utilities
• Asset Management
• Constrained Optimization
• Industrial PhD (UK)
• Infrastructure Design Optimization
• Machine Learning +
Water Engineering
• Discovered H2O in 2014
• Data Scientist
• 2015
• Virgin Media (UK)
• Domino Data Lab (Silicon Valley,
US)
• 2016 – Present
• H2O.ai (Silicon Valley, US)
3

4. About Me
4

5. Side Project #1 – Crime Data Visualization
5
https://github.com/woobe/rApps/tree/master/crimemap
http://insidebigdata.com/2013/11/30/visualization-week-crimemap/

6. Side Project #2 – Data Visualization Contest
6
https://github.com/woobe/rugsmaps http://blog.revolutionanalytics.com/2014/08/winner-for-revolution-analytics-user-group-map-contest.html

7. Side Project #3
7
Developing R Packages for Fun
rPlotter (2014)

8. About Me
8
R + H2O + Domino for Kaggle
Guest Blog Post for Domino & H2O (2014)
• The Long Story
• bit.ly/joe_kaggle_story

9. Agenda
• About H2O.ai
• Company
• Machine Learning Platform
• Tutorial
• H2O Python Module
• Download & Install
• Step-by-Step Examples:
• Basic Data Import / Manipulation
• Regression & Classification (Basics)
• Regression & Classification (Advanced)
• Using H2O in the Cloud
9

10. Agenda
• About H2O.ai
• Company
• Machine Learning Platform
• Tutorial
• H2O Python Module
• Download & Install
• Step-by-Step Examples:
• Basic Data Import / Manipulation
• Regression & Classification (Basics)
• Regression & Classification (Advanced)
• Using H2O in the Cloud
10
Background Information
For beginners
As if I am working on
Kaggle competitions
Short Break

11. About H2O.ai
11

12. Company Overview
Founded 2011 Venture-backed, debuted in 2012
Products • H2O Open Source In-Memory AI Prediction Engine
• Sparkling Water
• Steam
Mission Operationalize Data Science, and provide a platform for users to build beautiful data products
Team 70 employees
• Distributed Systems Engineers doing Machine Learning
• World-class visualization designers
Headquarters Mountain View, CA
12

13. 13
Our Team
Joe

14. Scientific Advisory Council
14

15. 15

16. 0
10000
20000
30000
40000
50000
60000
70000
1-Jan-15 1-Jul-15 1-Jan-16 1-Oct-16
# H2O Users
H2O Community Growth
Tremendous Momentum Globally
65,000+ users globally
(Sept 2016)
• 65,000+ users from
~8,000 companies in 140
countries. Top 5 from:
Large User Circle
* DATA FROM GOOGLE ANALYTICS EMBEDDED IN THE END USER PRODUCT
16
0
2000
4000
6000
8000
10000
1-Jan-15 1-Jul-15 1-Jan-16 1-Oct-16
# Companies Using H2O ~8,000+ companies
(Sept 2016)
+127%
+60%

17. #AroundTheWorldWithH2Oai
17

18. H2O for Kaggle Competitions
18

19. H2O for Academic Research
19
http://www.sciencedirect.com/science/article/pii/S0377221716308657
https://arxiv.org/abs/1509.01199

20. Users In Various Verticals Adore H2O
Financial Insurance MarketingTelecom Healthcare
20

21. 21
Joe (2015)
http://www.h2o.ai/gartner-magic-quadrant/

22. 22
Check
out our
website
h2o.ai

23. H2O Machine Learning Platform
23

24. 24

25. 25

26. H2O Overview
26

27. HDFS
S3
NFS
Distributed
In-Memory
Load Data
Loss-less
Compression
H2O Compute Engine
Production Scoring Environment
Exploratory &
Descriptive
Analysis
Feature
Engineering &
Selection
Supervised &
Unsupervised
Modeling
Model
Evaluation &
Selection
Predict
Data & Model
Storage
Model Export:
Plain Old Java Object
Your
Imagination
Data Prep Export:
Plain Old Java Object
Local
SQL
High Level Architecture
27

28. HDFS
S3
NFS
Distributed
In-Memory
Load Data
Loss-less
Compression
H2O Compute Engine
Production Scoring Environment
Exploratory &
Descriptive
Analysis
Feature
Engineering &
Selection
Supervised &
Unsupervised
Modeling
Model
Evaluation &
Selection
Predict
Data & Model
Storage
Model Export:
Plain Old Java Object
Your
Imagination
Data Prep Export:
Plain Old Java Object
Local
SQL
High Level Architecture
28
Import Data from
Multiple Sources

29. HDFS
S3
NFS
Distributed
In-Memory
Load Data
Loss-less
Compression
H2O Compute Engine
Production Scoring Environment
Exploratory &
Descriptive
Analysis
Feature
Engineering &
Selection
Supervised &
Unsupervised
Modeling
Model
Evaluation &
Selection
Predict
Data & Model
Storage
Model Export:
Plain Old Java Object
Your
Imagination
Data Prep Export:
Plain Old Java Object
Local
SQL
High Level Architecture
29
Fast, Scalable & Distributed
Compute Engine Written in
Java

30. HDFS
S3
NFS
Distributed
In-Memory
Load Data
Loss-less
Compression
H2O Compute Engine
Production Scoring Environment
Exploratory &
Descriptive
Analysis
Feature
Engineering &
Selection
Supervised &
Unsupervised
Modeling
Model
Evaluation &
Selection
Predict
Data & Model
Storage
Model Export:
Plain Old Java Object
Your
Imagination
Data Prep Export:
Plain Old Java Object
Local
SQL
High Level Architecture
30
Fast, Scalable & Distributed
Compute Engine Written in
Java

31. Supervised Learning
• Generalized Linear Models: Binomial,
Gaussian, Gamma, Poisson and Tweedie
• Naïve Bayes
Statistical
Analysis
Ensembles
• Distributed Random Forest: Classification
or regression models
• Gradient Boosting Machine: Produces an
ensemble of decision trees with increasing
refined approximations
Deep Neural
Networks
• Deep learning: Create multi-layer feed
forward neural networks starting with an
input layer followed by multiple layers of
nonlinear transformations
Algorithms Overview
Unsupervised Learning
• K-means: Partitions observations into k
clusters/groups of the same spatial size.
Automatically detect optimal k
Clustering
Dimensionality
Reduction
• Principal Component Analysis: Linearly transforms
correlated variables to independent components
• Generalized Low Rank Models: extend the idea of
PCA to handle arbitrary data consisting of numerical,
Boolean, categorical, and missing data
Anomaly
Detection
• Autoencoders: Find outliers using a
nonlinear dimensionality reduction using
deep learning
31

32. H2O Deep Learning in Action
32

33. HDFS
S3
NFS
Distributed
In-Memory
Load Data
Loss-less
Compression
H2O Compute Engine
Production Scoring Environment
Exploratory &
Descriptive
Analysis
Feature
Engineering &
Selection
Supervised &
Unsupervised
Modeling
Model
Evaluation &
Selection
Predict
Data & Model
Storage
Model Export:
Plain Old Java Object
Your
Imagination
Data Prep Export:
Plain Old Java Object
Local
SQL
High Level Architecture
33
Multiple Interfaces

34. H2O + Python
34

35. H2O + R
35

36. 36
H2O Flow (Web) Interface

37. HDFS
S3
NFS
Distributed
In-Memory
Load Data
Loss-less
Compression
H2O Compute Engine
Production Scoring Environment
Exploratory &
Descriptive
Analysis
Feature
Engineering &
Selection
Supervised &
Unsupervised
Modeling
Model
Evaluation &
Selection
Predict
Data & Model
Storage
Model Export:
Plain Old Java Object
Your
Imagination
Data Prep Export:
Plain Old Java Object
Local
SQL
High Level Architecture
37
Export Standalone Models
for Production

38. 38
docs.h2o.ai

39. H2O + Python Tutorial
39

40. Learning Objectives
• Start and connect to a local H2O cluster from Python.
• Import data from Python data frames, local files or web.
• Perform basic data transformation and exploration.
• Train regression and classification models using various H2O machine
learning algorithms.
• Evaluate models and make predictions.
• Improve performance by tuning and stacking.
• Connect to H2O cluster in the cloud.
40

41. 41

42. Install H2O
h2o.ai -> Download -> Install in Python
42

43. 43

44. Start and Connect to a
Local H2O Cluster
py_01_data_in_h2o.ipynb
44

45. Local H2O Cluster
45
Import H2O module
Start a local H2O cluster
nthreads = -1 means
using ALL CPU resources

46. 46
Information of Cluster

47. Importing Data into H2O
py_01_data_in_h2o.ipynb
47

48. 48
Import data into H2O cluster
(instead of Python’s memory)

49. 49
Directly from data on the web

50. 50
Convert from Pandas
to H2O data frame

51. Basic Data Transformation &
Exploration
py_02_data_manipulation.ipynb
(see notebooks)
51

52. 52
The Classic
Titanic Dataset

53. 53
Only two unique values
(0 or 1)

54. 54
“enum” is the data type of
categorical data in Java
Convert numerical to
categorical values

55. 55
Only three unique values
(1, 2 or 3)

56. Regression Models (Basics)
py_03a_regression_basics.ipynb
56

57. Supervised Learning
• Generalized Linear Models: Binomial,
Gaussian, Gamma, Poisson and Tweedie
• Naïve Bayes
Statistical
Analysis
Ensembles
• Distributed Random Forest: Classification
or regression models
• Gradient Boosting Machine: Produces an
ensemble of decision trees with increasing
refined approximations
Deep Neural
Networks
• Deep learning: Create multi-layer feed
forward neural networks starting with an
input layer followed by multiple layers of
nonlinear transformations
Algorithms Overview
Unsupervised Learning
• K-means: Partitions observations into k
clusters/groups of the same spatial size.
Automatically detect optimal k
Clustering
Dimensionality
Reduction
• Principal Component Analysis: Linearly transforms
correlated variables to independent components
• Generalized Low Rank Models: extend the idea of
PCA to handle arbitrary data consisting of numerical,
Boolean, categorical, and missing data
Anomaly
Detection
• Autoencoders: Find outliers using a
nonlinear dimensionality reduction using
deep learning
57

58. 58
docs.h2o.ai

59. 59
11 Numerical Features
Target

60. 60
Define 11 Numerical
Features using their
Column Names

61. 61
Split dataset so we can
measure out-of-bag
performance later

62. 62
Basic H2O Usage for GLM

63. 63
Regression Performance – MSE
Lower the better

64. 64
Model Summary

65. 65
Evaluate model performance
using test set

66. 66

67. 67
API for other ML
algorithms

68. 68
API for other ML
algorithms

69. Classification Models (Basics)
py_04_classification_basics.ipynb
69

70. 70
Target

71. 71
Convert numerical to
categorical values

72. 72
Define features manually
Split dataset so we can
measure out-of-bag
performance later

73. 73
Basic H2O Usage for GLM

74. Classification Performance – Confusion Matrix
74

75. Confusion Matrix
75

76. 76
Model Summary

77. 77
Evaluate model performance
using test set

78. 78
Predicted
Class
Probabilities of Each Class

79. 79
API for other ML
algorithms

80. 80
API for other ML
algorithms

81. End of Basics
Let’s have a break ☺
81

82. Regression Models (Tuning)
py_03b_regression_grid_search.ipynb
82

83. Improving Model Performance (Step-by-Step)
83
Model Settings MSE (CV) MSE (Test)
GBM with default settings N/A 0.4551
GBM with manual settings N/A 0.4433
Manual settings + cross-validation 0.4502 0.4433
Manual + CV + early stopping 0.4429 0.4287
CV + early stopping + full grid search 0.4378 0.4196
CV + early stopping + random grid search 0.4227 0.4047
Stacking models from random grid search N/A 0.3969
Lower Mean
Square Error
=
Better
Performance

84. 84
Using same dataset and split
as previous tutorial

85. 85
Baseline Model
Write down MSE on Test set

86. Improving Model Performance (Step-by-Step)
86
Model Settings MSE (CV) MSE (Test)
GBM with default settings N/A 0.4551
GBM with manual settings N/A 0.4433
Manual settings + cross-validation 0.4502 0.4433
Manual + CV + early stopping 0.4429 0.4287
CV + early stopping + full grid search 0.4378 0.4196
CV + early stopping + random grid search 0.4227 0.4047
Stacking models from random grid search N/A 0.3969

87. 87
Manual settings
based on experience

88. Improving Model Performance (Step-by-Step)
88
Model Settings MSE (CV) MSE (Test)
GBM with default settings N/A 0.4551
GBM with manual settings N/A 0.4433
Manual settings + cross-validation 0.4502 0.4433
Manual + CV + early stopping 0.4429 0.4287
CV + early stopping + full grid search 0.4378 0.4196
CV + early stopping + random grid search 0.4227 0.4047
Stacking models from random grid search N/A 0.3969

89. Cross-Validation
89

90. 90
Manual settings
based on experience
+
5-fold CV

91. 91
Average MSE
from 5-fold CV

92. Improving Model Performance (Step-by-Step)
92
Model Settings MSE (CV) MSE (Test)
GBM with default settings N/A 0.4551
GBM with manual settings N/A 0.4433
Manual settings + cross-validation 0.4502 0.4433
Manual + CV + early stopping 0.4429 0.4287
CV + early stopping + full grid search 0.4378 0.4196
CV + early stopping + random grid search 0.4227 0.4047
Stacking models from random grid search N/A 0.3969

93. Early Stopping
93

94. 94
Search for
lowest MSE
from 5-fold CV

95. Improving Model Performance (Step-by-Step)
95
Model Settings MSE (CV) MSE (Test)
GBM with default settings N/A 0.4551
GBM with manual settings N/A 0.4433
Manual settings + cross-validation 0.4502 0.4433
Manual + CV + early stopping 0.4429 0.4287
CV + early stopping + full grid search 0.4378 0.4196
CV + early stopping + random grid search 0.4227 0.4047
Stacking models from random grid search N/A 0.3969

96. Grid Search
96
Combination Parameter 1 Parameter 2
1 0.7 0.7
2 0.7 0.8
3 0.7 0.9
4 0.8 0.7
5 0.8 0.8
6 0.8 0.9
7 0.9 0.7
8 0.9 0.8
9 0.9 0.9

97. 97

98. 98
Sort Results by MSE
Best Model on Top
Lowest MSE

99. 99
Stopped at 187 trees
(automatic)

100. Improving Model Performance (Step-by-Step)
100
Model Settings MSE (CV) MSE (Test)
GBM with default settings N/A 0.4551
GBM with manual settings N/A 0.4433
Manual settings + cross-validation 0.4502 0.4433
Manual + CV + early stopping 0.4429 0.4287
CV + early stopping + full grid search 0.4378 0.4196
CV + early stopping + random grid search 0.4227 0.4047
Stacking models from random grid search N/A 0.3969

101. 101
Expand Search Space
Only search for 9
combinations

102. 102
Sort Results by MSE
Best Model on Top
Lowest MSE

103. Improving Model Performance (Step-by-Step)
103
Model Settings MSE (CV) MSE (Test)
GBM with default settings N/A 0.4551
GBM with manual settings N/A 0.4433
Manual settings + cross-validation 0.4502 0.4433
Manual + CV + early stopping 0.4429 0.4287
CV + early stopping + full grid search 0.4378 0.4196
CV + early stopping + random grid search 0.4227 0.4047
Stacking models from random grid search N/A 0.3969

104. Regression Models (Ensembles)
py_03c_regression_ensembles.ipynb
104

105. 105
https://github.com/h2oai/h2o-
meetups/blob/master/2017_02_23_
Metis_SF_Sacked_Ensembles_Deep_
Water/stacked_ensembles_in_h2o_fe
b2017.pdf

106. 106
Keep the Best Model after
Random Grid Search

107. 107
Keep the Best Model after
Random Grid Search

108. 108
Keep the Best Model after
Random Grid Search

109. 109
Lowest MSE =
Best Performance
API for Stacked Ensembles
Use the three models
from previous steps

110. Improving Model Performance (Step-by-Step)
110
Model Settings MSE (CV) MSE (Test)
GBM with default settings N/A 0.4551
GBM with manual settings N/A 0.4433
Manual settings + cross-validation 0.4502 0.4433
Manual + CV + early stopping 0.4429 0.4287
CV + early stopping + full grid search 0.4378 0.4196
CV + early stopping + random grid search 0.4227 0.4047
Stacking models from random grid search N/A 0.3969
Lowest MSE =
Best Performance

111. Classification Models (Ensembles)
py_04_classification_ensembles.ipynb
111

112. 112
Highest AUC =
Best Performance

113. H2O in the Cloud
py_05_h2o_in_the_cloud.ipynb
113

114. 114

115. 115

116. Recap
116

117. Learning Objectives
• Start and connect to a local H2O cluster from Python.
• Import data from Python data frames, local files or web.
• Perform basic data transformation and exploration.
• Train regression and classification models using various H2O machine
learning algorithms.
• Evaluate models and make predictions.
• Improve performance by tuning and stacking.
• Connect to H2O cluster in the cloud.
117

118. Improving Model Performance (Step-by-Step)
118
Model Settings MSE (CV) MSE (Test)
GBM with default settings N/A 0.4551
GBM with manual settings N/A 0.4433
Manual settings + cross-validation 0.4502 0.4433
Manual + CV + early stopping 0.4429 0.4287
CV + early stopping + full grid search 0.4378 0.4196
CV + early stopping + random grid search 0.4227 0.4047
Stacking models from random grid search N/A 0.3969
Lowest MSE =
Best Performance

119. 119

120. • Our Friends at
• Find us at Poznan R Meetup
• Today at 6:15 pm
• Uniwersytet Ekonomiczny w Poznaniu
Centrum Edukacyjne Usług
Elektronicznych
120
Thanks!
• Code, Slides & Documents
• bit.ly/h2o_meetups
• docs.h2o.ai
• Contact
• joe@h2o.ai
• @matlabulous
• github.com/woobe
• Please search/ask questions on
Stack Overflow
• Use the tag `h2o` (not H2 zero)