A practical introduction to Machine Learning with Python and scikit-learn.

1. by Ilya Kuzovkin
ilya.kuzovkin@gmail.com
Mooncascade ML Camp
2016
Machine Learning
ESSENTIAL CONCEPTS

2. ONE MACHINE LEARNING USE CASE

10. Can we ask a computer to
create those patterns
automatically?

11. Can we ask a computer to
create those patterns
automatically?
Yes

12. Can we ask a computer to
create those patterns
automatically?
Yes
How?

13. Raw data

14. Instance
Raw data
Class (label)
A data sample:
“7”

15. Instance
Raw data
Class (label)
A data sample:
“7”
How to represent it in a machine-readable form?

16. Instance
Raw data
Class (label)
A data sample:
“7”
How to represent it in a machine-readable form?
Feature extraction

17. Instance
Raw data
Class (label)
A data sample:
“7”
How to represent it in a machine-readable form?
Feature extraction
28px
28 px

18. Instance
Raw data
Class (label)
A data sample:
“7”
28px
28 px
784 pixels in total
Feature vector
(0, 0, 0, …, 28, 65, 128, 255, 101, 38,… 0, 0, 0)
How to represent it in a machine-readable form?
Feature extraction

19. Instance
Raw data
Class (label)
A data sample:
“7”
28px
28 px
784 pixels in total
Feature vector
(0, 0, 0, …, 28, 65, 128, 255, 101, 38,… 0, 0, 0)
How to represent it in a machine-readable form?
Feature extraction
(0, 0, 0, …, 28, 65, 128, 255, 101, 38,… 0, 0, 0)
(0, 0, 0, …, 13, 48, 102, 0, 46, 255,… 0, 0, 0)
(0, 0, 0, …, 17, 34, 12, 43, 122, 70,… 0, 7, 0)
(0, 0, 0, …, 98, 21, 255, 255, 231, 140,… 0, 0, 0)
“7”
“2”
“8”
“2”

20. Instance
Raw data
Class (label)
A data sample:
“7”
28px
28 px
784 pixels in total
Feature vector
(0, 0, 0, …, 28, 65, 128, 255, 101, 38,… 0, 0, 0)
How to represent it in a machine-readable form?
Feature extraction
(0, 0, 0, …, 28, 65, 128, 255, 101, 38,… 0, 0, 0)
(0, 0, 0, …, 13, 48, 102, 0, 46, 255,… 0, 0, 0)
(0, 0, 0, …, 17, 34, 12, 43, 122, 70,… 0, 7, 0)
Dataset
(0, 0, 0, …, 98, 21, 255, 255, 231, 140,… 0, 0, 0)
“7”
“2”
“8”
“2”

21. The data is in the right format — what’s next?

22. The data is in the right format — what’s next?
• C4.5
• Random forests
• Bayesian networks
• Hidden Markov models
• Artificial neural network
• Data clustering
• Expectation-maximization
algorithm
• Self-organizing map
• Radial basis function network
• Vector Quantization
• Generative topographic map
• Information bottleneck method
• IBSEAD
• Apriori algorithm
• Eclat algorithm
• FP-growth algorithm
• Single-linkage clustering
• Conceptual clustering
• K-means algorithm
• Fuzzy clustering
• Temporal difference learning
• Q-learning
• Learning Automata
• AODE
• Artificial neural network
• Backpropagation
• Naive Bayes classifier
• Bayesian network
• Bayesian knowledge base
• Case-based reasoning
• Decision trees
• Inductive logic
programming
• Gaussian process regression
• Gene expression
programming
• Group method of data
handling (GMDH)
• Learning Automata
• Learning Vector
Quantization
• Logistic Model Tree
• Decision tree
• Decision graphs
• Lazy learning
• Monte Carlo Method
• SARSA
• Instance-based learning
• Nearest Neighbor Algorithm
• Analogical modeling
• Probably approximately correct learning
(PACL)
• Symbolic machine learning algorithms
• Subsymbolic machine learning algorithms
• Support vector machines
• Random Forest
• Ensembles of classifiers
• Bootstrap aggregating (bagging)
• Boosting (meta-algorithm)
• Ordinal classification
• Regression analysis
• Information fuzzy networks (IFN)
• Linear classifiers
• Fisher's linear discriminant
• Logistic regression
• Naive Bayes classifier
• Perceptron
• Support vector machines
• Quadratic classifiers
• k-nearest neighbor
• Boosting
Pick an algorithm

23. The data is in the right format — what’s next?
• C4.5
• Random forests
• Bayesian networks
• Hidden Markov models
• Artificial neural network
• Data clustering
• Expectation-maximization
algorithm
• Self-organizing map
• Radial basis function network
• Vector Quantization
• Generative topographic map
• Information bottleneck method
• IBSEAD
• Apriori algorithm
• Eclat algorithm
• FP-growth algorithm
• Single-linkage clustering
• Conceptual clustering
• K-means algorithm
• Fuzzy clustering
• Temporal difference learning
• Q-learning
• Learning Automata
• AODE
• Artificial neural network
• Backpropagation
• Naive Bayes classifier
• Bayesian network
• Bayesian knowledge base
• Case-based reasoning
• Decision trees
• Inductive logic
programming
• Gaussian process regression
• Gene expression
programming
• Group method of data
handling (GMDH)
• Learning Automata
• Learning Vector
Quantization
• Logistic Model Tree
• Decision tree
• Decision graphs
• Lazy learning
• Monte Carlo Method
• SARSA
• Instance-based learning
• Nearest Neighbor Algorithm
• Analogical modeling
• Probably approximately correct learning
(PACL)
• Symbolic machine learning algorithms
• Subsymbolic machine learning algorithms
• Support vector machines
• Random Forest
• Ensembles of classifiers
• Bootstrap aggregating (bagging)
• Boosting (meta-algorithm)
• Ordinal classification
• Regression analysis
• Information fuzzy networks (IFN)
• Linear classifiers
• Fisher's linear discriminant
• Logistic regression
• Naive Bayes classifier
• Perceptron
• Support vector machines
• Quadratic classifiers
• k-nearest neighbor
• Boosting
Pick an algorithm

24. DECISION TREE
vs.

25. DECISION TREE
vs.
(0, …, 28, 65, …, 207, 101, 0, 0)
(0, …, 19, 34, …, 254, 54, 0, 0)
(0, …, 87, 59, …, 240, 52, 4, 0)
(0, …, 87, 52, …, 240, 19, 3, 0)
(0, …, 28, 64, …, 102, 101, 0, 0)
(0, …, 19, 23, …, 105, 54, 0, 0)
(0, …, 87, 74, …, 121, 51, 7, 0)
(0, …, 87, 112, …, 239, 52, 4, 0)

26. DECISION TREE
vs.
(0, …, 28, 65, …, 207, 101, 0, 0)
(0, …, 19, 34, …, 254, 54, 0, 0)
(0, …, 87, 59, …, 240, 52, 4, 0)
(0, …, 87, 52, …, 240, 19, 3, 0)
(0, …, 28, 64, …, 102, 101, 0, 0)
(0, …, 19, 23, …, 105, 54, 0, 0)
(0, …, 87, 74, …, 121, 51, 7, 0)
(0, …, 87, 112, …, 239, 52, 4, 0)
PIXEL
#417

27. DECISION TREE
vs.
(0, …, 28, 65, …, 207, 101, 0, 0)
(0, …, 19, 34, …, 254, 54, 0, 0)
(0, …, 87, 59, …, 240, 52, 4, 0)
(0, …, 87, 52, …, 240, 19, 3, 0)
(0, …, 28, 64, …, 102, 101, 0, 0)
(0, …, 19, 23, …, 105, 54, 0, 0)
(0, …, 87, 74, …, 121, 51, 7, 0)
(0, …, 87, 112, …, 239, 52, 4, 0)
PIXEL
#417
PIXEL
#417
>200 <200

28. DECISION TREE
vs.
(0, …, 28, 65, …, 207, 101, 0, 0)
(0, …, 19, 34, …, 254, 54, 0, 0)
(0, …, 87, 59, …, 240, 52, 4, 0)
(0, …, 87, 52, …, 240, 19, 3, 0)
(0, …, 28, 64, …, 102, 101, 0, 0)
(0, …, 19, 23, …, 105, 54, 0, 0)
(0, …, 87, 74, …, 121, 51, 7, 0)
(0, …, 87, 112, …, 239, 52, 4, 0)
PIXEL
#417
PIXEL
#417
>200 <200

29. DECISION TREE
vs.
(0, …, 28, 65, …, 207, 101, 0, 0)
(0, …, 19, 34, …, 254, 54, 0, 0)
(0, …, 87, 59, …, 240, 52, 4, 0)
(0, …, 87, 52, …, 240, 19, 3, 0)
(0, …, 28, 64, …, 102, 101, 0, 0)
(0, …, 19, 23, …, 105, 54, 0, 0)
(0, …, 87, 74, …, 121, 51, 7, 0)
(0, …, 87, 112, …, 239, 52, 4, 0)
PIXEL
#417
>200 <200

30. DECISION TREE
vs.
(0, …, 28, 65, …, 207, 101, 0, 0)
(0, …, 19, 34, …, 254, 54, 0, 0)
(0, …, 87, 59, …, 240, 52, 4, 0)
(0, …, 87, 52, …, 240, 19, 3, 0)
(0, …, 28, 64, …, 102, 101, 0, 0)
(0, …, 19, 23, …, 105, 54, 0, 0)
(0, …, 87, 74, …, 121, 51, 7, 0)
(0, …, 87, 112, …, 239, 52, 4, 0)
PIXEL
#417
>200 <200
PIXEL
#123

31. DECISION TREE
vs.
(0, …, 28, 65, …, 207, 101, 0, 0)
(0, …, 19, 34, …, 254, 54, 0, 0)
(0, …, 87, 59, …, 240, 52, 4, 0)
(0, …, 87, 52, …, 240, 19, 3, 0)
(0, …, 28, 64, …, 102, 101, 0, 0)
(0, …, 19, 23, …, 105, 54, 0, 0)
(0, …, 87, 74, …, 121, 51, 7, 0)
(0, …, 87, 112, …, 239, 52, 4, 0)
PIXEL
#417
>200 <200
PIXEL
#123
<100 >100
PIXEL
#123

32. DECISION TREE
vs.
(0, …, 28, 65, …, 207, 101, 0, 0)
(0, …, 19, 34, …, 254, 54, 0, 0)
(0, …, 87, 59, …, 240, 52, 4, 0)
(0, …, 87, 52, …, 240, 19, 3, 0)
(0, …, 28, 64, …, 102, 101, 0, 0)
(0, …, 19, 23, …, 105, 54, 0, 0)
(0, …, 87, 74, …, 121, 51, 7, 0)
(0, …, 87, 112, …, 239, 52, 4, 0)
PIXEL
#417
>200 <200
<100 >100
PIXEL
#123

33. DECISION TREE

34. DECISION TREE

35. ACCURACY

36. ACCURACY
Confusion matrix
Trueclass
Predicted class

37. ACCURACY
Confusion matrix
acc =
correctly classified
total number of samples
Trueclass
Predicted class

38. ACCURACY
Confusion matrix
acc =
correctly classified
total number of samples
Beware of an
imbalanced dataset!
Trueclass
Predicted class

39. ACCURACY
Confusion matrix
acc =
correctly classified
total number of samples
Beware of an
imbalanced dataset!
Consider the following model:
“Always predict 2”
Trueclass
Predicted class

40. ACCURACY
Confusion matrix
acc =
correctly classified
total number of samples
Beware of an
imbalanced dataset!
Consider the following model:
“Always predict 2”
Accuracy 0.9
Trueclass
Predicted class

41. DECISION TREE

42. DECISION TREE
“You said 100%
accurate?! Every 10th
digit your system
detects is wrong!”
Angry client

43. DECISION TREE
“You said 100%
accurate?! Every 10th
digit your system
detects is wrong!”
Angry client
We’ve trained our system on the data the client gave us. But our
system has never seen the new data the client applied it to.
And in the real life — it never will…

44. OVERFITTING
Simulate the real-life situation — split the dataset

45. OVERFITTING
Simulate the real-life situation — split the dataset

46. OVERFITTING
Simulate the real-life situation — split the dataset

47. OVERFITTING
Simulate the real-life situation — split the dataset

48. Underfitting!
“Too stupid”
OK
Overfitting!
“Too smart”
OVERFITTING

49. Underfitting!
“Too stupid”
OK
Overfitting!
“Too smart”
OVERFITTING
Our current decision tree has too much capacity,
it just has memorized all of the data.
Let’s make it less complex.

53. You probably did not notice, but we are overfitting again :(

54. TEST SET
20%
TRAINING SET
60%
THE WHOLE DATASET
VALIDATION SET
20%

55. TEST SET
20%
TRAINING SET
60%
THE WHOLE DATASET
VALIDATION SET
20%
Fit various models
and parameter
combinations on this
subset

56. TEST SET
20%
TRAINING SET
60%
THE WHOLE DATASET
VALIDATION SET
20%
Fit various models
and parameter
combinations on this
subset
• Evaluate the
models created
with different
parameters

57. TEST SET
20%
TRAINING SET
60%
THE WHOLE DATASET
VALIDATION SET
20%
Fit various models
and parameter
combinations on this
subset
• Evaluate the
models created
with different
parameters
!
• Estimate overfitting
TRA
VALI

58. TEST SET
20%
TRAINING SET
60%
THE WHOLE DATASET
VALIDATION SET
20%
Fit various models
and parameter
combinations on this
subset
• Evaluate the
models created
with different
parameters
!
• Estimate overfitting
TRA
VALI
TRA
VALI

59. TEST SET
20%
TRAINING SET
60%
THE WHOLE DATASET
VALIDATION SET
20%
Fit various models
and parameter
combinations on this
subset
• Evaluate the
models created
with different
parameters
!
• Estimate overfitting
TRA
VALI
TRA
VALI
TRA
VALI

60. TEST SET
20%
TRAINING SET
60%
THE WHOLE DATASET
VALIDATION SET
20%
Fit various models
and parameter
combinations on this
subset
• Evaluate the
models created
with different
parameters
!
• Estimate overfitting
TRA
VALI
TRA
VALI
TRA
VALI
TRA
VALI

61. TEST SET
20%
TRAINING SET
60%
THE WHOLE DATASET
VALIDATION SET
20%
Fit various models
and parameter
combinations on this
subset
• Evaluate the
models created
with different
parameters
!
• Estimate overfitting
TRA
VALI
TRA
VALI
TRA
VALI
TRA
VALI
TRA
VALI

62. TEST SET
20%
TRAINING SET
60%
THE WHOLE DATASET
VALIDATION SET
20%
Fit various models
and parameter
combinations on this
subset
• Evaluate the
models created
with different
parameters
!
• Estimate overfitting
Use only once to get
the final performance
estimate
TRA
VALI
TRA
VALI
TRA
VALI
TRA
VALI
TRA
VALI

63. TEST SET
20%
TRAINING SET
60%
VALIDATION SET
20%

64. TEST SET
20%
TRAINING SET
60%
VALIDATION SET
20%

65. CROSS-VALIDATION
TRAINING SET 60%
THE WHOLE DATASET
VALIDATION SET 20%

66. CROSS-VALIDATION
TRAINING SET 60%
THE WHOLE DATASET
VALIDATION SET 20%
What if we got too
optimistic validation set?

67. CROSS-VALIDATION
TRAINING SET 60%
THE WHOLE DATASET
VALIDATION SET 20%
What if we got too
optimistic validation set?
TRAINING SET 80%

68. CROSS-VALIDATION
TRAINING SET 60%
THE WHOLE DATASET
VALIDATION SET 20%
What if we got too
optimistic validation set?
TRAINING SET 80%
Fix the parameter value you ned to evaluate, say msl=15

69. CROSS-VALIDATION
TRAINING SET 60%
THE WHOLE DATASET
VALIDATION SET 20%
What if we got too
optimistic validation set?
TRAINING SET 80%
Fix the parameter value you ned to evaluate, say msl=15
TRAINING VAL
TRAINING VAL
TRAININGVAL
Repeat 10 times

70. CROSS-VALIDATION
TRAINING SET 60%
THE WHOLE DATASET
VALIDATION SET 20%
What if we got too
optimistic validation set?
TRAINING SET 80%
Fix the parameter value you ned to evaluate, say msl=15
TRAINING VAL
TRAINING VAL
TRAININGVAL
Repeat 10 times
}
Take average
validation score
over 10 runs —
it is a more
stable estimate.

74. MACHINE LEARNING PIPELINE
Take raw data Extract features
Split into TRAINING
and TEST
Pick an algorithm
and parameters
Train on the
TRAINING data
Evaluate on the
TRAINING data
with CV
Train on the
whole TRAINING
Fix the best
parameters
Evaluate on TEST
Report final
performance to
the client
Try our different algorithms
and parameters

75. MACHINE LEARNING PIPELINE
Take raw data Extract features
Split into TRAINING
and TEST
Pick an algorithm
and parameters
Train on the
TRAINING data
Evaluate on the
TRAINING data
with CV
Train on the
whole TRAINING
Fix the best
parameters
Evaluate on TEST
Report final
performance to
the client
Try our different algorithms
and parameters
“So it is ~87%…erm…
Could you do better?”

76. MACHINE LEARNING PIPELINE
Take raw data Extract features
Split into TRAINING
and TEST
Pick an algorithm
and parameters
Train on the
TRAINING data
Evaluate on the
TRAINING data
with CV
Train on the
whole TRAINING
Fix the best
parameters
Evaluate on TEST
Report final
performance to
the client
Try our different algorithms
and parameters
“So it is ~87%…erm…
Could you do better?”
Yes

77. • C4.5
• Random forests
• Bayesian networks
• Hidden Markov models
• Artificial neural network
• Data clustering
• Expectation-maximization
algorithm
• Self-organizing map
• Radial basis function network
• Vector Quantization
• Generative topographic map
• Information bottleneck method
• IBSEAD
• Apriori algorithm
• Eclat algorithm
• FP-growth algorithm
• Single-linkage clustering
• Conceptual clustering
• K-means algorithm
• Fuzzy clustering
• Temporal difference learning
• Q-learning
• Learning Automata
• AODE
• Artificial neural network
• Backpropagation
• Naive Bayes classifier
• Bayesian network
• Bayesian knowledge base
• Case-based reasoning
• Decision trees
• Inductive logic
programming
• Gaussian process regression
• Gene expression
programming
• Group method of data
handling (GMDH)
• Learning Automata
• Learning Vector
Quantization
• Logistic Model Tree
• Decision tree
• Decision graphs
• Lazy learning
• Monte Carlo Method
• SARSA
• Instance-based learning
• Nearest Neighbor Algorithm
• Analogical modeling
• Probably approximately correct learning
(PACL)
• Symbolic machine learning algorithms
• Subsymbolic machine learning algorithms
• Support vector machines
• Random Forest
• Ensembles of classifiers
• Bootstrap aggregating (bagging)
• Boosting (meta-algorithm)
• Ordinal classification
• Regression analysis
• Information fuzzy networks (IFN)
• Linear classifiers
• Fisher's linear discriminant
• Logistic regression
• Naive Bayes classifier
• Perceptron
• Support vector machines
• Quadratic classifiers
• k-nearest neighbor
• Boosting
Pick another algorithm

78. • C4.5
• Random forests
• Bayesian networks
• Hidden Markov models
• Artificial neural network
• Data clustering
• Expectation-maximization
algorithm
• Self-organizing map
• Radial basis function network
• Vector Quantization
• Generative topographic map
• Information bottleneck method
• IBSEAD
• Apriori algorithm
• Eclat algorithm
• FP-growth algorithm
• Single-linkage clustering
• Conceptual clustering
• K-means algorithm
• Fuzzy clustering
• Temporal difference learning
• Q-learning
• Learning Automata
• AODE
• Artificial neural network
• Backpropagation
• Naive Bayes classifier
• Bayesian network
• Bayesian knowledge base
• Case-based reasoning
• Decision trees
• Inductive logic
programming
• Gaussian process regression
• Gene expression
programming
• Group method of data
handling (GMDH)
• Learning Automata
• Learning Vector
Quantization
• Logistic Model Tree
• Decision tree
• Decision graphs
• Lazy learning
• Monte Carlo Method
• SARSA
• Instance-based learning
• Nearest Neighbor Algorithm
• Analogical modeling
• Probably approximately correct learning
(PACL)
• Symbolic machine learning algorithms
• Subsymbolic machine learning algorithms
• Support vector machines
• Random Forest
• Ensembles of classifiers
• Bootstrap aggregating (bagging)
• Boosting (meta-algorithm)
• Ordinal classification
• Regression analysis
• Information fuzzy networks (IFN)
• Linear classifiers
• Fisher's linear discriminant
• Logistic regression
• Naive Bayes classifier
• Perceptron
• Support vector machines
• Quadratic classifiers
• k-nearest neighbor
• Boosting
Pick another algorithm

79. RANDOM FOREST

80. RANDOM FOREST
Decision tree:
pick best out of all features

81. RANDOM FOREST
Decision tree:
pick best out of all features
Random forest:
pick best out of random
subset of features

82. RANDOM FOREST

83. RANDOM FOREST
pick best out of another
random subset of features

84. RANDOM FOREST
pick best out of another
random subset of features pick best out of yet another
random subset of features

85. RANDOM FOREST

86. RANDOM FOREST

87. RANDOM FOREST
class
instance

88. RANDOM FOREST
class
instance

89. RANDOM FOREST
class
instance

90. RANDOM FOREST
class
instance

93. Happy client

94. ALL OTHER USE CASES

95. Sound
Frequency
components
Genre
Bag of
words
Topic
Text
Pixel
values
Image
Cat or
dog
Video
Frame
pixels
Walking
or running
Database records Biometric data
Census
data
Average
salary
…
Dead or
alive

101. HANDS-ON SESSION

102. http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html