Three sentences summarizing the document: Unsupervised learning techniques like clustering can be used to identify hidden patterns in unlabeled data. Hierarchical agglomerative clustering works by iteratively merging the closest pairs of clusters until the stopping criteria is reached. Different linkage methods like single, complete, average and Ward linkage determine how the distance between clusters is calculated during the merging process.

1. Introduction to
Unsupervised Learning

2. Legal Notices and Disclaimers
This presentation is for informational purposes only. INTEL MAKES NO WARRANTIES,
EXPRESS OR IMPLIED, IN THIS SUMMARY.
Intel technologies’ features and benefits depend on system configuration and may require
enabled hardware, software or service activation. Performance varies depending on system
configuration. Check with your system manufacturer or retailer or learn more at intel.com.
This sample source code is released under the Intel Sample Source Code License Agreement.
Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries.
*Other names and brands may be claimed as the property of others.
Copyright © 2017, Intel Corporation. All rights reserved.

3. Types of Machine Learning
data points have known outcomeSupervised
Unsupervised data points have unknown outcome

4. Types of Machine Learning
data points have known outcomeSupervised
Unsupervised data points have unknown outcome

5. Dimensionality
Reduction
Clustering
Types of Unsupervised Learning
identify unknown structure in data
use structural characteristics to simplify data

6. Dimensionality
Reduction
Types of Unsupervised Learning
identify unknown structure in data
use structural characteristics to simplify data
Clustering

7. Unsupervised Learning Overview
unlabeled data
(no answers)
map new
data to
structure
new
unlabeled
data
fit
+
structure
predict
model

8. text articles
of unknown
topics
model
predict similar
articles
text articles of
unknown
topics
fit
+
+
predict
Clustering: Finding Distinct Groups
model
model

9. high
resolution
images
model
compressed
images
high
resolution
images
fit
+
+
model
model
predict
Dimensionality Reduction: Simplifying Structure

10. Introduction to Unsupervised Learning
Users of a web application:
One feature (age)
Age

11. Introduction to Unsupervised Learning
Users of a web application:
One feature (age)
Two clusters
Age

12. Introduction to Unsupervised Learning
Users of a web application:
One feature (age)
Three clusters
Age

13. Introduction to Unsupervised Learning
Users of a web application:
One feature (age)
Five clusters
Age

14. Age
Income
K-Means Algorithm
K = 2 (find two clusters)

15. Age
Income
K-Means Algorithm
K = 2, Randomly assign cluster centers

16. Age
Income
K-Means Algorithm
K = 2, Each point belongs to closest center

17. Age
Income
K-Means Algorithm
K = 2, Move each center to cluster's mean

18. Age
Income
K-Means Algorithm
K = 2, Each point belongs to closest center

19. Age
Income
K-Means Algorithm
K = 2, Move each center to cluster's mean

20. Age
Income
K-Means Algorithm
K = 2, Points don't change  Converged

21. Age
Income
K-Means Algorithm
K = 2, Each point belongs to closest center

22. Age
Income
K-Means Algorithm
K = 3

23. Age
Income
K-Means Algorithm
K = 3, Results depend on initial cluster assignment

24. Age
Income
Which Model is the Right One?

25. Which Model is the Right One?
• Inertia: sum of squared distance from each
point (𝑥𝑖) to its cluster (𝐶 𝑘)
𝑖=1
𝑛
(𝑥𝑖 − 𝐶 𝑘)2
• Smaller value corresponds to tighter clusters
• Other metrics can also be used

26. Age
Income
Which Model is the Right One?
Initiate multiple times,
take model with the best score

27. Age
Income
Which Model is the Right One?
Inertia = 12.645

28. Age
Income
Which Model is the Right One?
Inertia = 12.943

29. Age
Income
Which Model is the Right One?
Inertia = 13.112

30. Age
Income
Smarter Initialization of K-Means Clusters

31. Age
Income
Smarter Initialization of K-Means Clusters
Pick one point at random as initial point

32. Age
Income
Smarter Initialization of K-Means Clusters
Pick next point with 1/distance2 probability

33. Age
Income
Smarter Initialization of K-Means Clusters
Pick next point with 1/distance2 probability

34. Age
Income
Smarter Initialization of K-Means Clusters
Pick next point with 1/distance2 probability

35. Age
Income
Smarter Initialization of K-Means Clusters
Assign clusters

36. Choosing the Right Number of Clusters

37. • Sometimes the question has a K
Choosing the Right Number of Clusters

38. • Sometimes the question has a K
• Clustering similar jobs on 4 CPU cores (K=4)
Choosing the Right Number of Clusters

39. • Sometimes the question has a K
• Clustering similar jobs on 4 CPU cores (K=4)
• A clothing design in 10 different sizes to cover
most people (K=10)
Choosing the Right Number of Clusters

40. • Sometimes the question has a K
• Clustering similar jobs on 4 CPU cores (K=4)
• A clothing design in 10 different sizes to cover
most people (K=10)
• A navigation interface for browsing scientific
papers with 20 disciplines (K=20)
Choosing the Right Number of Clusters

41. Inertia
K
Choosing the Right Number of Clusters
• Inertia measures distance
of point to cluster
• Value decreases with
increasing K as long as
cluster density increases
2 4 6 8 10

42. Inertia
K
Choosing the Right Number of Clusters
• Inertia measures distance
of point to cluster
• Value decreases with
increasing K as long as
cluster density increases
2 4 6 8 10

43. K-Means: The Syntax
Import the class containing the clustering method
from sklearn.cluster import KMeans
Create an instance of the class
kmeans = KMeans(n_clusters=3,
init='k-means++')
Fit the instance on the data and then transform the data
X_trans = kmeans.fit_transform(X_sparse)
Can also be used in batch mode with MiniBatchKMeans.

44. K-Means: The Syntax
Import the class containing the clustering method
from sklearn.cluster import KMeans
Create an instance of the class
kmeans = KMeans(n_clusters=3,
init='k-means++')
Fit the instance on the data and then transform the data
X_trans = kmeans.fit_transform(X_sparse)
Can also be used in batch mode with MiniBatchKMeans.

45. K-Means: The Syntax
Import the class containing the clustering method
from sklearn.cluster import KMeans
Create an instance of the class
kmeans = KMeans(n_clusters=3,
init='k-means++')
Fit the instance on the data and then transform the data
X_trans = kmeans.fit_transform(X_sparse)
Can also be used in batch mode with MiniBatchKMeans.
final number
of clusters

46. K-Means: The Syntax
Import the class containing the clustering method
from sklearn.cluster import KMeans
Create an instance of the class
kmeans = KMeans(n_clusters=3,
init='k-means++')
Fit the instance on the data and then transform the data
X_trans = kmeans.fit_transform(X_sparse)
Can also be used in batch mode with MiniBatchKMeans.
kmeans++
cluster
initiation

47. K-Means: The Syntax
Import the class containing the clustering method
from sklearn.cluster import KMeans
Create an instance of the class
kmeans = KMeans(n_clusters=3,
init='k-means++')
Fit the instance on the data and then predict clusters for new data
kmeans = kmeans.fit(X1)
Can also be used in batch mode with MiniBatchKMeans.

48. K-Means: The Syntax
Import the class containing the clustering method
from sklearn.cluster import KMeans
Create an instance of the class
kmeans = KMeans(n_clusters=3,
init='k-means++')
Fit the instance on the data and then predict clusters for new data
kmeans = kmeans.fit(X1)
y_predict = kmeans.predict(X2)
Can also be used in batch mode with MiniBatchKMeans.

50. Distance Metrics

51. Distance Metric Choice
• Choice of distance metric is extremely
important to clustering success
• Each metric has strengths and most
appropriate use-cases…
• …but sometimes choice of distance
metric is also based on empirical
evaluation

52. Age
Income
Euclidean Distance

53. Age
Income
Euclidean Distance

54. Age
Income
Euclidean Distance (L2 Distance)
∆ Income
d
∆ Age
𝑑 = ∆𝐴𝑔𝑒2 + ∆𝐼𝑛𝑐𝑜𝑚𝑒2

55. Age
Income
Manhattan Distance (L1 or City Block Distance)
∆ Income
∆ Age
𝑑 = ∆𝐴𝑔𝑒 + ∆𝐼𝑛𝑐𝑜𝑚𝑒

56. Age
Income
Cosine Distance
cos(𝜃) =
𝐴 ∙ 𝐵
𝐴 𝐵
𝜃

57. Age
Income
Cosine Distance
cos(𝜃) =
𝐴 ∙ 𝐵
𝐴 𝐵
𝜃

58. Euclidean vs Cosine Distance
• Euclidean is useful for coordinate based
measurements
• Cosine is better for input data such as text
where location of occurrence is less important
• Euclidean distance is more sensitive to curse
of dimensionality (see lesson 12)

59. Euclidean vs Cosine Distance
• Euclidean is useful for coordinate based
measurements
• Cosine is better for data such as text where
location of occurrence is less important
• Euclidean distance is more sensitive to curse
of dimensionality (see lesson 12)

60. Euclidean vs Cosine Distance
• Euclidean is useful for coordinate based
measurements
• Cosine is better for data such as text where
location of occurrence is less important
• Euclidean distance is more sensitive to curse
of dimensionality (see lesson 12)

61. Jaccard Distance
1 −
𝐴 ∩ 𝐵
𝐴 ∪ 𝐵
= 1 −
𝑙𝑒𝑛(𝑠ℎ𝑎𝑟𝑒𝑑)
𝑙𝑒𝑛(𝑢𝑛𝑖𝑞𝑢𝑒)
Applies to sets (like word occurrence)
• Sentence A: “I like chocolate ice cream.”
• set A = {I, like, chocolate, ice, cream}
• Sentence B: “Do I want chocolate cream or vanilla cream?”
• set B = {Do, I, want, chocolate, cream, or, vanilla}

62. Jaccard Distance
1 −
𝐴 ∩ 𝐵
𝐴 ∪ 𝐵
= 1 −
3
9
• Sentence A: “I like chocolate ice cream.”
• set A = {I, like, chocolate, ice, cream}
• Sentence B: “Do I want chocolate cream or vanilla cream?”
• set B = {Do, I, want, chocolate, cream, or, vanilla}
Applies to sets (like word occurrence)

63. Distance Metrics: The Syntax
Import the general pairwise distance function
from sklearn.metrics import pairwise_distances
Calculate the distances
dist = pairwise_distances(X, Y,
metric='euclidean')
Other distance metric choices are: cosine, manhattan, jaccard, etc.
Distance metric methods can also be imported specifically, e.g.:
from sklearn.metrics import euclidean_distances

64. Distance Metrics: The Syntax
Import the general pairwise distance function
from sklearn.metrics import pairwise_distances
Calculate the distances
dist = pairwise_distances(X, Y,
metric='euclidean')
Other distance metric choices are: cosine, manhattan, jaccard, etc.
Distance metric methods can also be imported specifically, e.g.:
from sklearn.metrics import euclidean_distances

65. Distance Metrics: The Syntax
Import the general pairwise distance function
from sklearn.metrics import pairwise_distances
Calculate the distances
dist = pairwise_distances(X, Y,
metric='euclidean')
Other distance metric choices are: cosine, manhattan, jaccard, etc.
Distance metric methods can also be imported specifically, e.g.:
from sklearn.metrics import euclidean_distances
distance
metric choice

66. Distance Metrics: The Syntax
Import the general pairwise distance function
from sklearn.metrics import pairwise_distances
Calculate the distances
dist = pairwise_distances(X, Y,
metric='euclidean')
Other distance metric choices are: cosine, manhattan, jaccard, etc.
Distance metric methods can also be imported specifically, e.g.:
from sklearn.metrics import euclidean_distances

67. Distance Metrics: The Syntax
Import the general pairwise distance function
from sklearn.metrics import pairwise_distances
Calculate the distances
dist = pairwise_distances(X, Y,
metric='euclidean')
Other distance metric choices are: cosine, manhattan, jaccard, etc.
Distance metric methods can also be imported specifically, e.g.:
from sklearn.metrics import euclidean_distances

69. Hierarchical Agglomerative
Clustering

70. Age
Income
Hierarchical Agglomerative Clustering

71. Age
Income
Hierarchical Agglomerative Clustering
Find closest pair, merge into a cluster

72. Age
Income
Hierarchical Agglomerative ClusteringFind next closest pair and merge

73. Age
Income
Hierarchical Agglomerative Clustering
Find next closest pair and merge

74. Age
Income
Hierarchical Agglomerative Clustering
Keep merging closest pairs

75. Age
Income
Hierarchical Agglomerative Clustering
If the closest pair is two clusters, merge them

76. Age
Income
Hierarchical Agglomerative Clustering
Keep merging closest pairs and clusters

77. Age
Income
Hierarchical Agglomerative Clustering
Keep merging closest pairs and clusters

78. Age
Income
Hierarchical Agglomerative Clustering
Current number of clusters = 6

79. Age
Income
Hierarchical Agglomerative Clustering
Current number of clusters = 5

80. Age
Income
Hierarchical Agglomerative Clustering
Current number of clusters = 4

81. Age
Income
Hierarchical Agglomerative Clustering
Current number of clusters = 3

82. Age
Income
Hierarchical Agglomerative Clustering
Current number of clusters = 2

83. Age
Income
Hierarchical Agglomerative Clustering
Current number of clusters = 1

84. Agglomerative Clustering Stopping Conditions
the correct number of clusters is
reached
Condition 1
Condition 2
minimum cluster distance reaches a
set value

85. Agglomerative Clustering Stopping Conditions
the correct number of clusters is
reached
Condition 1
Condition 2
minimum average cluster distance
reaches a set value

86. Age
Income
Hierarchical Agglomerative Clustering
Current number of clusters = 5

87. Hierarchical Agglomerative Clustering
Current number of clusters = 5
Cluster
distance

88. Age
Income
Hierarchical Agglomerative Clustering
Current number of clusters = 4

89. Hierarchical Agglomerative Clustering
Current number of clusters = 4
Cluster
distance

90. Age
Income
Hierarchical Agglomerative Clustering
Current number of clusters = 3

91. Hierarchical Agglomerative Clustering
Current number of clusters = 3
Cluster
distance

92. Age
Income
Hierarchical Agglomerative Clustering
Current number of clusters = 2

93. Hierarchical Agglomerative Clustering
Current number of clusters = 2
Cluster
distance

94. Age
Income
Hierarchical Linkage Types
Single linkage: minimum pairwise distance between clusters

95. Age
Income
Hierarchical Linkage Types
Single linkage: minimum pairwise distance between clusters

96. Age
Income
Hierarchical Linkage Types
Complete linkage: maximum pairwise distance between clusters

97. Age
Income
Hierarchical Linkage Types
Complete linkage: maximum pairwise distance between clusters

98. Age
Income
Hierarchical Linkage Types
Average linkage: average pairwise distance between clusters

99. Age
Income
Hierarchical Linkage Types
Average linkage: average pairwise distance between clusters

100. Age
Income
Hierarchical Linkage Types
Ward linkage: merge based on best inertia

101. Age
Income
Hierarchical Linkage Types
Ward linkage: merge based on best inertia

102. Agglomerative Clustering: The Syntax
Import the class containing the clustering method
from sklearn.cluster import AgglomerativeClustering
Create an instance of the class
agg = AgglomerativeClustering(n_clusters=3,
affinity='euclidean',
linkage='ward')
Fit the instance on the data and then predict clusters for new data
agg = agg.fit(X1)
y_predict = agg.predict(X2)

103. Agglomerative Clustering: The Syntax
Import the class containing the clustering method
from sklearn.cluster import AgglomerativeClustering
Create an instance of the class
agg = AgglomerativeClustering(n_clusters=3,
affinity='euclidean',
linkage='ward')
Fit the instance on the data and then predict clusters for new data
agg = agg.fit(X1)
y_predict = agg.predict(X2)
final number
of clusters

104. Agglomerative Clustering: The Syntax
Import the class containing the clustering method
from sklearn.cluster import AgglomerativeClustering
Create an instance of the class
agg = AgglomerativeClustering(n_clusters=3,
affinity='euclidean',
linkage='ward')
Fit the instance on the data and then predict clusters for new data
agg = agg.fit(X1)
y_predict = agg.predict(X2)
cluster
affinity and
aggregation

105. Mini-
Batch
K-Means
Affinity
Propagatio
n
Mean
Shift
Spectral
Clustering
Ward DBSCAN
.01s 8.17s .02s .31s .21s .10s
0.1s 8.17s .03s .03s .26s .10s
.01s 8.45s .03s .04s .31s .11s
.02s 8.53s .06s .08s .21s .10s
Other Types of Clustering
Reference: http://scikit-learn.org/stable/auto_examples/cluster/plot_cluster_comparison.html