This document provides an overview of machine learning clustering techniques, including agglomerative clustering and k-means clustering. It discusses how agglomerative clustering successively merges the closest pair of clusters until one cluster remains, while k-means clustering initializes cluster centroids randomly and alternates between data point assignment and centroid updates. The document also introduces the bag-of-words model for representing documents or images as histograms of words or visual words.

1. Machine Learning using
Matlab
Lecture 9 Clustering part 1

2. Outline
● Unsupervised learning
● Agglomerative clustering
● K-means clustering
○ Bag-of-Words (BoW) model
● Other clustering algorithms

3. Unsupervised learning
● In unsupervised learning, the training data consists of a set of input vectors
without any corresponding target values
● Goals:
○ Clustering: discover groups of similar examples within the data
○ Density estimation: determine the distribution of data within the input space
○ Data visualization: project the data from a high-dimensional space down to two or three
dimensions
● Unsupervised learning was largely ignored by machine learning community
○ It is hard to say what the aim of unsupervised learning is
● Unsupervised learning is the future of machine learning
○ Large scale of unlabeled data
○ Very expensive and infeasible to annotate large data

4. Clustering - intuition
Unlabeled data After clustering

5. Clustering
● Suppose we have a data set , the objective of clustering is
to partition the data into different groups (clusters), where the data in each
group are similar to each other other than to those in other groups
● Clustering is one of the most commonly researched unsupervised learning
topic
● The only information clustering uses is the similarity between examples
● Clustering groups examples based of their mutual similarities

6. Clustering
● Clustering can help us:
○ Automatically organizing data
○ Understanding hidden structure in some data
● A good cluster algorithm is:
○ High within-cluster similarity
○ Low inter-cluster similarity
● Application:
○ Image segmentation
○ Social network analysis
○ Data compression
○ Gene expression data

7. Clustering
● Hierarchical clustering
○ Agglomerative: bottom-up
○ Divisive: top-down
● Centroid-based clustering
○ K-means clustering
○ Kernel k-means
○ Fuzzy c-means
● Distribution-based clustering
○ Gaussian Mixture Model (GMM)
● Density-based clustering
○ Mean-shift
○ Density-based spatial clustering of applications with noise (DBSCAN)

8. Distance measure
● Distance is inversely related to similarity, namely, large distance, low
similarity; small distance, high similarity
● Choice of the distance measure is very important for clustering
● Some commonly used distance metrics:
○ Euclidean distance (L2 norm):
○ Manhattan distance (L1 norm):
○ Mahalanobis distance: , where S is the covariance matrix
● Similarity is subjective and hard to define
● Different similarity criteria can lead to different clusters

9. The property of a distance measure
● Non-negative:
● Symmetry:
● Self-similarity:
● Triangle inequality:
● E.g., self-similarity, we shouldn’t conclude A looks like B, but B doesn't look
like A

10. Agglomerative clustering
● Initially, each point is treated as a
cluster
● Repeat:
○ Compute distance between all clusters in
terms of the defined distance measure
(store for efficiency)
○ Merge two nearest clusters into a new
cluster
○ Stop when there is only one cluster left
● Save both clustering and sequence of
cluster operations by “Dendrogram”

11. Agglomerative clustering

12. Agglomerative clustering
● Summary
○ Choose a cluster distance/dissimilarity method
○ Successively merge closet pair of clusters until only one cluster left
● Pros and cons
○ Easy to understand and implement
○ Don’t need to define the number of clusters
○ Possible to view partitions at different levels of granularities
○ Computational extensive, complexity is O(m2
logm)

13. Agglomerative clustering
Clustering gene expression data [Eisen et al 1998].

14. K-means clustering
● Input:
○ Number of clusters k
○ Training set
● Steps:
○ Randomly initialize k cluster centroids, i.e., mean/average
○ Repeat until converge
■ Assign each data to its closest centroid
■ Change the cluster center to the average of its assigned points
● Output:
○ Centroids of each cluster
○ The cluster label which is assigned to each data

15. K-means clustering

16. K-means clustering
● Let be the centroids of clusters
● Optimization objective:
● is the indicator to represent if the data i is assigned to cluster j
● Exact optimization of the k-means objective is NP-hard

17. K-means clustering
● The solution for k-means algorithm is heuristic
○ Fix centroids , optimize - assign step
○ Fix , optimize centroids - mean relocation step
● Each step is guaranteed to decrease the objective, thus guaranteed to
converge to a local optima.

18. K-means clustering
Algorithm:
● Randomly initialize k cluster centroids
● Repeat:
○ for i = 1 to m
■ Compute its indicator
○ for j = 1 to k
■ Compute the new cluster centroids

19. Random initialization
● K-means algorithm converges to a local optima, so it is extremely sensitive to
cluster center initialization.
● Bad initialization may lead to:
○ Poor convergence speed
○ Bad overall clustering

20. Handle bad initialization
1. Make initialized centroids evenly distribute in training set:
○ Choose first center as one of the examples, second which is the farthest from the first, third
which is the farthest from both, and so on
2. Try-and-see, namely, try multiple initializations and choose the best result:
○ For i = 1 to 100
■ Run K-means
■ Compute cost function
○ Pick the result that gave the lowest cost

21. Choosing the number of cluster k
● Elbow method: try different values of k, plot the k-means cost function J, and
find out the “elbow-point” in the figure

22. Image segmentation by k-means

23. K-means clustering
● K-means is the most popular and widely used
clustering algorithm
● Converge to a local optima
● Computational complexity in each iteration:
○ Assign data points to closest cluster center O(mk)
○ Change the cluster center to the average of its assigned
points O(m)
● “Hard assignment”, either 1 or 0 to each cluster
● Limitations:
○ Sensitive to outliers
○ Works well only for round shaped, and of roughly equal
sizes/density clusters
○ Can’t work well on non-linear clusters

24. Bag-of-Words model
● Origins from Natural Language Process (NLP):
○ A text (such as a sentence or a document) is represented as the bag (multiset) of its words,
disregarding grammar and even word order but keeping multiplicity.
● Orderless document representation: frequencies of words from a dictionary
● A simple example:
○ Document 1: John likes to watch movies. One of his favorite movie is XXX
○ Document 2: John also likes to play football.
○ Document 3: Tom likes to watch movie as well.
○ Dictionary: {movie, football}
○ [2 0] [0 1] [1 0]

25. Bag-of-Words model - intuition

26. Bag-of-Words model

27. Bag-of-Words model in computer vision
Algorithm:
● Extract scale-invariant features (SIFT [Lowe 2004], HOG [Dalal 2005], .etc)
from images
● Sample a subset of those features, construct a visual dictionary by k-means
(How many clusters k?)
● Quantize features using visual vocabulary
● Represent images by frequencies of “visual words”
The histogram representations of images can be used for classification using
some ML classification models.

28. Bag-of-Words model
● Pros:
○ flexible to geometry / deformations / viewpoint
○ compact summary of image content
○ provides vector representation for images
○ good recognition results in practice
● Cons:
○ basic model ignores geometry – must verify afterwards, or encode via features
○ background and foreground mixed when bag covers whole image
○ interest points or sampling: no guarantee to capture object-level parts
○ optimal vocabulary formation remains unclear

29. Agglomerative vs. k-means
Agglomerative cluster:
● gives different partitionings depending on
the level-of-resolution we are looking at
● doesn’t need the number of clusters to be
specified
● may be slow as it has to compute distance
many times O(m2
logm)
K-means clustering
● produces a single partitioning
● needs the number of clusters to be
specified
● more efficient O(m(k+1)i), where i is the
number of iteration