Cluster Analysis

Cluster Analysis
7/2/2019 Compiled by : Kamal Acharya 1

Cluster Analysis(Clustering/automatic classification/ data segmentation)
• Clustering is the process of grouping a set of data objects into multiple
groups or clusters so that objects within a cluster have high similarity,
but are very dissimilar to objects in other clusters.
• Dissimilarities and similarities are assessed based on the attribute
values describing the objects and often involve distance measures.

Contd..
• Clustering is known as unsupervised leaning because the class
label information is not present. For this reason, clustering is a
form of learning by observation, rather than learning by
examples.
• Different clustering methods may generate different clusterings
on the same data set.
• The partitioning is not performed by humans, but by the
clustering algorithm.

Contd..
• Hence, Clustering is used:
– As a stand-alone tool to get insight into data distribution
• Visualization of clusters may unveil important information
– As a preprocessing step for other algorithms
• Efficient indexing or compression often relies on clustering

Some Applications of Clustering
• Cluster analysis has been widely used in numerous applications
such as:
– In business intelligence
– In image recognization
– In web search
– In Outlier detection
– In biology

Contd..
• In Business intelligence:
– clustering can help marketers discover distinct groups in their
customer bases and characterize customer groups based on
purchasing patterns so that, for example, advertising can be
appropriately targeted..

Contd..
• In image recognization:
– In image recognition, clustering can be used to discover clusters or
“subclasses” in handwritten character recognition systems.
– For example: We can use clustering to determine subclasses for “1,” each
of which represents a variation on the way in which 1 can be written.

Contd..
• In web search
– document grouping: Clustering can be used to organize the
search results into groups and present the results in a concise
and easily accessible way.
– cluster Weblog data to discover groups of similar access
patterns.

Contd..
• In Outlier detection
– Clustering can also be used for outlier detection, where
outliers (values that are “far away” from any cluster) may be
more interesting than common cases.
– Applications of outlier detection include the detection of
credit card fraud and the monitoring of criminal activities in
electronic commerce.

Contd..
• In biology:
– In biology, it can be used to derive plant and animal
taxonomies, categorize genes with similar functionality, and
gain insight into structures inherent in populations.

What Is Good Clustering?
• A good clustering method will produce high quality clusters with
– high intra-class similarity
– low inter-class similarity
• The quality of a clustering result depends on both the similarity
measure used by the method and its implementation.
• The quality of a clustering method is also measured by its ability
to discover some or all of the hidden patterns.

Requirements for clustering as a data mining tool
• The following are typical requirements of clustering in data
mining.
– Scalability
– Ability to deal with different types of attributes
– Discovery of clusters with arbitrary shape
– Requirements for domain knowledge to determine input parameters
– Ability to deal with noisy data
– Incremental clustering and insensitivity to input order
– Capability of clustering high-dimensionality data
– Constraint-based clustering
– Interpretability and usability

Contd..
• Scalability:
– Many clustering algorithms work well on small data sets
containing fewer than several hundred data objects; however,
a large database may contain millions of objects.
– Clustering on a sample of a given large data set may lead to
biased results.
– Highly scalable clustering algorithms are needed.

Contd..
• Ability to deal with different types of attributes:
– Many algorithms are designed to cluster interval-based
(numerical) data.
– However, applications may require clustering other types of
data, such as binary, categorical (nominal), and ordinal data,
or mixtures of these data types.

Contd..
• Discovery of clusters with arbitrary shape:
– Many clustering algorithms determine clusters based on
Euclidean distance measures.
– Algorithms based on such distance measures tend to find
spherical clusters with similar size and density.
– However, a cluster could be of any shape.
– It is important to develop algorithms that can detect clusters
of arbitrary shape.

Contd..
• Minimal requirements for domain knowledge to
determine input parameters:
– Many clustering algorithms require users to input certain
parameters in cluster analysis (such as the number of desired
clusters).
– The clustering results can be quite sensitive to input parameters.
– Parameters are often difficult to determine, especially for data sets
containing high-dimensional objects.
– This not only burdens users, but it also makes the quality of
clustering difficult to control.

Contd..
• Ability to deal with noisy data:
– Most real-world databases contain outliers or missing,
unknown, or erroneous data.
– Some clustering algorithms are sensitive to such data and may
lead to clusters of poor quality.

Contd..
• Incremental clustering and insensitivity to the order of input
records:
– Some clustering algorithms cannot incorporate newly inserted
data (i.e., database updates) into existing clustering structures
and, instead, must determine a new clustering from scratch.
– Some clustering algorithms are sensitive to the order of input
data. That is, given a set of data objects, such an algorithm
may return dramatically different clusterings depending on
the order of presentation of the input objects.
– It is important to develop incremental clustering algorithms
and algorithms that are insensitive to the order of input.

Contd..
• High dimensionality:
– A database or a data warehouse can contain several
dimensions or attributes.
– Many clustering algorithms are good at handling low-
dimensional data, involving only two to three dimensions.
– Human eyes are good at judging the quality of clustering for
up to three dimensions.
– Finding clusters of data objects in high dimensional space is
challenging, especially considering that such data can be
sparse and highly skewed.

Contd..
• Constraint-based clustering:
– Real-world applications may need to perform clustering under
various kinds of constraints.
– Suppose that your job is to choose the locations for a given
number of new automatic banking machines (ATMs) in a city.
– To decide upon this, you may cluster households while
considering constraints such as the city’s rivers and highway
networks, and the type and number of customers per cluster.
– A challenging task is to find groups of data with good
clustering behavior that satisfy specified constraints.

Contd..
• Interpretability and usability:
– Users expect clustering results to be interpretable,
comprehensible, and usable.
– That is, clustering may need to be tied to specific semantic
interpretations and applications.
– It is important to study how an application goal may
influence the selection of clustering features and methods.

Aspects of clustering
• A clustering algorithm/methods
– Partitional clustering
– Hierarchical clustering
– …
• A distance (similarity, or dissimilarity) function
• Clustering quality
– Inter-clusters distance  maximized
– Intra-clusters distance  minimized
• The quality of a clustering result depends on the
algorithm, the distance function, and the application.

Major Clustering Methods:
• In general, the major fundamental clustering methods can be
classified into the following categories:
– Partitioning Methods
– Hierarchical Methods
– Density-Based Methods
– Grid-Based Methods

Contd..
• Partitioning Methods:
– A partitioning method constructs k partitions of the data, where each
partition represents a cluster and k <= n. That is, it classifies the data
into k groups, which together satisfy the following requirements:
• Each group must contain at least one object, and
• Each object must belong to exactly one group.
– A partitioning method creates an initial partitioning. It then uses an
iterative relocation technique that attempts to improve the
partitioning by moving objects from one group to another.
– The general criterion of a good partitioning is that objects in the
same cluster are close or related to each other, whereas objects of
different clusters are far apart or very different.

Contd..
• E.g., K-means, and K-medoids

Contd..
• Hierarchical Methods:
– A hierarchical method creates a hierarchical decomposition of
the given set of data objects.
– A hierarchical method can be classified as being either
agglomerative or divisive, based on how the hierarchical
decomposition is formed.

Contd..
• E.g.,

Contd..
• The agglomerative approach, also called the bottom-up approach, starts
with each object forming a separate group. It successively merges the
objects or groups that are close to one another, until all of the groups
are merged into one or until a termination condition holds.
• The divisive approach, also called the top-down approach, starts with
all of the objects in the same cluster. In each successive iteration, a
cluster is split up into smaller clusters, until eventually each object is in
one cluster, or until a termination condition holds.

Contd..
• Density-based methods:
– General idea is to continue growing the given cluster as long
as the density in the neighborhood exceeds some threshold;
that is, for each data point within a given cluster, the
neighborhood of a given radius has to contain at least a
minimum number of points.
– Such a method can be used to filter out noise (outliers)and
discover clusters of arbitrary shape.
– E.g., DBSCAN

Contd..
• E.g.,

Contd..
• Grid-based methods:
– Grid-based methods quantize the object space into a finite
number of cells that form a grid structure.
– All the clustering operations are performed on the grid
structure.
– E.g., STING

Contd..

Partitioning Methods
• Given a data set, D, of n objects, and k, the number of clusters to
form, a partitioning algorithm organizes the objects into k
partitions (k<=n), where each partition represents a cluster.

k-Means: A Centroid-Based Technique
• A centroid-based partitioning technique uses the centroid of a cluster,
Ci , to represent that cluster.
• The centroid of a cluster is its center point such as the mean or medoid
of the objects (or points) assigned to the cluster.
• The difference between an object and ci, the representative of
the cluster, is measured by dist(p, ci),
• where dist(i, j) is the Euclidean distance between two points

Contd..
• The k-means algorithm defines the centroid of a cluster as the
mean value of the points within the cluster. It proceeds as
follows:
– First, it randomly selects k of the objects in D, each of which initially
represents a cluster mean or center.
– For each of the remaining objects, an object is assigned to the cluster to
which it is the most similar, based on the Euclidean distance between the
object and the cluster mean.
– The k-means algorithm then iteratively improves the within-cluster
variation. For each cluster, it computes the new mean using the objects
assigned to the cluster in the previous iteration. All the objects are then
reassigned using the updated means as the new cluster centers.
– The iterations continue until the assignment is stable, that is, the clusters
formed in the current round are the same as those formed in the previous
round.

Contd..
• Algorithm:
– The k-means algorithm for partitioning, where each cluster’s center is
represented by the mean value of the objects in the cluster.

The K-Means Clustering Method
• Example
0
1
2
3
4
5
6
7
8
9
10
0 1 2 3 4 5 6 7 8 9 10
0
1
2
3
4
5
6
7
8
9
10
0 1 2 3 4 5 6 7 8 9 10
0
1
2
3
4
5
6
7
8
9
10
0 1 2 3 4 5 6 7 8 9 10
0
1
2
3
4
5
6
7
8
9
10
0 1 2 3 4 5 6 7 8 9 10
0
1
2
3
4
5
6
7
8
9
10
0 1 2 3 4 5 6 7 8 9 10
K=2
Arbitrarily choose K
object as initial cluster
center
Assign
each
objects
to most
similar
center
Update
the
cluster
means
Update
the
cluster
means
reassignreassign

Contd..
• Example1: Clusters the following instances of given data (2-
Dimensional form) with the help of K means algorithm (Take K
= 2)
Instance X Y
1 1 1.5
2 1 4.5
3 2 1.5
4 2 3.5
5 3 2.5
6 3 4

Contd..
• Example 2: Clusters the following instances of given data (2-
Dimensional form) with the help of K means algorithm (Take K
= 2)
Instance X Y
1 1 2.5
2 1 4.5
3 2.5 3
4 2 1.5
5 4.5 1.5
6 4 5

Hierarchical clustering
• A hierarchical clustering method works by grouping data objects
into a hierarchy or “tree” of clusters.
• Representing data objects in the form of a hierarchy is useful for
data summarization and visualization.

Contd..
• Depending on whether the hierarchical decomposition is formed
in a bottom-up (merging) or top-down (splitting) fashion a
hierarchical clustering method can be classified into two
categories:
– Agglomerative Hierarchical Clustering and
– Divisive Hierarchical Clustering

Contd..
• Agglomerative Hierarchical Clustering:
– uses a bottom-up strategy.
– starts by letting each object form its own cluster and
iteratively merges clusters into larger and larger clusters, until
all the objects are in a single cluster or certain termination
conditions(desired number of clusters) are satisfied.
– For the merging step, it finds the two clusters that are closest
to each other (according to some similarity measure), and
combines the two to form one cluster.

Contd..
• Example: a data set of five objects, {a, b, c, d, e}. Initially, AGNES
(AGglomerative NESting), the agglomerative method, places each object into
a cluster of its own. The clusters are then merged step-by-step according to
some criterion (e.g., minimum Euclidean distance).

Contd..
• Divisive hierarchical clustering :
– A divisive hierarchical clustering method employs a top-down
strategy.
– It starts by placing all objects in one cluster, which is the
hierarchy’s root.
– It then divides the root cluster into several smaller sub-clusters, and
recursively partitions those clusters into smaller ones.
– The partitioning process continues until each cluster at the lowest
level either containing only one object, or the objects within a
cluster are sufficiently similar to each other.

Contd..
• Example: DIANA (DIvisive ANAlysis), a divisive hierarchical clustering
method:
– a data set of five objects, {a, b, c, d, e}. All the objects are used to form
one initial cluster. The cluster is split according to some principle such as
the maximum Euclidean distance between the closest neighboring objects
in the cluster. The cluster-splitting process repeats until, eventually, each
new cluster contains only a single object.

Contd..
• agglomerative versus divisive hierarchical clustering:
– Organize objects into a hierarchy using a bottom-up or top-
down strategy, respectively.
– Agglomerative methods start with individual objects as
clusters, which are iteratively merged to form larger clusters.
– Conversely, divisive methods initially let all the given objects
form one cluster, which they iteratively split into smaller
clusters.

Contd..
• Hierarchical clustering methods can encounter difficulties regarding
the selection of merge or split points.
– Such a decision is critical, because once a group of objects is
merged or split, the process at the next step will operate on the
newly generated clusters. It will neither undo what was done
previously, nor perform object swapping between clusters.
– Thus, merge or split decisions, if not well chosen, may lead to low-
quality clusters.
• Moreover, the methods do not scale well because each decision of
merge or split needs to examine and evaluate many objects or clusters.

Density Based Methods
• Partitioning methods and hierarchical clustering are suitable for finding
spherical-shaped clusters.
• Moreover, they are also severely affected by the presence of noise and
outliers in the data.
• Unfortunately, real life data contain:
– Clusters of arbitrary shape such as oval, linear, s-shaped, etc.
– Many noise
• Solution : Density based methods

Contd..
• Basic Idea behind Density based methods:
– Model clusters as dense regions in the data space, separated by sparse
regions.
• Major features:
– Discover clusters of arbitrary shape(e.g., oval, s-shaped, etc)
– Handle noise
– Need density parameters as termination condition
• E.g., : DBSCAN(Density Based Spatial Clustering of Applications with Noise)

Density-Based Clustering: Background
• Neighborhood of point p=all points within distance e from p:
– NEps(p)={q | dist(p,q) <= e }
• Two parameters:
– e : Maximum radius of the neighbourhood
– MinPts: Minimum number of points in an e -neighbourhood of that point
• If the number of points in the e -neighborhood of p is at least
MinPts, then p is called a core object.
p
q
MinPts = 5
e = 1 cm

Contd..
• Directly density-reachable: A point p is directly density-
reachable from a point q wrt. e, MinPts if
– 1) p belongs to NEps(q)
– 2) core point condition: |NEps (q)| >= MinPts
p
q
MinPts = 5
e = 1 cm

Contd..
• Density-reachable:
– A point p is density-reachable from a point q wrt. Eps, MinPts if there is a
chain of points p1, …, pn, q = p1,….. pn = p such that pi+1 is directly
density-reachable from pi
p
q
p1

Contd..
• Density-connected:
– A point p is density-connected to a point q wrt. Eps, MinPts if there is a
point o such that both, p and q are density-reachable from o wrt. Eps and
MinPts.
p q
o

Contd..• Density = number of points within a specified radius (Eps).
• A point is a core point if it has at least a specified number of
points (MinPts) within Eps.
• These are points that are at the interior of a cluster
• Counts the point itself
• A border point is not a core point, but is in the neighborhood of a
core point
• A noise point is any point that is not a core point or a border
point
e.g.,: Minpts=7

DBSCAN(Density Based Spatial Clustering of Applications with Noise)
• To find the next cluster, DBSCAN randomly selects an unvisited object
from the remaining ones. The clustering process continues until all
objects are visited.

Contd..

Contd..
• Example:
– If Epsilon is 2 and minpoint is 2, what are the clusters that DBScan would
discover with the following 8 examples: A1=(2,10), A2=(2,5), A3=(8,4),
A4=(5,8), A5=(7,5), A6=(6,4), A7=(1,2), A8=(4,9).
• Solution :
– d(a,b) denotes the Eucledian distance between a and b. It is obtained
directly from the distance matrix calculated as follows:
– d(a,b)=sqrt((xb-xa)2+(yb-ya)2))

Contd..
A1 A2 A3 A4 A5 A6 A7 A8
A1 0 √25 √36 √13 √50 √52 √65 √5
A2 0 √37 √18 √25 √17 √10 √20
A3 0 √25 √2 √2 √53 √41
A4 0 √13 √17 √52 √2
A5 0 √2 √45 √25
A6 0 √29 √29
A7 0 √58
A8 0

Contd..
• N2(A1)={};
• N2(A2)={};
• N2(A3)={A5, A6};
• N2(A4)={A8};
• N2(A5)={A3, A6};
• N2(A6)={A3, A5};
• N2(A7)={};
• N2(A8)={A4};
• So A1, A2, and A7 are outliers, while we have two clusters C1={A4,
A8} and C2={A3, A5, A6}

Contd..

Advantages and Disadvantages of DBSCAN algorithm:
• Advantages:
– DBSCAN does not require one to specify the number of clusters in the
data priori, as opposed to k-means.
– DBSCAN can find arbitrarily shaped clusters
– DBSCAN is robust to outliers.
– DBSCAN is mostly insensitive to the ordering of the points in the
database.
– The parameters minPts and ε can be set by a domain expert, if the data is
well understood.

Contd..
• Disadvantages:
– DBSCAN is not entirely deterministic: border points that are reachable
from more than one cluster can be part of either cluster, depending on the
order the data is processed. Fortunately, this situation does not arise often,
and has little impact on the clustering result: both on core points and noise
points, DBSCAN is deterministic
– DBSCAN cannot cluster data sets well with large differences in densities,
since the minPts-ε combination cannot then be chosen appropriately for all
clusters.
– If the data and scale are not well understood, choosing a meaningful
distance threshold ε can be difficult.

Homework
• Explain the aims of cluster analysis.
• What is clustering? How is it different than supervised classification?
In what situation clustering can be useful?
• List and explain desired features of cluster analysis.
• Explain the different types of cluster analysis methods and discuss their
features.
• Describe the k-means algorithm and write its strengths and
weaknesses.
• Describe the features of Hierarchical clustering methods? In what
situations are these methods useful?

Thank You !
Compiled by : Kamal Acharya 647/2/2019

Cluster Analysis

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Cluster Analysis

Similar to Cluster Analysis (20)

More from Kamal Acharya

More from Kamal Acharya (20)

Recently uploaded

Recently uploaded (20)

Cluster Analysis