Cluster analysis using k-means
Professor, D.Sc. (Phys. & Math.)
Formulation of the problem
The task of cluster analysis is to divide the existing set of
points on a certain number of groups (clusters) so that the sum
of squares of points distances from cluster centers was minimal.
At the point of minimum all cluster centers coincide with the
centers of the corresponding areas of Voronoi diagram.
Hartigan and Wong Lloyd
The initial approximation
First step is to set the initial approximation of cluster centers.
To do this, such methods are most commonly used:
to set the centers of clusters directly;
to set the number of clusters k and take the first k
points coordinates as centers;
to set the number of clusters k and take the
randomly selected k points coordinates as centers (it is
appropriate to carry out calculations for several
random runs of the algorithm).
1. Placing of each point to the cluster center of which is the
nearest to it. As a measure of closeness squared Euclidean
distance is used most commonly, but other measures of
distance also may be selected.
2. Recalculation of cluster centers coordinates. If the measure
of closeness is the Euclidean distance (or its square), cluster
centers are calculated as the arithmetic means of corresponding
coordinates of points that belong to these clusters.
The iterations are stopped when the specified maximum
number of iterations is carried out, or if there is no longer
change of the clusters composition.
Limitations and shortcomings
Using of random
Slow work on large
Forming of data array
a1 = matrix(c(rnorm(20, mean = 5, sd = 1), rnorm(20, mean = 5,
sd = 1)), nrow=20, ncol = 2)
a2 = matrix(c(rnorm(20, mean = 5, sd = 1), rnorm(20, mean =
13, sd = 1)), nrow=20, ncol = 2)
a3 = matrix(c(rnorm(20, mean = 12, sd = 1), rnorm(20, mean =
6, sd = 1)), nrow=20, ncol = 2)
a4 = matrix(c(rnorm(20, mean = 12, sd = 1), rnorm(20, mean =
12, sd = 1)), nrow=20, ncol = 2)
a <- rbind(a1,a2,a3,a4)
Function rbind() forms matrix a, in which the first 20 rows are the
corresponding strings of matrix a1, next 20 – matrix a2 and so
Next, we must calculate the matrix of values of formed group
centers and display the results on a screen:
For forming the clusters by k-means method we can use the function:
kmeans(x, centers, iter.max = 10, nstart = 1, algorithm =
c("Hartigan-Wong", "Lloyd", "Forgy", "MacQueen") )
x – matrix of numerical data;
centers – initial approximation of cluster centers or number of
clusters (in the latter case, the appropriate number of randomly
selected rows of the matrix will be taken as the initial approximation
iter.max – maximum number of iterations;
nstart – number of random sets which must be chosen if centers – is
the number of clusters;
algorithm – choice of clustering algorithm.
Comparison of centers
4,613619 5,169488 4,613619 5,169488
4,570456 13,396202 4,570456 13,396202
11,855793 5,936099 11,855793 5,936099
12,197688 11,930728 12,197688 11,930728
5,531175 5,405187 5,545309 5,527677
5,340795 12,983168 5,472965 13,239925
11,770917 6,725708 11,842934 6,916365
Using command sd(resid.a) we can calculate residues
standard deviations. They are close to the given values of
standard deviations of initial arrays. It confirms the adequacy of
the clustering results.