Clustering large-scale data TU Berlin talk

Clustering large-scale data
Dan Filimon, Politehnica University Bucharest
Ted Dunning, MapR Technologies
Buzzwords talk at http://goo.gl/9FxbN
Saturday, June 1, 13

whoami
• Soon-to-be graduate of Politehnica Univerity of
Bucharest
• Apache Mahout committer
• Contact me at
• dﬁlimon@apache.org
• dangeorge.ﬁlimon@gmail.com

The problem
Group n d-dimensional datapoints into k disjoint
sets to minimize
ckX
c1
0
@
X
xij2Xi
dist(xij, ci)
1
A
Xi is the ith
cluster
ci is the centroid of the ith
cluster
xij is the jth
point from the ith
cluster
dist(x, y) = ||x y||2

k-means
points to cluster

k-means
initial centroids

k-means
ﬁrst iteration assignment

k-means
adjusted centroids

k-means
second iteration assignment

Details
• Quality
• How to initialize the centroids
• When to stop iterating
• How to deal with outliers
• Speed
• Complexity of cluster assignment

k-means as a MapReduce
• Can’t split k-means loop directly
• Must express a single k-means step as a MapReduce

Now for something
completely different
• Attempt to build a “sketch” of the data in one pass
• O(k log n) intermediate clusters
• Can ﬁt into memory
• Ball k-means on the sketch for k ﬁnal clusters

streaming k-means
ﬁrst point

streaming k-means
becomes centroid

streaming k-means
second point
far away

streaming k-means
becomes centroid
far away

streaming k-means
third point
close

streaming k-means
centroid is updated

streaming k-means
close
but becomes a
new centroid

streaming k-means

The big picture
MapReduce
• Can cluster all the points with just 1 MapReduce:
• m mappers run streaming k-means
• 1 reducer runs ball k-means to get k clusters

Quality
• Compared the quality on various small-medium UCI data
sets
• iris, seeds, movement, control, power
• k-means vs streaming k-means + ball k-means is
comparable quality-wise

iris randplot-2
0.5 1.0 1.5 2.0 2.5 3.0 3.5
0.51.01.52.02.53.03.5
0
51
0
62
0
0
0
0
38
Confusion matrix run 2
Adjusted Rand Index 1
Ball/Streaming k−means centroids
Mahoutk−meanscentroids

seeds compareplot
bskm km
4567
4.3116448 4.22126213333333
Clustering techniques compared all
Clustering type / overall mean cluster distance
Meanclusterdistance

movement compareplot
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
0 20 40 60
1.01.52.02.51.403378
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
1.50878
●
●
km distances
bskm distances
km mean
bskm mean
km vs bskm
Cluster index/Run
Averageclusterdistance

Overall (avg. 5 runs)
Dataset Clustering Avg. Dunn Avg. DB Avg. Cost Avg.ARI
iris
km 9.161 0.265 124.146
0.905iris
bskm 6.454 0.336 117.859
0.905
seeds
km 7.432 0.453 909.875
0.980seeds
bskm 6.886 0.505 916.511
0.980
movement
km 0.457 1.843 336.456
0.650movement
bskm 0.436 2.003 347.078
0.650
control
km 0.553 1.700 1014313
0.630control
bskm 0.753 1.434 1004917
0.630
power
km 0.107 1.380 73339083
0.605power
bskm 1.953 1.080 54422758
0.605

Speed (Threaded)

Speed (MapReduce)
0
17.5
35
52.5
70
20 clusters 1 iteration
KMeans StreamingKMeans
iteration times comparable
~2.4x faster

Speed (MapReduce)
0
75
150
225
300
1000 clusters 1 iteration
KMeans StreamingKMeans
beneﬁts from
approximate
nearest-neighbor
search
~8x faster

My take
Contribute to open source!

My take
Respect language conventions!

My take
Learn math!
Statistics and Linear Algebra especially

My take
Write tests!
They let you know when you break things

My take
Proﬁle your code!
Especially when it’s oddly slow :)

My take
Use source control effectively!
Merge often, have multiple branches

My take
Microbenchmarks are tricky in Java
Use frameworks like Caliper

My take
Ask lots of questions!
Don’t be afraid of saying something stupid

Thank you!
Questions?
dﬁlimon@apache.org
dangeorge.ﬁlimon@gmail.com

Clustering large-scale data TU Berlin talk

Recommended

Recommended

More Related Content

Recently uploaded

Recently uploaded (20)

Featured

Featured (20)

Clustering large-scale data TU Berlin talk