See conference video - http://www.lucidimagination.com/devzone/events/conferences/ApacheLuceneEurocon2011 For more than a decade internet search engines have helped users find documents they are looking for. However, what if users aren't looking for anything specific but want a summary of a large document collection and want to be surprised? One solution to this problem is document clustering. Clustering algorithms group documents that have similar content. Real-life examples of clustering are clustered search results of Google news, or tag clouds which group documents under a shared label. Apache Mahout is a framework for scalable machine learning on top of Apache Hadoop and can be used for large scale document clustering. This talk introduces clustering in general and shows you step-by-step how to configure Mahout clustering jobs to create a tag cloud from a document collection. This talk is suitable for people who have some experience with Hadoop and perhaps Mahout. Knowledge of clustering is not required.

1. Configuring
Clustering Jobs
Frank Scholten, DutchWorks
frank.scholten@dutchworks.nl, 19 october 2011

2. My Background
Frank Scholten @Frank_Scholten
Software Developer at
Blogger at
user & contributor
2

3. Agenda
What is clustering?
Intro to
Clustering
3

4. Clustering introduction
So much How to get a nice
data... overview?
4

5. What is clustering?
Grouping & summarizing data
Unsupervised machine learning
“...the assignment of a set of observations
into subsets so that observations in the
same clusters are similar in some sense...”
Source: Wikipedia
5

6. Applications
Market segmentation
Species identifications
Machine vision
Information retrieval & search
...and many more!
6

7. Example - Google news
7

8. 2-D Clustering example
Intra-cluster
distance
Inter-cluster
distance
Legend Point Cluster Cluster Center
8

9. K-Means algorithm
Select K random vectors
Specify distance measure + threshold
Every iteration
●
Add vector closest to cluster
●
Recompute center
●
Converged if no vectors within threshold
9

10. Intro to
10

11. Is this SPAM?
Classification
Collaborative
Filtering
And much more!
Clustering
11

12. The Project
Apache project started in 2008
Scalable machine learning, often with Hadoop
Steadily growing community
Version 0.6 coming soon
12

13. bin/mahout
frank@frankthetank:~$ mahout
no HADOOP_HOME set, running locally
An example program must be given as the
first argument.
Valid program names are:
arff.vector: : Generate Vectors from an ARFF
file or directory
canopy: : Canopy clustering
cat: : Print a file or resource as the
logistic regression models would see
it
...
13

14. Help
frank@frankthetank:~$ mahout kmeans ­­help
usage: <command> [Generic Options]
[Job­specific Options]
Generic Options:
...
Job­specific Options:
­­input (­i) input Path to job input directory.
­­output (­o) output The directory pathname for
output.
...
14

15. Java Drivers
String[] args = new String[] {
"­­input", input,
"­­output", output,
"­­clusters", clusters,
"­­clustering",
"­­numClusters", “10”
};
ToolRunner.run(conf, new KmeansDriver(), args);
15

16. Text clustering
process
[ 0.03, 0.95, 0.45, 0.34 ]
[ 0.02, 0.98, 0.73, 0.55 ]
Text files or Vectors
Sequence files
Lucene index
K-means Clusters
Find n-grams Dictionary
(CL-1, [0.32, 0.6, .. ]
(CL-1, [0.76, 0.1, .. ]
(CL-1,23) (CL-2, [0.98, 0.2, .. ]
Quick fox Dog (CL-1,37)
(CL-2,45)
Cluster labels Points
16

17. Text clustering
programs
$ mahout seqdirectory
Text files or Sequence files
Lucene index
[ 0.03, 0.95, .. ] $ mahout seq2sparse
[ 0.29, 0.98, .. ]
Sequence files Vectors
[ 0.03, 0.95, .. ]
[ 0.29, 0.98, .. ] $ mahout kmeans
Vectors Clusters
17

18. Clustering
18

19. Clustering
Publicly available monthly dumps
Posts ~ 5.5 GB ~ 1.4 M questions (April 2011)
Let's use to extract a tag cloud!
19

20. Clustering
Cluster
Vectorize Index
[ 0,1,0,1,1,1,0,0,1,0,1 ]
[ 0,1,1,1,1,0,0,0,0,0,1 ]
Text
Join content
& clusters
Java Git Lucene
Pre-process
XML & HTML Regular expressions
Post ID Version control
& Title
20

21. Clustering
How to implement?
Pre-process XML XML & HTML parsing
Vectorize Custom Analyzer
Cluster
Index
21

22. [ 0,1,0,1,1,1,0,0,1,0,1 ]
Vectorize [ 0,1,1,1,1,0,0,0,0,0,1 ]
Many options and flags
$ mahout seq2sparse
­­input ..
­­output ..
­­analyzerClass ..
­­maxDFPercent ..
­­minDF ..
22

23. Cluster
Run one of the clustering algorithms!
K-means, Fuzzy K-means, Canopy,
Mean-shift, Min-hash, LDA
All with different pros and cons
23

24. Index
Custom code to join data at index time
Index clusters
(cluster_id, cluster_name, size)
Index posts
(post_id, post_cluster_id, title)
24

25. Demo time!
25

26. Conclusions
Clustering is fun!
Vectorization & labeling improvements
Tools for cluster evaluation?
26

27. References
Mahout in Action – Just released!
Sean Owen, Ted Dunning, Robin Anil, Ellen Friedman
{user|dev}@mahout.apache.org
http://jira.apache.org/MAHOUT
http://www.searchworkings.org
27

28. Q&A
28