Successfully reported this slideshow.
Your SlideShare is downloading. ×

Hands on Mahout!

Upcoming SlideShare
Mahout part2
Mahout part2
Loading in …3

Check these out next

1 of 68 Ad

More Related Content

Slideshows for you (20)

Viewers also liked (15)


Similar to Hands on Mahout! (20)

More from OSCON Byrum (20)


Recently uploaded (20)

Hands on Mahout!

  1. Hands on! Speakers: Ted Dunning, Robin Anil OSCON 2011, Portland
  2. About Us <ul><li>Ted Dunning: Chief Application Architect at MapR Committer and PMC Member at Apache Mahout Previously: MusicMatch (Yahoo! Music), Veoh recommendation, ID Analytics </li></ul><ul><li>Robin Anil: Software Engineer at Google Committer and PMC Member at Apache Mahout Previously: Yahoo! (Display ads), Minekey recommendation </li></ul>
  3. Agenda <ul><li>Intro to Mahout (5 mins) </li></ul><ul><li>Overview of Algorithms in Mahout (10 mins) </li></ul><ul><li>Hands on Mahout! </li></ul><ul><ul><li>Clustering (30 mins) </li></ul></ul><ul><ul><li>Classification (30 mins) </li></ul></ul><ul><ul><li>Advanced topics with Q&A (15 mins) </li></ul></ul>
  4. Mission <ul><li>To build a scalable machine learning library </li></ul>
  5. Scale! <ul><li>Scale to large datasets </li></ul><ul><ul><li>Hadoop MapReduce implementations that scales linearly with data. </li></ul></ul><ul><ul><li>Fast sequential algorithms whose runtime doesn’t depend on the size of the data </li></ul></ul><ul><ul><li>Goal: To be as fast as possible for any algorithm </li></ul></ul><ul><li>Scalable to support your business case </li></ul><ul><ul><li>Apache Software License 2 </li></ul></ul><ul><li>Scalable community </li></ul><ul><ul><li>Vibrant, responsive and diverse </li></ul></ul><ul><ul><li>Come to the mailing list and find out more </li></ul></ul>
  6. Current state of ML libraries <ul><li>Lack community </li></ul><ul><li>Lack scalability </li></ul><ul><li>Lack documentations and examples </li></ul><ul><li>Lack Apache licensing </li></ul><ul><li>Are not well tested </li></ul><ul><li>Are Research oriented </li></ul><ul><li>Not built over existing production quality libraries </li></ul><ul><li>Lack “Deployability” </li></ul>
  7. Algorithms and Applications
  8. Clustering <ul><li>Call it fuzzy grouping based on a notion of similarity </li></ul>
  9. Mahout Clustering <ul><li>Plenty of Algorithms: K-Means, Fuzzy K-Means, Mean Shift, Canopy, Dirichlet </li></ul><ul><li>Group similar looking objects </li></ul><ul><li>Notion of similarity: Distance measure: </li></ul><ul><ul><li>Euclidean </li></ul></ul><ul><ul><li>Cosine </li></ul></ul><ul><ul><li>Tanimoto </li></ul></ul><ul><ul><li>Manhattan </li></ul></ul>
  10. Classification <ul><li>Predicting the type of a new object based on its features </li></ul><ul><li>The types are predetermined </li></ul><ul><li> Dog Cat </li></ul>
  11. Mahout Classification <ul><li>Plenty of algorithms </li></ul><ul><ul><li>Naïve Bayes </li></ul></ul><ul><ul><li>Complementary Naïve Bayes </li></ul></ul><ul><ul><li>Random Forests </li></ul></ul><ul><ul><li>Logistic Regression (SGD) </li></ul></ul><ul><ul><li>Support Vector Machines (patch ready) </li></ul></ul><ul><li>Learn a model from a manually classified data </li></ul><ul><li>Predict the class of a new object based on its features and the learned model </li></ul>
  12. Part 1 - Clustering
  13. Understanding data - Vectors <ul><li>The vector denoted by point (5, 3) is simply Array([5, 3]) or HashMap([0 => 5], [1 => 3]) </li></ul>X = 5 , Y = 3 (5, 3) Y X
  14. Representing Vectors – The basics <ul><li>Now think 3, 4, 5, ….. n-dimensional </li></ul><ul><li>Think of a document as a bag of words. </li></ul><ul><ul><li>“ she sells sea shells on the sea shore” </li></ul></ul><ul><li>Now map them to integers </li></ul><ul><ul><li>she => 0 </li></ul></ul><ul><ul><li>sells => 1 </li></ul></ul><ul><ul><li>sea => 2 </li></ul></ul><ul><ul><li>and so on </li></ul></ul><ul><li>The resulting vector [1.0, 1.0, 2.0, … ] </li></ul>
  15. Vectors <ul><li>Imagine one dimension for each word. </li></ul><ul><li>Each dimension is also called a feature </li></ul><ul><li>Two techniques </li></ul><ul><ul><li>Dictionary Based </li></ul></ul><ul><ul><li>Randomizer Based </li></ul></ul>
  16. Clustering Reuters dataset
  17. Step 1 – Convert dataset into a Hadoop Sequence File <ul><li> </li></ul><ul><li>Download (8.2 MB) and extract the SGML files. </li></ul><ul><ul><li>$ mkdir -p mahout-work/reuters-sgm </li></ul></ul><ul><ul><li>$ cd mahout-work/reuters-sgm && tar xzf ../reuters21578.tar.gz && cd .. && cd .. </li></ul></ul><ul><li>Extract content from SGML to text file </li></ul><ul><ul><li>$ bin/mahout org.apache.lucene.benchmark.utils.ExtractReuters mahout-work/reuters-sgm mahout-work/reuters-out </li></ul></ul>
  18. Step 1 – Convert dataset into a Hadoop Sequence File <ul><li>Use seqdirectory tool to convert text file into a Hadoop Sequence File </li></ul><ul><ul><li>$ bin/mahout seqdirectory </li></ul></ul><ul><ul><li>-i mahout-work/reuters-out </li></ul></ul><ul><ul><li>-o mahout-work/reuters-out-seqdir </li></ul></ul><ul><ul><li>-c UTF-8 -chunk 5 </li></ul></ul>
  19. Hadoop Sequence File <ul><li>Sequence of Records, where each record is a <Key, Value> pair </li></ul><ul><ul><li><Key1, Value1> </li></ul></ul><ul><ul><li><Key2, Value2> </li></ul></ul><ul><ul><li>… </li></ul></ul><ul><ul><li>… </li></ul></ul><ul><ul><li>… </li></ul></ul><ul><ul><li><Keyn, Valuen> </li></ul></ul><ul><li>Key and Value needs to be of class </li></ul><ul><ul><li>Key = Record name or File name or unique identifier </li></ul></ul><ul><ul><li>Value = Content as UTF-8 encoded string </li></ul></ul><ul><li>TIP: Dump data from your database directly into Hadoop Sequence Files (see next slide) </li></ul>
  20. Writing to Sequence Files <ul><li>Configuration conf = new Configuration(); </li></ul><ul><li>FileSystem fs = FileSystem.get(conf) ; </li></ul><ul><li>Path path = new Path(&quot;testdata/part-00000&quot;); </li></ul><ul><li>SequenceFile.Writer writer = new SequenceFile.Writer( </li></ul><ul><li>fs, conf, path, Text.class, Text.class ); </li></ul><ul><li>for (int i = 0; i < MAX_DOCS; i++) </li></ul><ul><li>writer.append( new Text(documents(i).Id()), </li></ul><ul><li>new Text(documents(i).Content())); </li></ul><ul><li>} </li></ul><ul><li>writer.close(); </li></ul>
  21. Generate Vectors from Sequence Files <ul><li>Steps </li></ul><ul><ul><li>Compute Dictionary </li></ul></ul><ul><ul><li>Assign integers for words </li></ul></ul><ul><ul><li>Compute feature weights </li></ul></ul><ul><ul><li>Create vector for each document using word-integer mapping and feature-weight Or </li></ul></ul><ul><li>Simply run $ bin/mahout seq2sparse </li></ul>
  22. Generate Vectors from Sequence Files <ul><li>$ bin/mahout seq2sparse -i mahout-work/reuters-out-seqdir/ -o mahout-work/reuters-out-seqdir-sparse-kmeans </li></ul><ul><li>Important options </li></ul><ul><ul><li>Ngrams </li></ul></ul><ul><ul><li>Lucene Analyzer for tokenizing </li></ul></ul><ul><ul><li>Feature Pruning </li></ul></ul><ul><ul><ul><li>Min support </li></ul></ul></ul><ul><ul><ul><li>Max Document Frequency </li></ul></ul></ul><ul><ul><ul><li>Min LLR (for ngrams) </li></ul></ul></ul><ul><ul><li>Weighting Method </li></ul></ul><ul><ul><ul><li>TF v/s TFIDF </li></ul></ul></ul><ul><ul><ul><li>lp-Norm </li></ul></ul></ul><ul><ul><ul><li>Log normalize length </li></ul></ul></ul>
  23. Start K-Means clustering <ul><li>$ bin/mahout kmeans -i mahout-work/reuters-out-seqdir-sparse-kmeans/tfidf-vectors/ -c mahout-work/reuters-kmeans-clusters -o mahout-work/reuters-kmeans -dm org.apache.mahout.distance.CosineDistanceMeasure –cd 0.1 -x 10 -k 20 –ow </li></ul><ul><li>Things to watch out for </li></ul><ul><ul><li>Number of iterations </li></ul></ul><ul><ul><li>Convergence delta </li></ul></ul><ul><ul><li>Distance Measure </li></ul></ul><ul><ul><li>Creating assignments </li></ul></ul>
  24. K-Means clustering c1 c2 c3
  25. K-Means clustering c1 c2 c3
  26. K-Means clustering c1 c2 c3 c1 c2 c3
  27. K-Means clustering c1 c2 c3
  28. Inspect clusters <ul><li>$ bin/mahout clusterdump -s mahout-work/reuters-kmeans/clusters-9 -d mahout-work/reuters-out-seqdir-sparse-kmeans/dictionary.file-0 -dt sequencefile -b 100 -n 20 </li></ul><ul><li>Typical output </li></ul><ul><li>:VL-21438{n=518 c=[0.56:0.019, 00:0.154, 00.03:0.018, 00.18:0.018, … </li></ul><ul><li>Top Terms: </li></ul><ul><li>iran => 3.1861672217321213 </li></ul><ul><li>strike => 2.567886952727918 </li></ul><ul><li>iranian => 2.133417966282966 </li></ul><ul><li>union => 2.116033937940266 </li></ul><ul><li>said => 2.101773806290277 </li></ul><ul><li>workers => 2.066259451354332 </li></ul><ul><li>gulf => 1.9501374918521601 </li></ul><ul><li>had => 1.6077752463145605 </li></ul><ul><li>he => 1.5355078004962228 </li></ul>
  29. FAQs <ul><li>How to get rid of useless words </li></ul><ul><li>How to see documents to cluster assignments </li></ul><ul><li>How to choose appropriate weighting </li></ul><ul><li>How to run this on a cluster </li></ul><ul><li>How to scale </li></ul><ul><li>How to choose k </li></ul><ul><li>How to improve similarity measurement </li></ul>
  30. FAQs <ul><li>How to get rid of useless words </li></ul><ul><ul><li>Increase minSupport and or decrease dfPercent </li></ul></ul><ul><ul><li>Use StopwordsAnalyzer </li></ul></ul><ul><li>How to see documents to cluster assignments </li></ul><ul><ul><li>Run clustering process at the end of centroid generation using –cl </li></ul></ul><ul><li>How to choose appropriate weighting </li></ul><ul><ul><li>If its long text, go with tfidf. Use normalization if documents different in length </li></ul></ul><ul><li>How to run this on a cluster </li></ul><ul><ul><li>Set HADOOP_CONF directory to point to your hadoop cluster conf directory </li></ul></ul><ul><li>How to scale </li></ul><ul><ul><li>Use small value of k to partially cluster data and then do full clustering on each cluster. </li></ul></ul>
  31. FAQs <ul><li>How to choose k </li></ul><ul><ul><li>Figure out based on the data you have. Trial and error </li></ul></ul><ul><ul><li>Or use Canopy Clustering and distance threshold to figure it out </li></ul></ul><ul><ul><li>Or use Spectral clustering </li></ul></ul><ul><li>How to improve Similarity Measurement </li></ul><ul><ul><li>Not all features are equal </li></ul></ul><ul><ul><li>Small weight difference for certain types creates a large semantic difference </li></ul></ul><ul><ul><li>Use WeightedDistanceMeasure </li></ul></ul><ul><ul><li>Or write a custom DistanceMeasure </li></ul></ul>
  32. Interesting problems <ul><li>Cluster users talking about OSCON’11 and cluster them based on what they are tweeting </li></ul><ul><ul><li>Can you suggest people to network with. </li></ul></ul><ul><li>Use user generate tags that people have given for musicians and cluster them </li></ul><ul><ul><li>Use the cluster to pre-populate suggest-box to autocomplete tags when users type </li></ul></ul><ul><li>Cluster movies based on abstract and description and show related movies. </li></ul><ul><ul><li>Note: How it can augment recommendations or collaborative filtering algorithms. </li></ul></ul>
  33. More clustering algorithms <ul><li>Canopy </li></ul><ul><li>Fuzzy K-Means </li></ul><ul><li>Mean Shift </li></ul><ul><li>Dirichlet process clustering </li></ul><ul><li>Spectral clustering. </li></ul>
  34. Part 2 - Classification
  35. Preliminaries <ul><li>Code is available from github: </li></ul><ul><ul><li> </li></ul></ul><ul><li>EC2 instances available </li></ul><ul><li>Thumb drives also available </li></ul><ul><li>Email to [email_address] </li></ul><ul><li>Twitter @ted_dunning </li></ul>
  36. A Quick Review <ul><li>What is classification? </li></ul><ul><ul><li>goes-ins: predictors </li></ul></ul><ul><ul><li>goes-outs: target variable </li></ul></ul><ul><li>What is classifiable data? </li></ul><ul><ul><li>continuous, categorical, word-like, text-like </li></ul></ul><ul><ul><li>uniform schema </li></ul></ul><ul><li>How do we convert from classifiable data to feature vector? </li></ul>
  37. Data Flow Not quite so simple
  38. Classifiable Data <ul><li>Continuous </li></ul><ul><ul><li>A number that represents a quantity, not an id </li></ul></ul><ul><ul><li>Blood pressure, stock price, latitude, mass </li></ul></ul><ul><li>Categorical </li></ul><ul><ul><li>One of a known, small set (color, shape) </li></ul></ul><ul><li>Word-like </li></ul><ul><ul><li>One of a possibly unknown, possibly large set </li></ul></ul><ul><li>Text-like </li></ul><ul><ul><li>Many word-like things, usually unordered </li></ul></ul>
  39. But that isn’t quite there <ul><li>Learning algorithms need feature vectors </li></ul><ul><ul><li>Have to convert from data to vector </li></ul></ul><ul><li>Can assign one location per feature </li></ul><ul><ul><li>or category </li></ul></ul><ul><ul><li>or word </li></ul></ul><ul><li>Can assign one or more locations with hashing </li></ul><ul><ul><li>scary </li></ul></ul><ul><ul><li>but safe on average </li></ul></ul>
  40. Data Flow
  42. The pipeline Classifiable Data Vectors
  43. Instance and Target Variable
  44. Instance and Target Variable
  45. Hashed Encoding
  46. What about collisions?
  47. Let’s write some code (cue relaxing background music)
  48. Generating new features <ul><li>Sometimes the existing features are difficult to use </li></ul><ul><li>Restating the geometry using new reference points may help </li></ul><ul><li>Automatic reference points using k-means can be better than manual references </li></ul>
  49. K-means using target
  50. K-means features
  51. More code! (cue relaxing background music)
  52. Integration Issues <ul><li>Feature extraction is ideal for map-reduce </li></ul><ul><ul><li>Side data adds some complexity </li></ul></ul><ul><li>Clustering works great with map-reduce </li></ul><ul><ul><li>Cluster centroids to HDFS </li></ul></ul><ul><li>Model training works better sequentially </li></ul><ul><ul><li>Need centroids in normal files </li></ul></ul><ul><li>Model deployment shouldn’t depend on HDFS </li></ul>
  53. Parallel Stochastic Gradient Descent Average models Train sub model Model I n p u t
  54. Variational Dirichlet Assignment Update model Gather sufficient statistics Model I n p u t
  55. Old tricks, new dogs <ul><li>Mapper </li></ul><ul><ul><li>Assign point to cluster </li></ul></ul><ul><ul><li>Emit cluster id, (1, point) </li></ul></ul><ul><li>Combiner and reducer </li></ul><ul><ul><li>Sum counts, weighted sum of points </li></ul></ul><ul><ul><li>Emit cluster id, (n, sum/n) </li></ul></ul><ul><li>Output to HDFS </li></ul>Read from HDFS to local disk by distributed cache Written by map-reduce Read from local disk from distributed cache
  56. Old tricks, new dogs <ul><li>Mapper </li></ul><ul><ul><li>Assign point to cluster </li></ul></ul><ul><ul><li>Emit cluster id, 1, point </li></ul></ul><ul><li>Combiner and reducer </li></ul><ul><ul><li>Sum counts, weighted sum of points </li></ul></ul><ul><ul><li>Emit cluster id, n, sum/n </li></ul></ul><ul><li>Output to HDFS </li></ul>MapR FS Read from NFS Written by map-reduce
  57. Modeling architecture Feature extraction and down sampling I n p u t Side-data Data join Sequential SGD Learning Map-reduce Now via NFS
  58. More in Mahout
  59. Topic modeling <ul><li>Grouping similar or co-occurring features into a topic </li></ul><ul><ul><li>Topic “Lol Cat”: </li></ul></ul><ul><ul><ul><li>Cat </li></ul></ul></ul><ul><ul><ul><li>Meow </li></ul></ul></ul><ul><ul><ul><li>Purr </li></ul></ul></ul><ul><ul><ul><li>Haz </li></ul></ul></ul><ul><ul><ul><li>Cheeseburger </li></ul></ul></ul><ul><ul><ul><li>Lol </li></ul></ul></ul>
  60. Mahout Topic Modeling <ul><li>Algorithm: Latent Dirichlet Allocation </li></ul><ul><ul><li>Input a set of documents </li></ul></ul><ul><ul><li>Output top K prominent topics and the features in each topic </li></ul></ul>
  61. Recommendations <ul><li>Predict what the user likes based on </li></ul><ul><ul><li>His/Her historical behavior </li></ul></ul><ul><ul><li>Aggregate behavior of people similar to him </li></ul></ul>
  62. Mahout Recommenders <ul><li>Different types of recommenders </li></ul><ul><ul><li>User based </li></ul></ul><ul><ul><li>Item based </li></ul></ul><ul><li>Full framework for storage, online online and offline computation of recommendations </li></ul><ul><li>Like clustering, there is a notion of similarity in users or items </li></ul><ul><ul><li>Cosine, Tanimoto, Pearson and LLR </li></ul></ul>
  63. Frequent Pattern Mining <ul><li>Find interesting groups of items based on how they co-occur in a dataset </li></ul>
  64. Mahout Parallel FPGrowth <ul><li>Identify the most commonly occurring patterns from </li></ul><ul><ul><li>Sales Transactions buy “Milk, eggs and bread” </li></ul></ul><ul><ul><li>Query Logs </li></ul></ul><ul><ul><ul><li>ipad -> apple, tablet, iphone </li></ul></ul></ul><ul><ul><li>Spam Detection </li></ul></ul><ul><ul><ul><li>Yahoo! </li></ul></ul></ul>
  65. Get Started <ul><li> </li></ul><ul><li>[email_address] - Developer mailing list </li></ul><ul><li>[email_address] - User mailing list </li></ul><ul><li>Check out the documentations and wiki for quickstart </li></ul><ul><li> Browse Code </li></ul><ul><li>Send me email! </li></ul><ul><ul><li>[email_address] </li></ul></ul><ul><ul><li>[email_address] </li></ul></ul><ul><ul><li>[email_address] </li></ul></ul><ul><li>Try out MapR! </li></ul><ul><ul><li> </li></ul></ul>
  66. Resources <ul><li>“ Mahout in Action” Owen, Anil, Dunning, Friedman </li></ul><ul><li>“ Taming Text” Ingersoll, Morton, Farris </li></ul><ul><li>“ Introducing Apache Mahout” </li></ul>
  67. Thanks to <ul><li>Apache Foundation </li></ul><ul><li>Mahout Committers </li></ul><ul><li>Google Summer of Code Organizers </li></ul><ul><li>And Students </li></ul><ul><li>OSCON </li></ul><ul><li>Open source! </li></ul>
  68. References <ul><li> </li></ul><ul><li>Cat </li></ul><ul><li>Dog </li></ul><ul><li>Milk Eggs Bread </li></ul><ul><li>Amazon Recommendations </li></ul><ul><li>twitter </li></ul>

Editor's Notes

  • 9.1 top left
  • 9.1 top right
  • 9.1 bottom left
  • 9.1 bottom right