SDEC2011 Mahout - the what, the how and the why
Upcoming SlideShare
Loading in...5
×
 

SDEC2011 Mahout - the what, the how and the why

on

  • 4,214 views

Mahout is an open source machine learning library from Apache. From its humble beginnings at Apache Lucene, the project has grown into a active community of developers, machine learning experts and ...

Mahout is an open source machine learning library from Apache. From its humble beginnings at Apache Lucene, the project has grown into a active community of developers, machine learning experts and enthusiasts. With v0.5 released recently, the project has been focussing full steam on developing stable APIs with an eye on our major milestone of v1.0. The speaker has been with Mahout from his days in college as a computer science student. The talk will focus on the major use cases of Mahout. The design decisions, things that worked, things that didn't, and things to expect in the future releases.

http://sdec.kr/

Statistics

Views

Total Views
4,214
Views on SlideShare
4,211
Embed Views
3

Actions

Likes
7
Downloads
289
Comments
0

1 Embed 3

http://localhost 3

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    SDEC2011 Mahout - the what, the how and the why SDEC2011 Mahout - the what, the how and the why Presentation Transcript

    • The what, the why and the howSpeaker: Robin Anil, Apache Mahout PMC Member SDEC, Seoul, Korea, June 2011
    • about:me● Apache Mahout PMC member● A ML Believer● Author of Mahout in Action● Software Engineer @ Google *in no particular order● Previous Life: Google Summer of Code student for 2 years.
    • about:agenda● Introducing Mahout● Why Mahout?● Birds eye view of Mahout● Classic Machine learning problems● Short overview of Mahout clustering
    • about:missionTo build a scalable machine learning library
    • Scale!● Scale to large datasets ○ Hadoop MapReduce implementations that scales linearly with data. ○ Fast sequential algorithms whose runtime doesn’t depend on the size of the data ○ Goal: To be as fast as possible for any algorithm● Scalable to support your business case ○ Apache Software License 2● Scalable community ○ Vibrant, responsive and diverse ○ Come to the mailing list and find out more
    • about:why● Lack community● Lack scalability● Lack documentations and examples● Lack Apache licensing● Are not well tested● Are Research oriented● Not built over existing production quality libraries
    • Birds eye view of Mahout● If you want to: ○ Encode ○ Analyze ○ Predict ○ Get top best
    • about:encode● Process data and convert to vectors● Dictionary based v/s Randomizer based● Get best signals for generating vectors ○ Collocation information (ngrams) ○ Lp Normalization Data Engineering Camp 1:1.0 2:1.0 3:1.0
    • about:analyze● Cluster and group data to● Cluster data ○ K-Means ○ Fuzzy K-Means ○ Canopy ○ Mean Shift ○ Dirichlet process clustering ○ Spectral Clustering● Co-cluster features / dimensionality reduction ○ Latent Dirichlet Allocation (LDA) ○ Singular Value Decomposition
    • about:clusteringNews clusters
    • about:lda● Grouping similar or co-occurring features into a topic ○ Topic “Lol Cat”: ■ Cat ■ Meow ■ Purr ■ Haz ■ Cheeseburger ■ Lol
    • about:predict● Classification and Recommendation● Classification: ○ Use features learn model ○ Apply model on unknown● Recommendation ○ Use pairwise(user-item) information to learn model ○ For a given user return highly likely items
    • about:classify● Predicting the type of a new object based on its features● The types are predetermined Dog Cat
    • about:classify● Plenty of algorithms ○ Naïve Bayes ○ Complementary Naïve Bayes ○ Random Forests ○ Stocastic Gradient Descent(regression)● Learn a model from a manually classified data● Predict the class of a new object based on its features and the learned model
    • about:recommend● Predict what the user likes based on ○ His/Her historical behavior ○ Aggregate behavior of people similar to him
    • about:recommend● Different types of recommenders ○ User based ○ Item based ○ Co-occurrence based● Full framework for storage, online online and offline computation of recommendations● Like clustering, there is a notion of similarity in users or items ○ Cosine, Tanimoto, Pearson and LLR
    • about:top-n● Frequent Pattern Mining: ○ Identify top-K patterns
    • about:frequent-pattern-mining● Find interesting groups of items based on how they co-occur in a dataset
    • about:parallel-fp-growth● Identify the most commonly occurring patterns from ○ Sales Transactions buy “Milk, eggs and bread” ○ Query Logs ipad -> apple, tablet, iphone ○ Spam Detection Yahoo! http://www.slideshare.net/hadoopusergroup/mail- antispam
    • summary:in-short● Plenty of overlap. There is no one algorithm to fit all problems.● Analyze and iterate fast● MapReduce implementations makes these Fly!
    • Did you know?● Apache Mahout uses Colt high-performance collections ○ Open HashMaps instead of Chained HashMaps ○ Arrays of Primitive types ○ Available as Mahout Math library● Mahout Vector uses integer encoding techniques to reduce space.● Fastest classifier in Mahout doesn’t use MapReduce! ○ And it learns online ○ And It doesn’t look at all the data.
    • How to use mahout● Command line launcher bin/mahout● See the list of tools and algorithms by running bin/mahout● Run any algorithm by its shortname: ○ bin/mahout kmeans –help● By default runs locally● export HADOOP_HOME = /pathto/hadoop-0.20.2/ ○ Runs on the cluster configured as per the conf files in the hadoop directory● Use driver classes to launch jobs: ○ KMeansDriver.runjob(Path input, Path output …)
    • Clustering Walkthrough (tiny example)● Input: set of text files in a directory● Download Mahout and unzip ○ mvn install ○ bin/mahout seqdirectory –i <input> –o <seq-output> ○ bin/mahout seq2sparse –i seq-output –o <vector-output> ○ bin/mahout kmeans –i<vector-output> -c <cluster-temp> -o <cluster-output> -k 10 –cd 0.01 –x 20
    • Clustering Walkthrough (a bit more)● Use bigrams: -ng 2● Prune low frequency: –s 10● Normalize: -n 2● Use a distance measure : -dm org.apache.mahout.common. distance.CosineDistanceMeasure
    • Clustering Walkthrough (viewing results) ● bin/mahout clusterdump –s cluster-output/clusters-9/part-00000 -d vector-output/dictionary.file-* -dt sequencefile -n 5 -b 100 ● Top terms in a typical clustercomic => 9.793121272867376comics => 6.115341078151356con => 5.015090566692931sdcc => 3.927590843402978webcomics => 2.916910980686997
    • Road to Mahout v1.0● Guiding Principles ○ Use the stable Hadoop API ○ Make vector the de-factor input format for all parts of code ○ Provide stable API for developers
    • Get Started● http://mahout.apache.org● dev@mahout.apache.org - Developer mailing list● user@mahout.apache.org - User mailing list● Check out the documentations and wiki for quickstart● http://svn.apache.org/repos/asf/mahout/trunk/ Browse Code
    • Resources● “Mahout in Action” Owen, Anil, Dunning, Friedman http://www.manning.com/owen● “Taming Text” Ingersoll, Morton, Farris http://www.manning.com/ingersoll● “Introducing Apache Mahout” http://www.ibm.com/developerworks/java/library/j-mahout/
    • Thanks to● Apache Foundation● Mahout Committers● Google Summer of Code Organizers● And Students● Open source!● And NHN for hosting this at Seoul!● and the wonderful engineers (present and future) in the room.
    • References● news.google.com● Cat http://www.flickr.com/photos/gattou/3178745634/● Dog http://www.flickr.com/photos/30800139@N04/3879737638/● Milk Eggs Bread http://www.flickr. com/photos/nauright/4792775946/● Amazon Recommendations● twitter