Google knows what you want to search for, Amazon what you want to buy, and Facebook knows who your friends are before you connect to them. Are they reading your minds? the answer is yes. In this session I'll teach you how to use Mahout, an Apache machine learning library, and read the minds of your users based on the information you have already gathered. And if your dataset is too big, we'll use Hadoop to analyze it on a distributed environment. http://www.javaedge.com/jedge/abstracts.jsp

1. Mahout By: Ariel Kogan

2. Java Framework Team on IDI 10 years of experience on IT 6 years of experience on Java Masters in Informatics Engineering specializing on Artificial Intelligence Has a weird accent Who’s this guy? Aliyah http://www.flickr.com/photos/triphenawong/4752510292/

3. Machine Learning Mahout Recommender Engines Clustering Categorization Hadoop Agenda

4. Machine Learning

5. Machine Learning Whatchatalkin' 'bout, Willis?

6. Recommender Engines Clustering Classification Well known use cases for: Machine Learning

7. Machine Learning Recommender Engines: Amazon

8. Machine Learning Recommender Engines: Facebook

9. Machine Learning Clustering: Google News

10. Machine Learning Classification: Spam Detection

11. Machine Learning Classification: Picasa face recognition

12. Because it’s interesting Because it makes money Why learning “Machine Learning”? Machine Learning

13. Mahout

14. Open Source project by the Apache Software Foundation Goal: To build scalable machine learning libraries. Large data sets (Hadoop) Commercially friendly Apache Software license Community What is it? Mahout

15. Mahout - [muh-hout] - (mə’haʊt) A mahout is a person who keeps and drives an elephant. The name Mahout comes from the project's use of Apache Hadoop — which has a yellow elephant as its logo — for scalability and fault tolerance. What’s that name? Mahout

16. Mahout Mahout and its related projects

17. Mahout History

18. Mahout History Mahout is presented on AlphaCSP’s The Edge 2010 Taste Collaborative Filtering has donated it's codebase to the Mahout project Release 0.1 Release 0.2 Release 0.3 Release 0.4 2010 2008 2009 The Lucene Project Management Committee announces the creation of the Mahout subproject Mahout becomes an Apache top level project

19. Mahout Mailing lists activity

20. Weka (since 1999) 38 Java projects listed on mloss.org (Machine Learning Open Source Software) Yet another Framework? Similar Products Mahout

21. Large amount of input data Techniques work better Nature of the deploying context Must produce results quickly The amount of input is so large that it is not feasible to process it all on one computer, even a powerful one Machine Learning Challenges Mahout

22. Mahout core algorithms are implemented on top of Apache Hadoop using the map/reduce paradigm. Scalability Mahout

23. Programming model introduced by Google in 2004 Many real world tasks are expressible in this model (“Map-Reduce for Machine Learning on Multicore”, Stanford CS Department’s paper, 2006) Provides automatic parallelization and distribution Runs on large clusters of compute nodes Highly scalable Hadoop is Apache’s open source implementation MapReduce Mahout

24. Mahout

25. Mahout

26. Recommender Engines

27. Approaches User based Item based Collaborative filtering vs Content-based recommendation Recommender Engines

28. Data model Users Items Preferences (ratings) ItemSimilarity UserSimilarity UserNeighborhood Recommender What do we need? Recommender Engines

29. Recommender Engines T-bone Chocolate Lettuce Rump http://www.flickr.com/photos/martinimike/3770274175/ http://www.flickr.com/photos/fotoosvanrobin/3182238046/ http://www.flickr.com/photos/this_girl_daydreams/3190110968/ http://www.flickr.com/photos/19998197@N00/3238445535/

30. Recommender Engines 5 -5

31. Recommender Engines Kuki The Vegan Gilad Ariel

32. Recommender Engines // We create a DataModel based on the information contained on food.csv DataModel model = newFileDataModel(new File(“food.csv")); // We use one of the several user similarity functions we have available UserSimilarity similarity = newPearsonCorrelationSimilarity(model); // Same thing with the UserNeighborhood definition UserNeighborhood neighborhood = newNearestNUserNeighborhood(hoodSize, similarity, model); // Finally we can build or recommender Recommender recommender = newGenericUserBasedRecommender(model, neighborhood, similarity); // And ask for recommendations for a specific user List<RecommendedItem> recommendations = recommender.recommend(userId, howMany); for (RecommendedItem recommendation : recommendations) { System.out.println(recommendation); } CachingUserSimilarity EuclideanDistanceSimilarity GenericUserSimilarity LogLikelihoodSimilarity PearsonCorrelationSimilarity SpearmanCorrelationSimilarity TanimotoCoefficientSimilarity UncenteredCosineSimilarity

33. Recommender Engines What would we recommend to Ariel? T-bone rating 4.0 Recommendation for Ariel

34. Recommender Engines Kuki The Vegan Gilad Ariel

35. 10 most popular Random selection What other customers are looking at right now Bestsellers Best prices Nothing at all No initial information Recommender Engines

36. Clustering

37. Clustering is about drawing lines Clustering

38. Clustering Clustering Steps

39. Possible weather conditions recognition Clustering CLUSTERING temperature wind direction humidity wind speed http://www.icons-land.com

40. Clustering Vector representation 25 / 50 = 0.5

41. Clustering Samples Generation 300 samples Mean: [0.0, 2.0] SD: 0.1 500 samples Mean: [1.0, 1.0] SD: 3.0 300 samples Mean: [1.0, 0.0] SD: 0.5

42. Clustering Iterations with Fuzzy K-Means

43. Clustering Clustering Discovery Original data generation Discovered clusters

44. Clustering CosineDistanceMeasure EuclideanDistanceMeasure MahalanobisDistanceMeasure ManhattanDistanceMeasure SquaredEuclideanDistanceMeasure TanimotoDistanceMeasure WeightedDistanceMeasure WeightedEuclideanDistanceMeasure WeightedManhattanDistanceMeasure

45. Categorization

46. Categorization Categorization Steps

47. Our example: What do we want to do? Categorization Java Classifier Document Sport

48. Categorization Documents Preparation Label <tab> evidence1 <space> evidence2 BayesFileFormatter (Lucene’s Analyzers) Labeled Documents Training Test

49. Categorization Using the classifier

50. Categorization Categorization testing, the confusion matrix Summary ------------------------------------------------------- Correctly Classified Instances : 93 93% Incorrectly Classified Instances : 7 7% Total Classified Instances : 100 ======================================================= Confusion Matrix ------------------------------------------------------- java sport <--Classified as 56 3 | 59 java 4 37 | 41 sport

51. Take me to the cluster

52. The size of our dataset can’t be handled by a single machine. Scale-up vs scale-out. We need the results on nearly real time. Why do we need distributed computing? Hadoop

53. Hadoop Data Results Hadoop Compute Cluster

54. We need to: Configure the job Submit it Control its execution Query its state We want to: Just run our machine learning algorithm! Hadoop Jobs Hadoop

55. Mahout provides an out of the box AbstractJob class and several Jobs and Drivers implementations in order to run Machine Learning algorithms on the cluster without any hassle. Mahout’s AbstractJob and Drivers Hadoop

56. Our code, including a Job Mahout jars Hadoop jars Everyone’s dependencies jars Resources The dataset What we need Hadoop

57. Hadoop Packaging a Job – The Maven solution pom.xml

58. Hadoop Job feeding Job Dataset Hadoop Compute Cluster

59. Hadoop We take the project’s dependencies

60. Hadoop Using an Ant task, we pack everything together

61. Hadoop We attach the Job jar on the packaging phase

62. Hadoop Running our Job Upload our job to the HDFS $ hadoopfs -put myjob.job /myjob.job $ hadoopfs -put dataset.dat /dataset.dat $ hadoop jar /myjob.jobc.a.RecommenderJob/dataset.dat/output.dat Upload the dataset to the HDFS jar class input output Run the job

63. Our dataset is too big and we need the results fast. Mahout gives us out of the box all we need to run on Hadoop. We can pack all together with Maven. Our Machine Learning algorithms are running on the cluster! Summary Hadoop

64. Mahout’s website Mahout in Action – May 2011 (est.) Introducing Apache Mahout @ IBM developerWorks References

65. Thank You Any questions? We appreciate your feedback