Introduction to Mahout Machine Learning with HDInsight

MAKING BUSINESS INTELLIGENT
www.pragmaticworks.com
Introduction to Mahout
with HDInsight (Hadoop)
Chris Price
Senior BI Consultant
@BluewaterSQL

Intro
Chris Price
Senior BI Consultant with Pragmatic Works
Author
Regular Speaker
Data Geek & Super Dad!
@BluewaterSQL
http://bluewatersql.wordpress.com/
cprice@pragmaticworks.com

Survey
 Whose currently using Machine Learning?
 Google
 Facebook
 LinkedIn
 Twitter
 Amazon
 Wal-Mart

Outline
 Mahout Introduction
 The Algorithms
 Hands On:
 A recommendation engine

Riding the Elephant
 Born out of the Apache Lucene project
 Top-level Apache project
 A scalable machine learning library
 Fast, Efficient & Pragmatic
 Many of the algorithms can be run on Hadoop

Algorithms
 Collaborative Filtering
 Item/User Recommenders
 Clustering
 Grouping movies by type
 Classification
 Categorizing documents
 Frequent Itemset
 Market basket analysis

Collaborative Filtering
 Find subset of users who have similar
taste/preferences to target user and use this
subset for recommendations
 Types:
 User-Based
 Item-Based
 Examples:
 Amazon

Clustering
 Group similar objects
 Examples:
 News Aggregator
 Customer Grouping

Clustering
 Algorithms:
 K-Means
 Fuzzy K-Means
 Mean Shift
 Canopy
 Dirichlet
 Similarity Distance:
 Euclidean
 Squared Euclidean
 Cosine
 Tanimoto
 Manhattan
** Also weighted measures

Clustering

Classification
 Using a pre-determined set of groups:
 Predict the type of a new object based on its
features
 Classifiable Data
 Continuous – Quantitative Value (i.e. Stock Price)
 Categorical – Small known set (i.e. Colors)
 Word-Like – Large unknown set
 Text-Like – Many word-like that are unordered
 Examples:
 Spam Identification
 Photo Facial Recognition

Frequent Itemset
 Examples:
 Product Placement
 Market Basket Analysis
 Query Recommendations

Mahout on HDInsight
 Installation
 Download
 http://www.apache.org/dyn/closer.cgi/mahout/
 Unpack
 Add to Path (Environment Variable)

Recommendation Engine
 Define the Business Objective
 Metrics
 Context
 Identify Data
 Sources
 Normalization
 Data Shift
 Which Algorithm?
 Integration?

Business Objective
Navigational
Inefficiency
Cross-Sell
Up-Sell
Increase #
of Orders
Increase Items
per Order
Increase
Average
Item Price
Website
Increase
Revenue

Handling Context
???
January
20 degrees & Snowing…..

Data Acquisition
 Sources of Data for Recommendation
 Implicit
 Ratings
 Feedback
 Demographics
 Pyschographics (Personality/Lifestyle/Attitude),
 Ephemeral Need (Need for a moment)
 Explicit
 Purchase History
 Click/Browse History
 Product/Item
 Taxonomy
 Attributes
 Descriptions

Data Preparation
 Preparation
 Remove Outliers (Z-Score)
 Remove frequent buyers (Skew)
 Normalize Data (Unity-Based)
 Beware of Data Shift

Algorithms
 Collaborative Filtering (Mahout)
 User-Based
 Item-Based
 Content-Based (Mahout Clustering)
 Data Mining (SSAS)
 Association
 Clustering

CF Recommendations
 Neighborhood Formation
 Similarity Metrics
 Pearson Correlation
 Euclidean Distance
 Spearman Correlation
 Cosine
 Tanimoto Coefficient
 Log-Likelihood

CF Pseudo-Code
for each item i that u has no preference
for each user v that has a preference for i
compute similarity s between u and v
calculate running average of v‘s
preference for i, weighted by s
return top ranked (weighted average) i
Restrict to Neighborhood

Testing
 Smell Test
 Built-In (Requires Java Coding)
 Root Mean Squared Error (RMSE)
 Average Absolute Difference
RandomUtils.useTestSeed()
Evaluator.evaluate(builder,null,0.7,1.0)

Recommendation Engine Steps
 1 – Generate List of ItemIDs
 2 – Create Preference Vector
 3 – Count Unique Users
 4 – Transpose Preference Vectors
 5 – Row Similarity
 Compute Weights
 Computer Similarities
 Similarity Matrix
 6 – Pre-Partial Multiply, Similarity Matrix
 7 – Pre-Partial Multiply, Preferences
 8 – Partial Multiple (Steps 6 & 7)
 9 – Filter Items
 10 – Aggregate & Recommend

Batch Integration
 ETL Data to HDFS
 SSIS
 Map/Reduce
 Process with Mahout
 ETL Results
 Map/Reduce
 Hive/Sqoop

Hands-On Demo

Resources
 Mahout in Action
Sean Owen, Robin Anil,
Ted Dunning, Ellen Friedman
 Hadoop: The Definitive Guide
 Tom White

www.pragmaticworks.comMAKING BUSINESS INTELLIGENT
Thank you!
@BluewaterSQL
http://bluewatersql.wordpress.com/
cprice@pragmaticworks.com

Introduction to Mahout Machine Learning with HDInsight

Recommended

Recommended

More Related Content

What's hot

What's hot (9)

Viewers also liked

Viewers also liked (6)

Similar to Introduction to Mahout Machine Learning with HDInsight

Similar to Introduction to Mahout Machine Learning with HDInsight (20)

Recently uploaded

Recently uploaded (20)

Introduction to Mahout Machine Learning with HDInsight

Editor's Notes