Datameer, profile picture
Uploaded byDatameer
1,804 views

How to do Predictive Analytics with Limited Data

The document discusses methods for predictive analytics using limited data, emphasizing semi-supervised learning and various modeling techniques like clustering, generative models, and support vector machines. It explores the assumptions underlying these methods, such as clustering, low density, and manifold assumptions, as well as practical implementations like label propagation and stochastic gradient descent in Hadoop. The document aims to provide insights on leveraging both labeled and unlabeled data to improve predictive model performance.

Embed presentation

How to do Predictive Analytics with Limited Data Ulrich Rueckert Š 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
Agenda Introduction and Motivation Semi-Supervised Learning • Generative Models • Large Margin Approaches • Similarity Based Methods Conclusion © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
Predictive Modeling Traditional Supervised Learning • • • • • Promotion on bookseller’s web page Customers can rate books. Will a new customer like this book? Training set: observations on previous customers Test set: new customers What happens if only few customers rate a book? © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13 Test Data Age Attributes Income 60K 80K 95K 52K 45K 75K 51K + + + 52 47K - 47 25 38K 22K - 33 47K ? + 20 43 26 41K - 35 ? LikesBook + 65 60 67K 39 Age 24 LikesBook 22 Target Label Income + Training Data Model Age 22 Income 67K LikesBook + 39 41K - Prediction
Predictive Modeling Test Data Traditional Supervised Learning • • • • • Age Test set: new customers What happens if only few customers rate a book? 41K ? 95K 52K ? ? 20 45K ? 43 75K ? 26 52 51K 47K ? ? 47 38K ? 25 22K ? 33 Training set: observations on previous customers ? 60 35 Will a new customer like this book? 67K 39 Customers can rate books. LikesBook 22 Promotion on bookseller’s web page Income 47K ? Target Label Attributes Age Income LikesBook 24 60K + 65 80K - Model Training Data Prediction © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
Predictive Modeling Traditional Supervised Learning • • • • • Promotion on bookseller’s web page Customers can rate books. Will a new customer like this book? Training set: observations on previous customers Test set: new customers What happens if only few customers rate a book? © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13 Test Data Age Attributes Income 60K 80K 95K 52K 45K 75K 51K ? + ? 52 47K - 47 25 38K 22K ? ? 33 47K ? ? 20 43 26 41K ? 35 ? LikesBook + 65 60 67K 39 Age 24 LikesBook 22 Target Label Income ? Training Data Model Age 22 Income 67K LikesBook + 39 41K - Prediction
Supervised Learning Classification • For now: each data instance is a point in a 2D coordinate system • • Color denotes target label Model is given as decision boundary What’s the correct model? • • In theory: no way to tell • All learning systems have underlying assumptions Smoothness assumption: similar instances have similar labels © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
Supervised Learning Classification • For now: each data instance is a point in a 2D coordinate system • • Color denotes target label Model is given as decision boundary What’s the correct model? • • In theory: no way to tell • All learning systems have underlying assumptions Smoothness assumption: similar instances have similar labels © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
Supervised Learning Classification • For now: each data instance is a point in a 2D coordinate system • • Color denotes target label Model is given as decision boundary What’s the correct model? • • In theory: no way to tell • All learning systems have underlying assumptions Smoothness assumption: similar instances have similar labels © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
Supervised Learning Classification • For now: each data instance is a point in a 2D coordinate system • • Color denotes target label Model is given as decision boundary What’s the correct model? • • In theory: no way to tell • All learning systems have underlying assumptions Smoothness assumption: similar instances have similar labels © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
Supervised Learning Classification • For now: each data instance is a point in a 2D coordinate system • • Color denotes target label Model is given as decision boundary What’s the correct model? • • In theory: no way to tell • All learning systems have underlying assumptions Smoothness assumption: similar instances have similar labels © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
Semi-Supervised Learning Can we make use of the unlabeled data? • • In theory: no ... but we can make assumptions Popular Assumptions • • • Clustering assumption Low density assumption Manifold assumption © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
Semi-Supervised Learning Can we make use of the unlabeled data? • • In theory: no ... but we can make assumptions Popular Assumptions • • • Clustering assumption Low density assumption Manifold assumption © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
The Clustering Assumption Clustering • Partition instances into groups (clusters) of similar instances • Many different algorithms: k-Means, EM, DBSCAN, etc. • Available e.g. on Mahout Clustering Assumption • The two classification targets are distinct clusters • Simple semi-supervised learning: cluster, then perform majority vote © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
The Clustering Assumption Clustering • Partition instances into groups (clusters) of similar instances • Many different algorithms: k-Means, EM, DBSCAN, etc. • Available e.g. on Mahout Clustering Assumption • The two classification targets are distinct clusters • Simple semi-supervised learning: cluster, then perform majority vote © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
The Clustering Assumption Clustering • Partition instances into groups (clusters) of similar instances • Many different algorithms: k-Means, EM, DBSCAN, etc. • Available e.g. on Mahout Clustering Assumption • The two classification targets are distinct clusters • Simple semi-supervised learning: cluster, then perform majority vote © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
The Clustering Assumption Clustering • Partition instances into groups (clusters) of similar instances • Many different algorithms: k-Means, EM, DBSCAN, etc. • Available e.g. on Mahout Clustering Assumption • The two classification targets are distinct clusters • Simple semi-supervised learning: cluster, then perform majority vote © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
Generative Models Mixture of Gaussians • Assumption: the data in each cluster is generated by a normal distribution • Find most probable location and shape of clusters given data Expectation-Maximization • • Two step optimization procedure • • Each step is one MapReduce job Keeps estimates of cluster assignment probabilities for each instance Might converge to local optimum © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
Generative Models Mixture of Gaussians • Assumption: the data in each cluster is generated by a normal distribution • Find most probable location and shape of clusters given data Expectation-Maximization • • Two step optimization procedure • • Each step is one MapReduce job Keeps estimates of cluster assignment probabilities for each instance Might converge to local optimum © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
Generative Models Mixture of Gaussians • Assumption: the data in each cluster is generated by a normal distribution • Find most probable location and shape of clusters given data Expectation-Maximization • • Two step optimization procedure • • Each step is one MapReduce job Keeps estimates of cluster assignment probabilities for each instance Might converge to local optimum © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
Generative Models Mixture of Gaussians • Assumption: the data in each cluster is generated by a normal distribution • Find most probable location and shape of clusters given data Expectation-Maximization • • Two step optimization procedure • • Each step is one MapReduce job Keeps estimates of cluster assignment probabilities for each instance Might converge to local optimum © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
Generative Models Mixture of Gaussians • Assumption: the data in each cluster is generated by a normal distribution • Find most probable location and shape of clusters given data Expectation-Maximization • • Two step optimization procedure • • Each step is one MapReduce job Keeps estimates of cluster assignment probabilities for each instance Might converge to local optimum © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
Generative Models Mixture of Gaussians • Assumption: the data in each cluster is generated by a normal distribution • Find most probable location and shape of clusters given data Expectation-Maximization • • Two step optimization procedure • • Each step is one MapReduce job Keeps estimates of cluster assignment probabilities for each instance Might converge to local optimum © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
Beyond Mixtures of Gaussians Expectation-Maximization • • Can be adjusted to all kinds of mixture models E.g. use Naive Bayes as mixture model for text classification Self-Training • • • • Learn model on labeled instances only Apply model to unlabeled instances Learn new model on all instances Repeat until convergence © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
The Low Density Assumption Assumption • The area between the two classes has low density • Does not assume any specific form of cluster Support Vector Machine • • • Decision boundary is linear Maximizes margin to closest instances Can be learned in one Map-Reduce step (Stochastic Gradient Descent) © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
The Low Density Assumption Assumption • The area between the two classes has low density • Does not assume any specific form of cluster Support Vector Machine • • • Decision boundary is linear Maximizes margin to closest instances Can be learned in one Map-Reduce step (Stochastic Gradient Descent) © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
The Low Density Assumption Assumption • The area between the two classes has low density • Does not assume any specific form of cluster Support Vector Machine • • • Decision boundary is linear Maximizes margin to closest instances Can be learned in one Map-Reduce step (Stochastic Gradient Descent) © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
The Low Density Assumption Semi-Supervised Support Vector Machine • Minimize distance to labeled and unlabeled instances • Parameter to fine-tune influence of unlabeled instances • Additional constraint: keep class balance correct Implementation • • Simple extension of SVM But non-convex optimization problem © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
The Low Density Assumption Semi-Supervised Support Vector Machine • Minimize distance to labeled and unlabeled instances • Parameter to fine-tune influence of unlabeled instances • Additional constraint: keep class balance correct Implementation • • Simple extension of SVM But non-convex optimization problem © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
The Low Density Assumption Semi-Supervised Support Vector Machine • Minimize distance to labeled and unlabeled instances • Parameter to fine-tune influence of unlabeled instances • Additional constraint: keep class balance correct Implementation • • Simple extension of SVM But non-convex optimization problem © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
The Low Density Assumption Semi-Supervised Support Vector Machine • • • Minimize distance to labeled and unlabeled instances Parameter to fine-tune influence of unlabeled instances Additional constraint: keep class balance correct Implementation • • Margins of labeled instances Regularizer 2 min kwk w +C +C n X `(yi (wT xi + b)) i=1 n+m X ? `(|wT xi + b|) i=n+1 Simple extension of SVM But non-convex optimization problem © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13 Margins of unlabeled instances
Semi-Supervised SVM Stochastic Gradient Descent • • One run over the data in random order • Steps get smaller over time Each misclassified or unlabeled instance moves classifier a bit Implementation on Hadoop • Mapper: send data to reducer in random order • Reducer: update linear classifier for unlabeled or misclassified instances • Many random runs to find best one © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
Semi-Supervised SVM Stochastic Gradient Descent • • One run over the data in random order • Steps get smaller over time Each misclassified or unlabeled instance moves classifier a bit Implementation on Hadoop • Mapper: send data to reducer in random order • Reducer: update linear classifier for unlabeled or misclassified instances • Many random runs to find best one © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
Semi-Supervised SVM Stochastic Gradient Descent • • One run over the data in random order • Steps get smaller over time Each misclassified or unlabeled instance moves classifier a bit Implementation on Hadoop • Mapper: send data to reducer in random order • Reducer: update linear classifier for unlabeled or misclassified instances • Many random runs to find best one © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
Semi-Supervised SVM Stochastic Gradient Descent • • One run over the data in random order • Steps get smaller over time Each misclassified or unlabeled instance moves classifier a bit Implementation on Hadoop • Mapper: send data to reducer in random order • Reducer: update linear classifier for unlabeled or misclassified instances • Many random runs to find best one © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
Semi-Supervised SVM Stochastic Gradient Descent • • One run over the data in random order • Steps get smaller over time Each misclassified or unlabeled instance moves classifier a bit Implementation on Hadoop • Mapper: send data to reducer in random order • Reducer: update linear classifier for unlabeled or misclassified instances • Many random runs to find best one © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
The Manifold Assumption The Assumption • Training data is (roughly) contained in a low dimensional manifold • One can perform learning in a more meaningful low-dimensional space • Avoids curse of dimensionality Similarity Graphs • Idea: compute similarity scores between instances • Create network where the nearest neighbors are connected © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
The Manifold Assumption The Assumption • Training data is (roughly) contained in a low dimensional manifold • One can perform learning in a more meaningful low-dimensional space • Avoids curse of dimensionality Similarity Graphs • Idea: compute similarity scores between instances • Create network where the nearest neighbors are connected © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
The Manifold Assumption The Assumption • Training data is (roughly) contained in a low dimensional manifold • One can perform learning in a more meaningful low-dimensional space • Avoids curse of dimensionality Similarity Graphs • Idea: compute similarity scores between instances • Create network where the nearest neighbors are connected © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
The Manifold Assumption The Assumption • Training data is (roughly) contained in a low dimensional manifold • One can perform learning in a more meaningful low-dimensional space • Avoids curse of dimensionality Similarity Graphs • Idea: compute similarity scores between instances • Create network where the nearest neighbors are connected © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
Label Propagation Main Idea • Propagate label information to neighboring instances • • Then repeat until convergence Similar to PageRank Theory • Known to converge under weak conditions • Equivalent to matrix inversion © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
Label Propagation Main Idea • Propagate label information to neighboring instances • • Then repeat until convergence Similar to PageRank Theory • Known to converge under weak conditions • Equivalent to matrix inversion © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
Label Propagation Main Idea • Propagate label information to neighboring instances • • Then repeat until convergence Similar to PageRank Theory • Known to converge under weak conditions • Equivalent to matrix inversion © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
Label Propagation Main Idea • Propagate label information to neighboring instances • • Then repeat until convergence Similar to PageRank Theory • Known to converge under weak conditions • Equivalent to matrix inversion © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
Nearest Neighbor Join Block Nested Loop Join • • 1st MR job: partition data into blocks, compute nearest neighbors between blocks 2nd MR job: filter out the overall nearest neighbors Reducers Use spatial information to avoid unnecessary comparisons • R trees, space filling curves, locality sensitive hashing • Example implementations available online © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13 1 1 1 2 1 3 Smarter Approaches • Data 1 4 1 1 1 1
Nearest Neighbor Join Block Nested Loop Join • • 1st MR job: partition data into blocks, compute nearest neighbors between blocks 2nd MR job: filter out the overall nearest neighbors Reducers Use spatial information to avoid unnecessary comparisons • R trees, space filling curves, locality sensitive hashing • Example implementations available online © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13 1 1 1 1 2 1 1 2 2 1 2 2 2 2 3 Smarter Approaches • Data 1 2 4 1 2
Nearest Neighbor Join Block Nested Loop Join • • 1st MR job: partition data into blocks, compute nearest neighbors between blocks 2nd MR job: filter out the overall nearest neighbors Reducers Use spatial information to avoid unnecessary comparisons • R trees, space filling curves, locality sensitive hashing • Example implementations available online © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13 1 1 1 1 2 1 3 1 4 2 2 1 2 2 2 3 2 4 3 Smarter Approaches • Data 3 1 3 2 3 3 3 4 4 4 1 4 2 4 3 4 4
Nearest Neighbor Join Block Nested Loop Join • • 1st MR job: partition data into blocks, compute nearest neighbors between blocks 2nd MR job: filter out the overall nearest neighbors Smarter Approaches • Data 1 2 3 Use spatial information to avoid unnecessary comparisons • R trees, space filling curves, locality sensitive hashing • Example implementations available online © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13 4 Nearest Neighbors
Nearest Neighbor Join Block Nested Loop Join • • 1st MR job: partition data into blocks, compute nearest neighbors between blocks 2nd MR job: filter out the overall nearest neighbors Smarter Approaches • Data 1 2 3 Use spatial information to avoid unnecessary comparisons • R trees, space filling curves, locality sensitive hashing • Example implementations available online © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13 4 Nearest Neighbors Result
Nearest Neighbor Join Block Nested Loop Join • • 1st MR job: partition data into blocks, compute nearest neighbors between blocks 2nd MR job: filter out the overall nearest neighbors Smarter Approaches • Data 1 1 1 1 2 2 2 1 2 2 2 3 3 2 3 3 3 4 4 3 4 4 3 Use spatial information to avoid unnecessary comparisons • R trees, space filling curves, locality sensitive hashing • Example implementations available online © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13 Reducers 4
Nearest Neighbor Join Block Nested Loop Join • 1st MR job: partition data into blocks, compute nearest neighbors between blocks • 2nd MR job: filter out the overall nearest neighbors Smarter Approaches • Use spatial information to avoid unnecessary comparisons • R trees, space filling curves, locality sensitive hashing • Example implementations available online © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
Nearest Neighbor Join Block Nested Loop Join • 1st MR job: partition data into blocks, compute nearest neighbors between blocks • 2nd MR job: filter out the overall nearest neighbors Smarter Approaches • Use spatial information to avoid unnecessary comparisons • R trees, space filling curves, locality sensitive hashing • Example implementations available online © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
Label Propagation on Hadoop Iterative Procedure • • Mapper: Emit neighbor-label pairs • Repeat until convergence Reducer: Collect incoming labels per node and combine them into new label Improvement • Cluster data and perform within-cluster propagation locally • Sometimes it’s more efficient to perform matrix inversion instead © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
Label Propagation on Hadoop Iterative Procedure • • Mapper: Emit neighbor-label pairs • Repeat until convergence Reducer: Collect incoming labels per node and combine them into new label Improvement • Cluster data and perform within-cluster propagation locally • Sometimes it’s more efficient to perform matrix inversion instead © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
Label Propagation on Hadoop Iterative Procedure • • Mapper: Emit neighbor-label pairs • Repeat until convergence Reducer: Collect incoming labels per node and combine them into new label Improvement • Cluster data and perform within-cluster propagation locally • Sometimes it’s more efficient to perform matrix inversion instead © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
Label Propagation on Hadoop Iterative Procedure • • Mapper: Emit neighbor-label pairs • Repeat until convergence Reducer: Collect incoming labels per node and combine them into new label Improvement • Cluster data and perform within-cluster propagation locally • Sometimes it’s more efficient to perform matrix inversion instead © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
Label Propagation on Hadoop Iterative Procedure • • Mapper: Emit neighbor-label pairs • Repeat until convergence Reducer: Collect incoming labels per node and combine them into new label Improvement • Cluster data and perform within-cluster propagation locally • Sometimes it’s more efficient to perform matrix inversion instead © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
Conclusion Semi-Supervised Learning • Only few training instances have labels • Unlabeled instances can still provide valuable signal Different assumptions lead to different approaches • • • Cluster assumption: generative models Low density assumption: semi-supervised support vector machines Manifold assumption: label propagation Code available: https://github.com/Datameer-Inc/lesel © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
@Datameer Thursday, October 31, 13

More Related Content

PDF
Social Network Analysis (SNA) Made Easy
byJeff Mohr
 
PPTX
Recommender Systems: Advances in Collaborative Filtering
byChangsung Moon
 
PDF
Putting the Magic in Data Science
bySean Taylor
 
PDF
Streaming Outlier Analysis for Fun and Scalability
byDataWorks Summit/Hadoop Summit
 
PPTX
Using the search engine as recommendation engine
byLars Marius Garshol
 
PDF
Recommender Systems
byT212
 
PDF
Like a Pack of Wolves: Community Structure of Web Trackers
byVasia Kalavri
 
PDF
Physics Inspired Approaches to Community Detection
byfuzzysphere
 
Social Network Analysis (SNA) Made Easy
byJeff Mohr
 
Recommender Systems: Advances in Collaborative Filtering
byChangsung Moon
 
Putting the Magic in Data Science
bySean Taylor
 
Streaming Outlier Analysis for Fun and Scalability
byDataWorks Summit/Hadoop Summit
 
Using the search engine as recommendation engine
byLars Marius Garshol
 
Recommender Systems
byT212
 
Like a Pack of Wolves: Community Structure of Web Trackers
byVasia Kalavri
 
Physics Inspired Approaches to Community Detection
byfuzzysphere
 

Viewers also liked

PDF
Community detection in graphs
byNicola Barbieri
 
PDF
Label propagation - Semisupervised Learning with Applications to NLP
byDavid Przybilla
 
PDF
Big data matrix factorizations and Overlapping community detection in graphs
byDavid Gleich
 
PDF
Community Detection in Social Media
bySymeon Papadopoulos
 
PPTX
Analysis of Overlapping Communities in Signed Complex Networks
byMohsen Shahriari
 
PPTX
Language of Politics on Twitter - 03 Analysis
byYelena Mejova
 
PDF
MOSES: Community finding using Model-based Overlapping Seed ExpanSion
byaaronmcdaid
 
Community detection in graphs
byNicola Barbieri
 
Label propagation - Semisupervised Learning with Applications to NLP
byDavid Przybilla
 
Big data matrix factorizations and Overlapping community detection in graphs
byDavid Gleich
 
Community Detection in Social Media
bySymeon Papadopoulos
 
Analysis of Overlapping Communities in Signed Complex Networks
byMohsen Shahriari
 
Language of Politics on Twitter - 03 Analysis
byYelena Mejova
 
MOSES: Community finding using Model-based Overlapping Seed ExpanSion
byaaronmcdaid
 

Similar to How to do Predictive Analytics with Limited Data

PPTX
AI-900 - Fundamental Principles of ML.pptx
bykprasad8
 
PPTX
Introduction to Machine Learning
byRahul Jain
 
PPT
3 DM Classification HFCS kilometres .ppt
byabinettemesgen450
 
PPTX
Model Development And Evaluation in ML.pptx
bybismayabaliarsingh00
 
PPTX
Introduction to Machine Learning
bySujith Jayaprakash
 
PDF
Intro to machine learning
byTamir Taha
 
PPTX
Data Mining - The Big Picture!
byKhalid Salama
 
PPTX
Big Data Analytics - Unit 3.pptx
byPlacementsBCA
 
PPT
126622gghyytgggffggggggfsssssssssdff70.ppt
byShahidHussain66691
 
PDF
An Introduction to Supervised Machine Learning and Pattern Classification: Th...
bySebastian Raschka
 
PDF
Getting Started with Machine Learning
byHumberto Marchezi
 
PDF
Machine Learning - Algorithms and simple business cases
byClaudio Mirti
 
PDF
Machine Learning Session by Artivatic AI Data Labs
byArtivatic.ai
 
PPTX
Best practices machine learning final
byDianna Doan
 
PDF
لموعد الإثنين 03 يناير 2022 143 مبادرة #تواصل_تطوير المحاضرة ال 143 من المباد...
byEgyptian Engineers Association
 
PDF
IRJET- Study and Evaluation of Classification Algorithms in Data Mining
byIRJET Journal
 
PPTX
Supervised learning
byJohnson Ubah
 
PPTX
Search Engines
bybutest
 
PDF
Introduction to Big Data Science
byAlbert Bifet
 
PDF
AI
byvenkatvajradhar1
 
AI-900 - Fundamental Principles of ML.pptx
bykprasad8
 
Introduction to Machine Learning
byRahul Jain
 
3 DM Classification HFCS kilometres .ppt
byabinettemesgen450
 
Model Development And Evaluation in ML.pptx
bybismayabaliarsingh00
 
Introduction to Machine Learning
bySujith Jayaprakash
 
Intro to machine learning
byTamir Taha
 
Data Mining - The Big Picture!
byKhalid Salama
 
Big Data Analytics - Unit 3.pptx
byPlacementsBCA
 
126622gghyytgggffggggggfsssssssssdff70.ppt
byShahidHussain66691
 
An Introduction to Supervised Machine Learning and Pattern Classification: Th...
bySebastian Raschka
 
Getting Started with Machine Learning
byHumberto Marchezi
 
Machine Learning - Algorithms and simple business cases
byClaudio Mirti
 
Machine Learning Session by Artivatic AI Data Labs
byArtivatic.ai
 
Best practices machine learning final
byDianna Doan
 
لموعد الإثنين 03 يناير 2022 143 مبادرة #تواصل_تطوير المحاضرة ال 143 من المباد...
byEgyptian Engineers Association
 
IRJET- Study and Evaluation of Classification Algorithms in Data Mining
byIRJET Journal
 
Supervised learning
byJohnson Ubah
 
Search Engines
bybutest
 
Introduction to Big Data Science
byAlbert Bifet
 
AI
byvenkatvajradhar1
 

More from Datameer

PDF
Analyzing Unstructured Data in Hadoop Webinar
byDatameer
 
PDF
The Economics of SQL on Hadoop
byDatameer
 
PDF
How to Avoid Pitfalls in Big Data Analytics Webinar
byDatameer
 
PDF
Getting Started with Big Data for Business Managers
byDatameer
 
PDF
Webinar - Big Data: Power to the User
byDatameer
 
PDF
BI, Hive or Big Data Analytics?
byDatameer
 
PDF
Customer Case Studies of Self-Service Big Data Analytics
byDatameer
 
PDF
Complement Your Existing Data Warehouse with Big Data & Hadoop
byDatameer
 
PDF
Extending BI with Big Data Analytics
byDatameer
 
PDF
Fight Fraud with Big Data Analytics
byDatameer
 
PDF
Instant Visualizations in Every Step of Analysis
byDatameer
 
PPTX
Datameer6 for prospects - june 2016_v2
byDatameer
 
PDF
Lean Production Meets Big Data: A Next Generation Use Case
byDatameer
 
PDF
Online Fraud Detection Using Big Data Analytics Webinar
byDatameer
 
PDF
Understand Your Customer Buying Journey with Big Data
byDatameer
 
PPTX
Is Your Hadoop Environment Secure?
byDatameer
 
PDF
Why Use Hadoop?
byDatameer
 
PDF
Why Use Hadoop for Big Data Analytics?
byDatameer
 
PDF
Webinar - Introducing Datameer 4.0: Visual, End-to-End
byDatameer
 
PDF
The State of Big Data Adoption: A Glance at Top Industries Adopting Big Data ...
byDatameer
 
Analyzing Unstructured Data in Hadoop Webinar
byDatameer
 
The Economics of SQL on Hadoop
byDatameer
 
How to Avoid Pitfalls in Big Data Analytics Webinar
byDatameer
 
Getting Started with Big Data for Business Managers
byDatameer
 
Webinar - Big Data: Power to the User
byDatameer
 
BI, Hive or Big Data Analytics?
byDatameer
 
Customer Case Studies of Self-Service Big Data Analytics
byDatameer
 
Complement Your Existing Data Warehouse with Big Data & Hadoop
byDatameer
 
Extending BI with Big Data Analytics
byDatameer
 
Fight Fraud with Big Data Analytics
byDatameer
 
Instant Visualizations in Every Step of Analysis
byDatameer
 
Datameer6 for prospects - june 2016_v2
byDatameer
 
Lean Production Meets Big Data: A Next Generation Use Case
byDatameer
 
Online Fraud Detection Using Big Data Analytics Webinar
byDatameer
 
Understand Your Customer Buying Journey with Big Data
byDatameer
 
Is Your Hadoop Environment Secure?
byDatameer
 
Why Use Hadoop?
byDatameer
 
Why Use Hadoop for Big Data Analytics?
byDatameer
 
Webinar - Introducing Datameer 4.0: Visual, End-to-End
byDatameer
 
The State of Big Data Adoption: A Glance at Top Industries Adopting Big Data ...
byDatameer
 

Recently uploaded

PPTX
Mastering SQL Server Replication: Types, Setup & Troubleshooting
byGeoPITS Global Pvt Ltd
 
PDF
Skills to Pass the UiPath Agentic Automation Associate (UiAAA) Certification
byUiPathCommunity
 
PPTX
apidays Australia 2025 | Building AI RAG Applications with No Code.pptx
byapidays
 
PDF
Enterprise Data Services_ A Development Guide with Use Cases, Trends and Impl...
by2601harsh23
 
PPTX
Understanding-Search-Algorithms_09-12-25.pptx
byshaikmahammedtauheed
 
PPTX
SOFTWARE DEVELOPMENT PROCESS - INTRODUCTION
byParithi Thamizh
 
PDF
How PayPal Account Verification Works – Complete Guide for Online Businesses
byjhdhj3989
 
PDF
Why Are Cloud Migration Services Essential for Modern Business Growth?
byGeoPITS Global Pvt Ltd
 
PPTX
Achieve enterprise automation with SAP BTP solutions and SAP Signavio
bydarrellkiwi
 
PDF
5G Explained! A High Level Overview [Introduction]
byLuciano Motta
 
PPTX
Spacecraft Guidance Quick Research Guide by Arthur Morgan
byArthur Morgan
 
PDF
How to Design Premium Health (Public Health) PPT Using AI? Top Strategies
byPublic Health Concern Nepal
 
PDF
Introduction to Linux and Basic Commands
byiDev Semarang
 
PPTX
Strategic Shelf Planning for the New Year Turning Resolutions into Revenue .pptx
byAnoop Ashok
 
PDF
Deploying Windows Clients & Managing Identities
byVICTOR MAESTRE RAMIREZ
 
PDF
Transcript: What ONIX can do: Leveraging metadata to support the discoverabil...
byBookNet Canada
 
PPTX
Dell poweredge-customer-presentation.pptx
byhungungphu123
 
PDF
What ONIX can do: Leveraging metadata to support the discoverability of First...
byBookNet Canada
 
PDF
Router-DHCP-Printer-SharingRouter-DHCP-Printer-Sharing
byarbendi2160
 
PPTX
Single Cell Protein from Methane or Methanol - Process development research c...
byJohn Downs
 
Mastering SQL Server Replication: Types, Setup & Troubleshooting
byGeoPITS Global Pvt Ltd
 
Skills to Pass the UiPath Agentic Automation Associate (UiAAA) Certification
byUiPathCommunity
 
apidays Australia 2025 | Building AI RAG Applications with No Code.pptx
byapidays
 
Enterprise Data Services_ A Development Guide with Use Cases, Trends and Impl...
by2601harsh23
 
Understanding-Search-Algorithms_09-12-25.pptx
byshaikmahammedtauheed
 
SOFTWARE DEVELOPMENT PROCESS - INTRODUCTION
byParithi Thamizh
 
How PayPal Account Verification Works – Complete Guide for Online Businesses
byjhdhj3989
 
Why Are Cloud Migration Services Essential for Modern Business Growth?
byGeoPITS Global Pvt Ltd
 
Achieve enterprise automation with SAP BTP solutions and SAP Signavio
bydarrellkiwi
 
5G Explained! A High Level Overview [Introduction]
byLuciano Motta
 
Spacecraft Guidance Quick Research Guide by Arthur Morgan
byArthur Morgan
 
How to Design Premium Health (Public Health) PPT Using AI? Top Strategies
byPublic Health Concern Nepal
 
Introduction to Linux and Basic Commands
byiDev Semarang
 
Strategic Shelf Planning for the New Year Turning Resolutions into Revenue .pptx
byAnoop Ashok
 
Deploying Windows Clients & Managing Identities
byVICTOR MAESTRE RAMIREZ
 
Transcript: What ONIX can do: Leveraging metadata to support the discoverabil...
byBookNet Canada
 
Dell poweredge-customer-presentation.pptx
byhungungphu123
 
What ONIX can do: Leveraging metadata to support the discoverability of First...
byBookNet Canada
 
Router-DHCP-Printer-SharingRouter-DHCP-Printer-Sharing
byarbendi2160
 
Single Cell Protein from Methane or Methanol - Process development research c...
byJohn Downs
 

How to do Predictive Analytics with Limited Data

  • 1.
    How to doPredictive Analytics with Limited Data Ulrich Rueckert Š 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 2.
    Agenda Introduction and Motivation Semi-SupervisedLearning • Generative Models • Large Margin Approaches • Similarity Based Methods Conclusion © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 3.
    Predictive Modeling Traditional Supervised Learning • • • • • Promotionon bookseller’s web page Customers can rate books. Will a new customer like this book? Training set: observations on previous customers Test set: new customers What happens if only few customers rate a book? © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13 Test Data Age Attributes Income 60K 80K 95K 52K 45K 75K 51K + + + 52 47K - 47 25 38K 22K - 33 47K ? + 20 43 26 41K - 35 ? LikesBook + 65 60 67K 39 Age 24 LikesBook 22 Target Label Income + Training Data Model Age 22 Income 67K LikesBook + 39 41K - Prediction
  • 4.
    Predictive Modeling Test Data TraditionalSupervised Learning • • • • • Age Test set: new customers What happens if only few customers rate a book? 41K ? 95K 52K ? ? 20 45K ? 43 75K ? 26 52 51K 47K ? ? 47 38K ? 25 22K ? 33 Training set: observations on previous customers ? 60 35 Will a new customer like this book? 67K 39 Customers can rate books. LikesBook 22 Promotion on bookseller’s web page Income 47K ? Target Label Attributes Age Income LikesBook 24 60K + 65 80K - Model Training Data Prediction © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 5.
    Predictive Modeling Traditional Supervised Learning • • • • • Promotionon bookseller’s web page Customers can rate books. Will a new customer like this book? Training set: observations on previous customers Test set: new customers What happens if only few customers rate a book? © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13 Test Data Age Attributes Income 60K 80K 95K 52K 45K 75K 51K ? + ? 52 47K - 47 25 38K 22K ? ? 33 47K ? ? 20 43 26 41K ? 35 ? LikesBook + 65 60 67K 39 Age 24 LikesBook 22 Target Label Income ? Training Data Model Age 22 Income 67K LikesBook + 39 41K - Prediction
  • 6.
    Supervised Learning Classification • For now:each data instance is a point in a 2D coordinate system • • Color denotes target label Model is given as decision boundary What’s the correct model? • • In theory: no way to tell • All learning systems have underlying assumptions Smoothness assumption: similar instances have similar labels © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 7.
    Supervised Learning Classification • For now:each data instance is a point in a 2D coordinate system • • Color denotes target label Model is given as decision boundary What’s the correct model? • • In theory: no way to tell • All learning systems have underlying assumptions Smoothness assumption: similar instances have similar labels © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 8.
    Supervised Learning Classification • For now:each data instance is a point in a 2D coordinate system • • Color denotes target label Model is given as decision boundary What’s the correct model? • • In theory: no way to tell • All learning systems have underlying assumptions Smoothness assumption: similar instances have similar labels © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 9.
    Supervised Learning Classification • For now:each data instance is a point in a 2D coordinate system • • Color denotes target label Model is given as decision boundary What’s the correct model? • • In theory: no way to tell • All learning systems have underlying assumptions Smoothness assumption: similar instances have similar labels © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 10.
    Supervised Learning Classification • For now:each data instance is a point in a 2D coordinate system • • Color denotes target label Model is given as decision boundary What’s the correct model? • • In theory: no way to tell • All learning systems have underlying assumptions Smoothness assumption: similar instances have similar labels © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 11.
    Semi-Supervised Learning Can wemake use of the unlabeled data? • • In theory: no ... but we can make assumptions Popular Assumptions • • • Clustering assumption Low density assumption Manifold assumption © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 12.
    Semi-Supervised Learning Can wemake use of the unlabeled data? • • In theory: no ... but we can make assumptions Popular Assumptions • • • Clustering assumption Low density assumption Manifold assumption © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 13.
    The Clustering Assumption Clustering • Partitioninstances into groups (clusters) of similar instances • Many different algorithms: k-Means, EM, DBSCAN, etc. • Available e.g. on Mahout Clustering Assumption • The two classification targets are distinct clusters • Simple semi-supervised learning: cluster, then perform majority vote © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 14.
    The Clustering Assumption Clustering • Partitioninstances into groups (clusters) of similar instances • Many different algorithms: k-Means, EM, DBSCAN, etc. • Available e.g. on Mahout Clustering Assumption • The two classification targets are distinct clusters • Simple semi-supervised learning: cluster, then perform majority vote © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 15.
    The Clustering Assumption Clustering • Partitioninstances into groups (clusters) of similar instances • Many different algorithms: k-Means, EM, DBSCAN, etc. • Available e.g. on Mahout Clustering Assumption • The two classification targets are distinct clusters • Simple semi-supervised learning: cluster, then perform majority vote © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 16.
    The Clustering Assumption Clustering • Partitioninstances into groups (clusters) of similar instances • Many different algorithms: k-Means, EM, DBSCAN, etc. • Available e.g. on Mahout Clustering Assumption • The two classification targets are distinct clusters • Simple semi-supervised learning: cluster, then perform majority vote © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 17.
    Generative Models Mixture ofGaussians • Assumption: the data in each cluster is generated by a normal distribution • Find most probable location and shape of clusters given data Expectation-Maximization • • Two step optimization procedure • • Each step is one MapReduce job Keeps estimates of cluster assignment probabilities for each instance Might converge to local optimum © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 18.
    Generative Models Mixture ofGaussians • Assumption: the data in each cluster is generated by a normal distribution • Find most probable location and shape of clusters given data Expectation-Maximization • • Two step optimization procedure • • Each step is one MapReduce job Keeps estimates of cluster assignment probabilities for each instance Might converge to local optimum © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 19.
    Generative Models Mixture ofGaussians • Assumption: the data in each cluster is generated by a normal distribution • Find most probable location and shape of clusters given data Expectation-Maximization • • Two step optimization procedure • • Each step is one MapReduce job Keeps estimates of cluster assignment probabilities for each instance Might converge to local optimum © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 20.
    Generative Models Mixture ofGaussians • Assumption: the data in each cluster is generated by a normal distribution • Find most probable location and shape of clusters given data Expectation-Maximization • • Two step optimization procedure • • Each step is one MapReduce job Keeps estimates of cluster assignment probabilities for each instance Might converge to local optimum © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 21.
    Generative Models Mixture ofGaussians • Assumption: the data in each cluster is generated by a normal distribution • Find most probable location and shape of clusters given data Expectation-Maximization • • Two step optimization procedure • • Each step is one MapReduce job Keeps estimates of cluster assignment probabilities for each instance Might converge to local optimum © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 22.
    Generative Models Mixture ofGaussians • Assumption: the data in each cluster is generated by a normal distribution • Find most probable location and shape of clusters given data Expectation-Maximization • • Two step optimization procedure • • Each step is one MapReduce job Keeps estimates of cluster assignment probabilities for each instance Might converge to local optimum © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 23.
    Beyond Mixtures ofGaussians Expectation-Maximization • • Can be adjusted to all kinds of mixture models E.g. use Naive Bayes as mixture model for text classification Self-Training • • • • Learn model on labeled instances only Apply model to unlabeled instances Learn new model on all instances Repeat until convergence © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 24.
    The Low DensityAssumption Assumption • The area between the two classes has low density • Does not assume any specific form of cluster Support Vector Machine • • • Decision boundary is linear Maximizes margin to closest instances Can be learned in one Map-Reduce step (Stochastic Gradient Descent) © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 25.
    The Low DensityAssumption Assumption • The area between the two classes has low density • Does not assume any specific form of cluster Support Vector Machine • • • Decision boundary is linear Maximizes margin to closest instances Can be learned in one Map-Reduce step (Stochastic Gradient Descent) © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 26.
    The Low DensityAssumption Assumption • The area between the two classes has low density • Does not assume any specific form of cluster Support Vector Machine • • • Decision boundary is linear Maximizes margin to closest instances Can be learned in one Map-Reduce step (Stochastic Gradient Descent) © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 27.
    The Low DensityAssumption Semi-Supervised Support Vector Machine • Minimize distance to labeled and unlabeled instances • Parameter to fine-tune influence of unlabeled instances • Additional constraint: keep class balance correct Implementation • • Simple extension of SVM But non-convex optimization problem © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 28.
    The Low DensityAssumption Semi-Supervised Support Vector Machine • Minimize distance to labeled and unlabeled instances • Parameter to fine-tune influence of unlabeled instances • Additional constraint: keep class balance correct Implementation • • Simple extension of SVM But non-convex optimization problem © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 29.
    The Low DensityAssumption Semi-Supervised Support Vector Machine • Minimize distance to labeled and unlabeled instances • Parameter to fine-tune influence of unlabeled instances • Additional constraint: keep class balance correct Implementation • • Simple extension of SVM But non-convex optimization problem © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 30.
    The Low DensityAssumption Semi-Supervised Support Vector Machine • • • Minimize distance to labeled and unlabeled instances Parameter to fine-tune influence of unlabeled instances Additional constraint: keep class balance correct Implementation • • Margins of labeled instances Regularizer 2 min kwk w +C +C n X `(yi (wT xi + b)) i=1 n+m X ? `(|wT xi + b|) i=n+1 Simple extension of SVM But non-convex optimization problem © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13 Margins of unlabeled instances
  • 31.
    Semi-Supervised SVM Stochastic GradientDescent • • One run over the data in random order • Steps get smaller over time Each misclassified or unlabeled instance moves classifier a bit Implementation on Hadoop • Mapper: send data to reducer in random order • Reducer: update linear classifier for unlabeled or misclassified instances • Many random runs to find best one © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 32.
    Semi-Supervised SVM Stochastic GradientDescent • • One run over the data in random order • Steps get smaller over time Each misclassified or unlabeled instance moves classifier a bit Implementation on Hadoop • Mapper: send data to reducer in random order • Reducer: update linear classifier for unlabeled or misclassified instances • Many random runs to find best one © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 33.
    Semi-Supervised SVM Stochastic GradientDescent • • One run over the data in random order • Steps get smaller over time Each misclassified or unlabeled instance moves classifier a bit Implementation on Hadoop • Mapper: send data to reducer in random order • Reducer: update linear classifier for unlabeled or misclassified instances • Many random runs to find best one © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 34.
    Semi-Supervised SVM Stochastic GradientDescent • • One run over the data in random order • Steps get smaller over time Each misclassified or unlabeled instance moves classifier a bit Implementation on Hadoop • Mapper: send data to reducer in random order • Reducer: update linear classifier for unlabeled or misclassified instances • Many random runs to find best one © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 35.
    Semi-Supervised SVM Stochastic GradientDescent • • One run over the data in random order • Steps get smaller over time Each misclassified or unlabeled instance moves classifier a bit Implementation on Hadoop • Mapper: send data to reducer in random order • Reducer: update linear classifier for unlabeled or misclassified instances • Many random runs to find best one © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 36.
    The Manifold Assumption TheAssumption • Training data is (roughly) contained in a low dimensional manifold • One can perform learning in a more meaningful low-dimensional space • Avoids curse of dimensionality Similarity Graphs • Idea: compute similarity scores between instances • Create network where the nearest neighbors are connected © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 37.
    The Manifold Assumption TheAssumption • Training data is (roughly) contained in a low dimensional manifold • One can perform learning in a more meaningful low-dimensional space • Avoids curse of dimensionality Similarity Graphs • Idea: compute similarity scores between instances • Create network where the nearest neighbors are connected © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 38.
    The Manifold Assumption TheAssumption • Training data is (roughly) contained in a low dimensional manifold • One can perform learning in a more meaningful low-dimensional space • Avoids curse of dimensionality Similarity Graphs • Idea: compute similarity scores between instances • Create network where the nearest neighbors are connected © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 39.
    The Manifold Assumption TheAssumption • Training data is (roughly) contained in a low dimensional manifold • One can perform learning in a more meaningful low-dimensional space • Avoids curse of dimensionality Similarity Graphs • Idea: compute similarity scores between instances • Create network where the nearest neighbors are connected © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 40.
    Label Propagation Main Idea • Propagatelabel information to neighboring instances • • Then repeat until convergence Similar to PageRank Theory • Known to converge under weak conditions • Equivalent to matrix inversion © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 41.
    Label Propagation Main Idea • Propagatelabel information to neighboring instances • • Then repeat until convergence Similar to PageRank Theory • Known to converge under weak conditions • Equivalent to matrix inversion © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 42.
    Label Propagation Main Idea • Propagatelabel information to neighboring instances • • Then repeat until convergence Similar to PageRank Theory • Known to converge under weak conditions • Equivalent to matrix inversion © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 43.
    Label Propagation Main Idea • Propagatelabel information to neighboring instances • • Then repeat until convergence Similar to PageRank Theory • Known to converge under weak conditions • Equivalent to matrix inversion © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 44.
    Nearest Neighbor Join BlockNested Loop Join • • 1st MR job: partition data into blocks, compute nearest neighbors between blocks 2nd MR job: filter out the overall nearest neighbors Reducers Use spatial information to avoid unnecessary comparisons • R trees, space filling curves, locality sensitive hashing • Example implementations available online © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13 1 1 1 2 1 3 Smarter Approaches • Data 1 4 1 1 1 1
  • 45.
    Nearest Neighbor Join BlockNested Loop Join • • 1st MR job: partition data into blocks, compute nearest neighbors between blocks 2nd MR job: filter out the overall nearest neighbors Reducers Use spatial information to avoid unnecessary comparisons • R trees, space filling curves, locality sensitive hashing • Example implementations available online © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13 1 1 1 1 2 1 1 2 2 1 2 2 2 2 3 Smarter Approaches • Data 1 2 4 1 2
  • 46.
    Nearest Neighbor Join BlockNested Loop Join • • 1st MR job: partition data into blocks, compute nearest neighbors between blocks 2nd MR job: filter out the overall nearest neighbors Reducers Use spatial information to avoid unnecessary comparisons • R trees, space filling curves, locality sensitive hashing • Example implementations available online © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13 1 1 1 1 2 1 3 1 4 2 2 1 2 2 2 3 2 4 3 Smarter Approaches • Data 3 1 3 2 3 3 3 4 4 4 1 4 2 4 3 4 4
  • 47.
    Nearest Neighbor Join BlockNested Loop Join • • 1st MR job: partition data into blocks, compute nearest neighbors between blocks 2nd MR job: filter out the overall nearest neighbors Smarter Approaches • Data 1 2 3 Use spatial information to avoid unnecessary comparisons • R trees, space filling curves, locality sensitive hashing • Example implementations available online © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13 4 Nearest Neighbors
  • 48.
    Nearest Neighbor Join BlockNested Loop Join • • 1st MR job: partition data into blocks, compute nearest neighbors between blocks 2nd MR job: filter out the overall nearest neighbors Smarter Approaches • Data 1 2 3 Use spatial information to avoid unnecessary comparisons • R trees, space filling curves, locality sensitive hashing • Example implementations available online © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13 4 Nearest Neighbors Result
  • 49.
    Nearest Neighbor Join BlockNested Loop Join • • 1st MR job: partition data into blocks, compute nearest neighbors between blocks 2nd MR job: filter out the overall nearest neighbors Smarter Approaches • Data 1 1 1 1 2 2 2 1 2 2 2 3 3 2 3 3 3 4 4 3 4 4 3 Use spatial information to avoid unnecessary comparisons • R trees, space filling curves, locality sensitive hashing • Example implementations available online © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13 Reducers 4
  • 50.
    Nearest Neighbor Join BlockNested Loop Join • 1st MR job: partition data into blocks, compute nearest neighbors between blocks • 2nd MR job: filter out the overall nearest neighbors Smarter Approaches • Use spatial information to avoid unnecessary comparisons • R trees, space filling curves, locality sensitive hashing • Example implementations available online © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 51.
    Nearest Neighbor Join BlockNested Loop Join • 1st MR job: partition data into blocks, compute nearest neighbors between blocks • 2nd MR job: filter out the overall nearest neighbors Smarter Approaches • Use spatial information to avoid unnecessary comparisons • R trees, space filling curves, locality sensitive hashing • Example implementations available online © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 52.
    Label Propagation onHadoop Iterative Procedure • • Mapper: Emit neighbor-label pairs • Repeat until convergence Reducer: Collect incoming labels per node and combine them into new label Improvement • Cluster data and perform within-cluster propagation locally • Sometimes it’s more efficient to perform matrix inversion instead © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 53.
    Label Propagation onHadoop Iterative Procedure • • Mapper: Emit neighbor-label pairs • Repeat until convergence Reducer: Collect incoming labels per node and combine them into new label Improvement • Cluster data and perform within-cluster propagation locally • Sometimes it’s more efficient to perform matrix inversion instead © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 54.
    Label Propagation onHadoop Iterative Procedure • • Mapper: Emit neighbor-label pairs • Repeat until convergence Reducer: Collect incoming labels per node and combine them into new label Improvement • Cluster data and perform within-cluster propagation locally • Sometimes it’s more efficient to perform matrix inversion instead © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 55.
    Label Propagation onHadoop Iterative Procedure • • Mapper: Emit neighbor-label pairs • Repeat until convergence Reducer: Collect incoming labels per node and combine them into new label Improvement • Cluster data and perform within-cluster propagation locally • Sometimes it’s more efficient to perform matrix inversion instead © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 56.
    Label Propagation onHadoop Iterative Procedure • • Mapper: Emit neighbor-label pairs • Repeat until convergence Reducer: Collect incoming labels per node and combine them into new label Improvement • Cluster data and perform within-cluster propagation locally • Sometimes it’s more efficient to perform matrix inversion instead © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 57.
    Conclusion Semi-Supervised Learning • Only fewtraining instances have labels • Unlabeled instances can still provide valuable signal Different assumptions lead to different approaches • • • Cluster assumption: generative models Low density assumption: semi-supervised support vector machines Manifold assumption: label propagation Code available: https://github.com/Datameer-Inc/lesel © 2013 Datameer, Inc. All rights reserved. Thursday, October 31, 13
  • 58.