DataWorks Summit, profile picture
Uploaded byDataWorks Summit
3,893 views

Similarity at Scale

The document discusses techniques for generating recommendations at scale using similarity algorithms. It begins by defining different types of similarity problems like clustering, deduplication, recommendations, and entity resolution. It then discusses what similarity means and how it is different from exact matching. Different algorithms for measuring similarity between objects are described, including Jaccard coefficient and cosine similarity. The challenges of scaling similarity calculations to large datasets are discussed. Recommendation algorithms like analyzing co-occurrence of items in user sessions are explained. The use of Lucene search and the Mahout RowSimilarityJob to calculate item-item similarity matrices for generating related item recommendations are also covered.

Embed presentation

Copyright (c) 2014 Scale Unlimited. 1 Similarity at Scale Fuzzy matching and recommendations using Hadoop, Solr, and heuristics Ken Krugler Scale Unlimited
Copyright (c) 2014 Scale Unlimited. The Twitter Pitch Wide class of problems that rely on "good" similarity Fast Accurate Scalable Benefit from my mistakes Scale Unlimited - consulting & training Talking about solutions to real problems 2
Copyright (c) 2014 Scale Unlimited. What are similarity problems? Clustering Grouping similar advertisers Deduplication Joining noisy sets of POI data Recommendations Suggesting pages to users Entity resolution Fuzzy matching of people and companies 3
Copyright (c) 2014 Scale Unlimited. What is "Similarity"? Exact matching is easy(er) Accuracy is a given Fast and scalable can still be hard Lots of key/value systems like Cassandra, HBase, etc. Fuzzy matching is harder Two "things" aren't exactly the same Similarity is based on comparing features 4
Copyright (c) 2014 Scale Unlimited. Between two articles? Features could be a bag of words Are these two articles the same? 5 Bosnia is the largest geographic region of the modern state with a moderate continental climate, marked by hot summers and cold, snowy winters. The inland is a geographically larger region and has a moderate continental climate, bookended by hot summers and cold and snowy winters.
Copyright (c) 2014 Scale Unlimited. What about now? Easy to create challenging situations for a person Which is an impossible problem for a computer Need to distinguish between "conceptually similar" and "derived from" 6 Bosnia is the largest geographic region of the modern state with a moderate continental climate, marked by hot summers and cold, snowy winters. Bosnia has a warm European climate, though the summers can be hot and the winters are often cold and wet.
Copyright (c) 2014 Scale Unlimited. Between two records? Features could be field values Are these two people the same? 7 Name Address City State Zip Bob Bogus Robert Bogus 220 3rd Avenue 220 3rd Avenue Seattle Seattle WA WA 98104-2608 98104
Copyright (c) 2014 Scale Unlimited. What about now? Need to get rid of false differences caused by abbreviations How does a computer know what's a "significant" difference? 8 Name Address City State Zip Bob Bogus Robert H. Bogus Apt 102, 3220 3rd Ave 220 3rd Avenue South Seattle Seattle Washington WA 98104
Copyright (c) 2014 Scale Unlimited. Between two users? Features could be... Items a user has bought Are these two users the same? 9 User 1 User 2
Copyright (c) 2014 Scale Unlimited. What about now? Need more generic features E.g. product categories 10 User 1 User 2
Copyright (c) 2014 Scale Unlimited. How to measure similarity? Assuming you have some features for two "things" How does a program determine their degree of similarity? You want a number that represents their "closeness" Typically 1.0 means exactly the same And 0.0 means completely different 11
Copyright (c) 2014 Scale Unlimited. Jaccard Coefficient Ratio of number of items in common / total number of items Where "items" typical means unique values (sets of things) So 1.0 is exactly the same, and 0.0 is completely different 12 Jaccard(A, B) = A!B A"B
Copyright (c) 2014 Scale Unlimited. Cosine Similarity Assume a document only has three unique words cat, dog, goldfish Set x = frequency of cat Set y = frequency of dog Set z = frequency of goldfish The result is a "term vector" with 3 dimensions Calculate cosine of angle between term vectors This is their "cosine similarity" 13
Copyright (c) 2014 Scale Unlimited. Why is scalability hard? Assume you have 8.5 million businesses in the US There are ≈ N^2/2 pairs to evaluate That's 36 trillion comparisons Sometimes you can quickly trim this problem E.g. if you assume the ZIP code exists, and must match Then this becomes about 4 billion comparisons But often you don't have a "magic" field 14
Copyright (c) 2011-2014 Scale Unlimited. All Rights Reserved. Reproduction or distribution of this document in any form without prior written permission is forbidden. DataStax Web Site Page Recommender 15
Copyright (c) 2014 Scale Unlimited. How to recommend pages? Besides manually adding a bunch of links... Which is tedious, doesn't scale well, and gets busy 16
Copyright (c) 2014 Scale Unlimited. Can we exploit other users? Classic shopping cart analysis "Users who bought X also bought Y" Based on actual activity, versus (noisy, skewed) ratings 17
Copyright (c) 2014 Scale Unlimited. What's the general approach? We have web logs with IP addresses, time, path to page 157.55.33.39 - - [18/Mar/2014:00:01:00 -0500] "GET /solutions/nosql HTTP/1.1" A browsing session is a series of requests from one IP address With some maximum time gap between requests Find sessions "similar to" the current user's session Recommend pages from these similar sessions 18
Copyright (c) 2014 Scale Unlimited. How to find similar sessions? Create a Lucene search index with one document per session Each indexed document contains the page paths for one session session-1 /path/to/page1, /path/to/page2, /path/to/page3 session-2 /path/to/pageX, /path/to/pageY Search for paths from the current user's session 19
Copyright (c) 2014 Scale Unlimited. Why is this a search issue? Solr (search in general) is all about similarity Find documents similar to the words in my query Cosine similarity is used to calculate similarity Between the term vector for my query and the term vector of each document 20
Copyright (c) 2014 Scale Unlimited. What's the algorithm? Find sessions similar to the target (current user's) session Calculate similarity between these sessions and the target session Aggregate similarity scores for all paths from these sessions Remove paths that are already in the target session Recommend the highest scoring path(s) 21
Copyright (c) 2014 Scale Unlimited. Why do you sum similarities? Give more weight to pages from sessions that are more similar Pages from more similar sessions are assumed to be more interesting 22 F D B C A Jaccard = 0.2 (1 / 5) Session 2 vs Target Session E D B C A Jaccard = 0.4 (2 / 5) Session 1 vs Target Session D E F 0.6 (0.4 + 0.2) 0.4 0.2 Page Score
Copyright (c) 2014 Scale Unlimited. What are some problems? The classic problem is that we recommend "common" pages 23
Copyright (c) 2014 Scale Unlimited. What are some problems? The classic problem is that we recommend "common" pages E.g. if you haven't viewed the top-level page in your session 23
Copyright (c) 2014 Scale Unlimited. What are some problems? The classic problem is that we recommend "common" pages E.g. if you haven't viewed the top-level page in your session But this page is very common in most of the other sessions 23
Copyright (c) 2014 Scale Unlimited. What are some problems? The classic problem is that we recommend "common" pages E.g. if you haven't viewed the top-level page in your session But this page is very common in most of the other sessions So then it becomes one of the top recommended page 23
Copyright (c) 2014 Scale Unlimited. What are some problems? The classic problem is that we recommend "common" pages E.g. if you haven't viewed the top-level page in your session But this page is very common in most of the other sessions So then it becomes one of the top recommended page But that generally stinks as a recommendation 23
Copyright (c) 2014 Scale Unlimited. Can RowSimilarityJob help? 24 Page 1 Page 2 Page 3 Page 1 Page 2 Page 3 2.1 0.8 2.1 4.5 0.8 4.5
Copyright (c) 2014 Scale Unlimited. Can RowSimilarityJob help? Part of the Mahout open source project 24 Page 1 Page 2 Page 3 Page 1 Page 2 Page 3 2.1 0.8 2.1 4.5 0.8 4.5
Copyright (c) 2014 Scale Unlimited. Can RowSimilarityJob help? Part of the Mahout open source project Takes as input a table of users (one per row) with lists of items 24 Page 1 Page 2 Page 3 Page 1 Page 2 Page 3 2.1 0.8 2.1 4.5 0.8 4.5
Copyright (c) 2014 Scale Unlimited. Can RowSimilarityJob help? Part of the Mahout open source project Takes as input a table of users (one per row) with lists of items Generates an item-item co-occurrence matrix 24 Page 1 Page 2 Page 3 Page 1 Page 2 Page 3 2.1 0.8 2.1 4.5 0.8 4.5
Copyright (c) 2014 Scale Unlimited. Can RowSimilarityJob help? Part of the Mahout open source project Takes as input a table of users (one per row) with lists of items Generates an item-item co-occurrence matrix Values are weights calculated using log-likelihood ratio (LLR) 24 Page 1 Page 2 Page 3 Page 1 Page 2 Page 3 2.1 0.8 2.1 4.5 0.8 4.5
Copyright (c) 2014 Scale Unlimited. Can RowSimilarityJob help? Part of the Mahout open source project Takes as input a table of users (one per row) with lists of items Generates an item-item co-occurrence matrix Values are weights calculated using log-likelihood ratio (LLR) Unsurprising (common) items get low weights 24 Page 1 Page 2 Page 3 Page 1 Page 2 Page 3 2.1 0.8 2.1 4.5 0.8 4.5
Copyright (c) 2014 Scale Unlimited. Can RowSimilarityJob help? Part of the Mahout open source project Takes as input a table of users (one per row) with lists of items Generates an item-item co-occurrence matrix Values are weights calculated using log-likelihood ratio (LLR) Unsurprising (common) items get low weights If we run it on our data, where users = sessions and items = pages 24 Page 1 Page 2 Page 3 Page 1 Page 2 Page 3 2.1 0.8 2.1 4.5 0.8 4.5
Copyright (c) 2014 Scale Unlimited. Can RowSimilarityJob help? Part of the Mahout open source project Takes as input a table of users (one per row) with lists of items Generates an item-item co-occurrence matrix Values are weights calculated using log-likelihood ratio (LLR) Unsurprising (common) items get low weights If we run it on our data, where users = sessions and items = pages We get page-page co-occurrence matrix 24 Page 1 Page 2 Page 3 Page 1 Page 2 Page 3 2.1 0.8 2.1 4.5 0.8 4.5
Copyright (c) 2014 Scale Unlimited. How to use co-occurrence? 25 Page 1 Page 2 Page 3 Page 1 Page 2 Page 3 2.1 0.8 2.1 4.5 0.8 4.5 Related Pages Page 1 Page 2 Page 3 Page 2 Page 1, Page 3 Page 2
Copyright (c) 2014 Scale Unlimited. How to use co-occurrence? Convert the matrix into an index 25 Page 1 Page 2 Page 3 Page 1 Page 2 Page 3 2.1 0.8 2.1 4.5 0.8 4.5 Related Pages Page 1 Page 2 Page 3 Page 2 Page 1, Page 3 Page 2
Copyright (c) 2014 Scale Unlimited. How to use co-occurrence? Convert the matrix into an index Each row is one Lucene document 25 Page 1 Page 2 Page 3 Page 1 Page 2 Page 3 2.1 0.8 2.1 4.5 0.8 4.5 Related Pages Page 1 Page 2 Page 3 Page 2 Page 1, Page 3 Page 2
Copyright (c) 2014 Scale Unlimited. How to use co-occurrence? Convert the matrix into an index Each row is one Lucene document Drop any low-scoring entries 25 Page 1 Page 2 Page 3 Page 1 Page 2 Page 3 2.1 0.8 2.1 4.5 0.8 4.5 Related Pages Page 1 Page 2 Page 3 Page 2 Page 1, Page 3 Page 2
Copyright (c) 2014 Scale Unlimited. How to use co-occurrence? Convert the matrix into an index Each row is one Lucene document Drop any low-scoring entries Create list of "related" pages 25 Page 1 Page 2 Page 3 Page 1 Page 2 Page 3 2.1 0.8 2.1 4.5 0.8 4.5 Related Pages Page 1 Page 2 Page 3 Page 2 Page 1, Page 3 Page 2
Copyright (c) 2014 Scale Unlimited. How to use co-occurrence? Convert the matrix into an index Each row is one Lucene document Drop any low-scoring entries Create list of "related" pages 25 Page 1 Page 2 Page 3 Page 1 Page 2 Page 3 2.1 0.8 2.1 4.5 0.8 4.5 Related Pages Page 1 Page 2 Page 3 Page 2 Page 1, Page 3 Page 2
Copyright (c) 2014 Scale Unlimited. How to use co-occurrence? Convert the matrix into an index Each row is one Lucene document Drop any low-scoring entries Create list of "related" pages Search in Related Pages field 25 Page 1 Page 2 Page 3 Page 1 Page 2 Page 3 2.1 0.8 2.1 4.5 0.8 4.5 Related Pages Page 1 Page 2 Page 3 Page 2 Page 1, Page 3 Page 2
Copyright (c) 2014 Scale Unlimited. How to use co-occurrence? Convert the matrix into an index Each row is one Lucene document Drop any low-scoring entries Create list of "related" pages Search in Related Pages field Using pages from current session 25 Page 1 Page 2 Page 3 Page 1 Page 2 Page 3 2.1 0.8 2.1 4.5 0.8 4.5 Related Pages Page 1 Page 2 Page 3 Page 2 Page 1, Page 3 Page 2
Copyright (c) 2014 Scale Unlimited. How to use co-occurrence? Convert the matrix into an index Each row is one Lucene document Drop any low-scoring entries Create list of "related" pages Search in Related Pages field Using pages from current session So Page 2 recommends Page 1 & 3 25 Page 1 Page 2 Page 3 Page 1 Page 2 Page 3 2.1 0.8 2.1 4.5 0.8 4.5 Related Pages Page 1 Page 2 Page 3 Page 2 Page 1, Page 3 Page 2
Copyright (c) 2011-2014 Scale Unlimited. All Rights Reserved. Reproduction or distribution of this document in any form without prior written permission is forbidden. EWS Entity Resolution 26
Copyright (c) 2014 Scale Unlimited. What is Early Warning? Early Warning helps banks fight fraud It's owned by the top 5 US banks And gets data from 800+ financial institutions So they have details on most US bank accounts When somebody signs up for an account They need to quickly match the person to "known entities" And derive a risk score based on related account details 27
Copyright (c) 2014 Scale Unlimited. Why do they need similarity? Assume you have information on 100s of millions of entities Name(s), address(es), phone number(s), etc. And often a unique ID (Social Security Number, EIN, etc) Why is this a similarity problem? Data is noisy - typos, abbreviations, partial data People lie - much fraud starts with opening an account using bad data 28
Copyright (c) 2014 Scale Unlimited. How does search help? We can quickly build a list of candidate entities, using search Query contains field data provided by the client bank Significantly less than 1 second for 30 candidate entities Then do more precise, sophisticated and CPU-intensive scoring The end result is a ranked list of entities with similarity scores Which then is used to look up account status, fraud cases, etc. 29
Copyright (c) 2014 Scale Unlimited. What's the data pipeline? Incoming data is cleaned up/normalized in Hadoop Simple things like space stripping Also phone number formatting ZIP+4 expansion into just ZIP plus full Other normalization happens inside of Solr This gets loaded into Cassandra tables And automatically indexed by Solr, via DataStax Enterprise 30 ZIP+4 Terms 95014-2127 95014, 2127 Phone Terms 4805551212 480, 5551212
Copyright (c) 2014 Scale Unlimited. What's the Solr setup? Each field in the index has very specific analysis Simple things like normalization Synonym expansion for names, abbreviations Split up fields so partial matches work At query time we can weight the importance of each field Which helps order the top N candidates similar to their real match scores E.g. an SSN matching means much more than a first name matching 31
Copyright (c) 2011-2014 Scale Unlimited. All Rights Reserved. Reproduction or distribution of this document in any form without prior written permission is forbidden. Batch Similarity 32
Copyright (c) 2014 Scale Unlimited. Can we do batch similarity? Search works well for real-time similarity But batch processing at scale maxes out the search system We can use two different techniques with Hadoop for batch SimHash - good for text document similarity Parallel Set-Similarity Joins - good for record similarity 33
Copyright (c) 2014 Scale Unlimited. What is SimHash? Assume a document is a set of (unique) words Calculate a hash for each word Probability that the minimum hash is the same for two documents... ...is magically equal to the Jaccard Coefficient 34 Term Hash bosnia is the largest geographic 78954874223 53466156768 5064199193 3193621783 -5718349925
Copyright (c) 2014 Scale Unlimited. What is a SimHash workflow? Calculate N hash values Easy way is to use the N smallest hash values Calculate number of matching hash values between doc pairs (M) Then the Jaccard Coefficient is ≈ M/N Only works if N is much smaller than # of unique words in docs Implementation of this in cascading.utils open source project https://github.com/ScaleUnlimited/cascading.utils 35
Copyright (c) 2014 Scale Unlimited. What is Set-Similarity Join? Joining records in two sets that are "close enough" aka "fuzzy join" Requires generation of "tokens" from record field(s) Typically words from text Simple implementation has three phases First calculate counts for each unique token value Then output <token, record> for N most common tokens of each record Group by token, compare records in each group 36
Copyright (c) 2014 Scale Unlimited. How does fuzzy join work? For two records to be "similar enough"... They need to share one of their common tokens Generalization of the ZIP code "magic field" approach Basic implementation has a number of issues Passing around copies of full record is inefficient Too-common tokens create huge groups for comparison Two records compared multiple times 37
Copyright (c) 2011-2014 Scale Unlimited. All Rights Reserved. Reproduction or distribution of this document in any form without prior written permission is forbidden. Summary 38
Copyright (c) 2014 Scale Unlimited. The Net-Net 39
Copyright (c) 2014 Scale Unlimited. The Net-Net Similarity is a common requirement for many applications 39
Copyright (c) 2014 Scale Unlimited. The Net-Net Similarity is a common requirement for many applications Recommendations 39
Copyright (c) 2014 Scale Unlimited. The Net-Net Similarity is a common requirement for many applications Recommendations Entity matching 39
Copyright (c) 2014 Scale Unlimited. The Net-Net Similarity is a common requirement for many applications Recommendations Entity matching Combining Hadoop with search is a powerful combination 39
Copyright (c) 2014 Scale Unlimited. The Net-Net Similarity is a common requirement for many applications Recommendations Entity matching Combining Hadoop with search is a powerful combination Scalability 39
Copyright (c) 2014 Scale Unlimited. The Net-Net Similarity is a common requirement for many applications Recommendations Entity matching Combining Hadoop with search is a powerful combination Scalability Performance 39
Copyright (c) 2014 Scale Unlimited. The Net-Net Similarity is a common requirement for many applications Recommendations Entity matching Combining Hadoop with search is a powerful combination Scalability Performance Flexibility 39
Copyright (c) 2014 Scale Unlimited. Questions? Feel free to contact me http://www.scaleunlimited.com/contact/ Take a look at Pat Ferrel's Hadoop + Solr recommender http://github.com/pferrel/solr-recommender Check out Mahout http://mahout.apache.org Read paper & code for fuzzyjoin project http://asterix.ics.uci.edu/fuzzyjoin/ 40

More Related Content

PDF
Chicago Spark Meetup 03 01 2016 - Spark and Recommendations
byChris Fregly
 
PDF
Advanced Analytics and Recommendations with Apache Spark - Spark Maryland/DC ...
byChris Fregly
 
PDF
5th Athens Big Data Meetup - PipelineIO Workshop - Real-Time Training and Dep...
byAthens Big Data
 
PDF
USF Seminar Series: Apache Spark, Machine Learning, Recommendations Feb 05 2016
byChris Fregly
 
PDF
Advanced Apache Spark Meetup Approximations and Probabilistic Data Structures...
byChris Fregly
 
PDF
Spark, Similarity, Approximations, NLP, Recommendations - Boulder Denver Spar...
byChris Fregly
 
PDF
Dallas DFW Data Science Meetup Jan 21 2016
byChris Fregly
 
PDF
Advanced Apache Spark Meetup Spark and Elasticsearch 02-15-2016
byChris Fregly
 
Chicago Spark Meetup 03 01 2016 - Spark and Recommendations
byChris Fregly
 
Advanced Analytics and Recommendations with Apache Spark - Spark Maryland/DC ...
byChris Fregly
 
5th Athens Big Data Meetup - PipelineIO Workshop - Real-Time Training and Dep...
byAthens Big Data
 
USF Seminar Series: Apache Spark, Machine Learning, Recommendations Feb 05 2016
byChris Fregly
 
Advanced Apache Spark Meetup Approximations and Probabilistic Data Structures...
byChris Fregly
 
Spark, Similarity, Approximations, NLP, Recommendations - Boulder Denver Spar...
byChris Fregly
 
Dallas DFW Data Science Meetup Jan 21 2016
byChris Fregly
 
Advanced Apache Spark Meetup Spark and Elasticsearch 02-15-2016
byChris Fregly
 

What's hot

PDF
DC Spark Users Group March 15 2016 - Spark and Netflix Recommendations
byChris Fregly
 
PDF
Singapore Spark Meetup Dec 01 2015
byChris Fregly
 
PDF
Istanbul Spark Meetup Nov 28 2015
byChris Fregly
 
PDF
Spark Summit East NYC Meetup 02-16-2016
byChris Fregly
 
PDF
Melbourne Spark Meetup Dec 09 2015
byChris Fregly
 
PDF
Toronto Spark Meetup Dec 14 2015
byChris Fregly
 
PDF
Sydney Spark Meetup Dec 08, 2015
byChris Fregly
 
PDF
Copenhagen Spark Meetup Nov 25, 2015
byChris Fregly
 
PDF
Budapest Big Data Meetup Nov 26 2015
byChris Fregly
 
PDF
Spark After Dark 2.0 - Apache Big Data Conf - Vancouver - May 11, 2016
byChris Fregly
 
PDF
Barcelona Spain Apache Spark Meetup Oct 20, 2015: Spark Streaming, Kafka, MLl...
byChris Fregly
 
PDF
Stockholm Spark Meetup Nov 23 2015 Spark After Dark 1.5
byChris Fregly
 
PDF
Scotland Data Science Meetup Oct 13, 2015: Spark SQL, DataFrames, Catalyst, ...
byChris Fregly
 
PDF
Helsinki Spark Meetup Nov 20 2015
byChris Fregly
 
PDF
Zurich, Berlin, Vienna Spark and Big Data Meetup Nov 02 2015
byChris Fregly
 
PDF
Practical Data Science Workshop - Recommendation Systems - Collaborative Filt...
byChris Fregly
 
PDF
Augury and Omens Aside, Part 1:
 The Business Case for Apache Mesos
byPaco Nathan
 
PDF
GalvanizeU Seattle: Eleven Almost-Truisms About Data
byPaco Nathan
 
PPTX
Tech
byManabuYoneyama
 
PDF
Sparksee overview
bySparsity Technologies
 
DC Spark Users Group March 15 2016 - Spark and Netflix Recommendations
byChris Fregly
 
Singapore Spark Meetup Dec 01 2015
byChris Fregly
 
Istanbul Spark Meetup Nov 28 2015
byChris Fregly
 
Spark Summit East NYC Meetup 02-16-2016
byChris Fregly
 
Melbourne Spark Meetup Dec 09 2015
byChris Fregly
 
Toronto Spark Meetup Dec 14 2015
byChris Fregly
 
Sydney Spark Meetup Dec 08, 2015
byChris Fregly
 
Copenhagen Spark Meetup Nov 25, 2015
byChris Fregly
 
Budapest Big Data Meetup Nov 26 2015
byChris Fregly
 
Spark After Dark 2.0 - Apache Big Data Conf - Vancouver - May 11, 2016
byChris Fregly
 
Barcelona Spain Apache Spark Meetup Oct 20, 2015: Spark Streaming, Kafka, MLl...
byChris Fregly
 
Stockholm Spark Meetup Nov 23 2015 Spark After Dark 1.5
byChris Fregly
 
Scotland Data Science Meetup Oct 13, 2015: Spark SQL, DataFrames, Catalyst, ...
byChris Fregly
 
Helsinki Spark Meetup Nov 20 2015
byChris Fregly
 
Zurich, Berlin, Vienna Spark and Big Data Meetup Nov 02 2015
byChris Fregly
 
Practical Data Science Workshop - Recommendation Systems - Collaborative Filt...
byChris Fregly
 
Augury and Omens Aside, Part 1:
 The Business Case for Apache Mesos
byPaco Nathan
 
GalvanizeU Seattle: Eleven Almost-Truisms About Data
byPaco Nathan
 
Tech
byManabuYoneyama
 
Sparksee overview
bySparsity Technologies
 

Similar to Similarity at Scale

PPT
Similarity at scale
byKen Krugler
 
PDF
IRJET- Python Based Machine Learning for Profile Matching
byIRJET Journal
 
PDF
Efficient Similarity Computation for Collaborative Filtering in Dynamic Envir...
byOlivier Jeunen
 
PPT
Similarity & Recommendation - CWI Scientific Meeting - Sep 27th, 2013
byArjen de Vries
 
PDF
IJERD (www.ijerd.com) International Journal of Engineering Research and Devel...
byIJERD Editor
 
PPTX
Finding-Similar-Items-Similarity-of-Sets-and-Collaborative-Filtering.pptx
byUpparalliManjunath
 
PPTX
Dunning ml-conf-2014
byMapR Technologies
 
PDF
Computing Neighbourhoods with Language Models in a Collaborative Filtering Sc...
byDaniel Valcarce
 
PDF
Performance Analysis and Parallelization of CosineSimilarity of Documents
byIRJET Journal
 
PPTX
Finding-Similar-Items ppt mining Massive Datasets
bytina523707
 
PDF
Information retrieval to recommender systems
byData Science Society
 
PPT
Textmining Retrieval And Clustering
byguest0edcaf
 
PPT
Textmining Retrieval And Clustering
byDataminingTools Inc
 
PPT
Textmining Retrieval And Clustering
byDatamining Tools
 
PDF
11.0004www.iiste.org call for paper.on demand quality of web services using r...
byAlexander Decker
 
PDF
4.on demand quality of web services using ranking by multi criteria 31-35
byAlexander Decker
 
PDF
Using Graph Algorithms For Advanced Analytics - Part 4 Similarity 30 graph al...
byTigerGraph
 
PPT
similarities-knn-1.ppt
bysatvikpatil5
 
PPTX
Collaborative filtering common_problems_and_solutions
byVivek Aanand Ganesan
 
PPT
Finding Co-solvers on Twitter, with the Little Help from Linked Data
byMilan Stankovic
 
Similarity at scale
byKen Krugler
 
IRJET- Python Based Machine Learning for Profile Matching
byIRJET Journal
 
Efficient Similarity Computation for Collaborative Filtering in Dynamic Envir...
byOlivier Jeunen
 
Similarity & Recommendation - CWI Scientific Meeting - Sep 27th, 2013
byArjen de Vries
 
IJERD (www.ijerd.com) International Journal of Engineering Research and Devel...
byIJERD Editor
 
Finding-Similar-Items-Similarity-of-Sets-and-Collaborative-Filtering.pptx
byUpparalliManjunath
 
Dunning ml-conf-2014
byMapR Technologies
 
Computing Neighbourhoods with Language Models in a Collaborative Filtering Sc...
byDaniel Valcarce
 
Performance Analysis and Parallelization of CosineSimilarity of Documents
byIRJET Journal
 
Finding-Similar-Items ppt mining Massive Datasets
bytina523707
 
Information retrieval to recommender systems
byData Science Society
 
Textmining Retrieval And Clustering
byguest0edcaf
 
Textmining Retrieval And Clustering
byDataminingTools Inc
 
Textmining Retrieval And Clustering
byDatamining Tools
 
11.0004www.iiste.org call for paper.on demand quality of web services using r...
byAlexander Decker
 
4.on demand quality of web services using ranking by multi criteria 31-35
byAlexander Decker
 
Using Graph Algorithms For Advanced Analytics - Part 4 Similarity 30 graph al...
byTigerGraph
 
similarities-knn-1.ppt
bysatvikpatil5
 
Collaborative filtering common_problems_and_solutions
byVivek Aanand Ganesan
 
Finding Co-solvers on Twitter, with the Little Help from Linked Data
byMilan Stankovic
 

More from DataWorks Summit

PPTX
Data Science Crash Course
byDataWorks Summit
 
PPTX
Floating on a RAFT: HBase Durability with Apache Ratis
byDataWorks Summit
 
PPTX
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
byDataWorks Summit
 
PDF
HBase Tales From the Trenches - Short stories about most common HBase operati...
byDataWorks Summit
 
PPTX
Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...
byDataWorks Summit
 
PPTX
Managing the Dewey Decimal System
byDataWorks Summit
 
PPTX
Practical NoSQL: Accumulo's dirlist Example
byDataWorks Summit
 
PPTX
HBase Global Indexing to support large-scale data ingestion at Uber
byDataWorks Summit
 
PPTX
Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix
byDataWorks Summit
 
PPTX
Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi
byDataWorks Summit
 
PPTX
Supporting Apache HBase : Troubleshooting and Supportability Improvements
byDataWorks Summit
 
PPTX
Security Framework for Multitenant Architecture
byDataWorks Summit
 
PDF
Presto: Optimizing Performance of SQL-on-Anything Engine
byDataWorks Summit
 
PPTX
Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...
byDataWorks Summit
 
PPTX
Extending Twitter's Data Platform to Google Cloud
byDataWorks Summit
 
PPTX
Event-Driven Messaging and Actions using Apache Flink and Apache NiFi
byDataWorks Summit
 
PPTX
Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger
byDataWorks Summit
 
PPTX
Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...
byDataWorks Summit
 
PDF
Computer Vision: Coming to a Store Near You
byDataWorks Summit
 
PPTX
Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark
byDataWorks Summit
 
Data Science Crash Course
byDataWorks Summit
 
Floating on a RAFT: HBase Durability with Apache Ratis
byDataWorks Summit
 
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
byDataWorks Summit
 
HBase Tales From the Trenches - Short stories about most common HBase operati...
byDataWorks Summit
 
Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...
byDataWorks Summit
 
Managing the Dewey Decimal System
byDataWorks Summit
 
Practical NoSQL: Accumulo's dirlist Example
byDataWorks Summit
 
HBase Global Indexing to support large-scale data ingestion at Uber
byDataWorks Summit
 
Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix
byDataWorks Summit
 
Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi
byDataWorks Summit
 
Supporting Apache HBase : Troubleshooting and Supportability Improvements
byDataWorks Summit
 
Security Framework for Multitenant Architecture
byDataWorks Summit
 
Presto: Optimizing Performance of SQL-on-Anything Engine
byDataWorks Summit
 
Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...
byDataWorks Summit
 
Extending Twitter's Data Platform to Google Cloud
byDataWorks Summit
 
Event-Driven Messaging and Actions using Apache Flink and Apache NiFi
byDataWorks Summit
 
Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger
byDataWorks Summit
 
Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...
byDataWorks Summit
 
Computer Vision: Coming to a Store Near You
byDataWorks Summit
 
Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark
byDataWorks Summit
 

Recently uploaded

PDF
Getting Started with Apache Spark: Big Data Made Simple [Free Meetup]
byHaim Michael
 
PDF
Track Phone Location by Number (2026)_ What’s Actually Possible for Free (and...
byEmily Woods
 
PPTX
TechSprint WinterHack — Top 10 Teams Pitching Session we are excited for pit...
bybajpaitusharoffon678
 
PPTX
Beyond the Algorithm: Designing Human-Centric Public Service with AI
byAlan Dix
 
PDF
Dev Dives: Build and deploy agentic automations - the unified way
byUiPathCommunity
 
PPTX
Career-Opportunities in Industrial Arts Grade 8 PPT Lesson 2
byFSBTLEDNathanVince
 
PDF
The Manifesto for AI-enabled Governance !
byYannis Charalabidis
 
PDF
Supercharge Your Copilot-Driven Collaboration with Microsoft 365 Agents SDK
byAntti Koskela
 
PDF
Why Many Smart Device Platforms Fail to Scale?
byPromeraki Developments
 
PDF
AI and Zero Trust: What it takes to do it right
byMark Simos
 
PDF
Rustici Software: eLearning standards in the age of AI
byRustici Software
 
PDF
Track Phone Location via Number (2026)_ What Actually Works, What’s a Scam, a...
byEmily Woods
 
PPTX
Introduction to Artificial Intelligence (AI).pptx
byammar414783
 
PDF
How to build hackthon projects from ZERO!
byishantyadav1111
 
PDF
Mesh WiFi Router: The Smart Solution for Fast, Seamless, Whole-Home Internet
byAeroMesh Systems
 
PDF
Understanding Foldable 3-Wheel Electric Scooters for Everyday Use
byTopmate
 
PDF
The Emergence of Empathic XR: AWE Asia 2026 keynote
byMark Billinghurst
 
PDF
Designing a Blog Using Wordpress
bymarkzubi50
 
PDF
Mount File Systems using UUID and Label - RHCSA (RH134).pdf
byLinuxCert Guru
 
PPTX
TechSprint Inauguration at SJB Institute of Technology held on 18 December 2025
bysuhasspgdg
 
Getting Started with Apache Spark: Big Data Made Simple [Free Meetup]
byHaim Michael
 
Track Phone Location by Number (2026)_ What’s Actually Possible for Free (and...
byEmily Woods
 
TechSprint WinterHack — Top 10 Teams Pitching Session we are excited for pit...
bybajpaitusharoffon678
 
Beyond the Algorithm: Designing Human-Centric Public Service with AI
byAlan Dix
 
Dev Dives: Build and deploy agentic automations - the unified way
byUiPathCommunity
 
Career-Opportunities in Industrial Arts Grade 8 PPT Lesson 2
byFSBTLEDNathanVince
 
The Manifesto for AI-enabled Governance !
byYannis Charalabidis
 
Supercharge Your Copilot-Driven Collaboration with Microsoft 365 Agents SDK
byAntti Koskela
 
Why Many Smart Device Platforms Fail to Scale?
byPromeraki Developments
 
AI and Zero Trust: What it takes to do it right
byMark Simos
 
Rustici Software: eLearning standards in the age of AI
byRustici Software
 
Track Phone Location via Number (2026)_ What Actually Works, What’s a Scam, a...
byEmily Woods
 
Introduction to Artificial Intelligence (AI).pptx
byammar414783
 
How to build hackthon projects from ZERO!
byishantyadav1111
 
Mesh WiFi Router: The Smart Solution for Fast, Seamless, Whole-Home Internet
byAeroMesh Systems
 
Understanding Foldable 3-Wheel Electric Scooters for Everyday Use
byTopmate
 
The Emergence of Empathic XR: AWE Asia 2026 keynote
byMark Billinghurst
 
Designing a Blog Using Wordpress
bymarkzubi50
 
Mount File Systems using UUID and Label - RHCSA (RH134).pdf
byLinuxCert Guru
 
TechSprint Inauguration at SJB Institute of Technology held on 18 December 2025
bysuhasspgdg
 

Similarity at Scale

  • 1.
    Copyright (c) 2014Scale Unlimited. 1 Similarity at Scale Fuzzy matching and recommendations using Hadoop, Solr, and heuristics Ken Krugler Scale Unlimited
  • 2.
    Copyright (c) 2014Scale Unlimited. The Twitter Pitch Wide class of problems that rely on "good" similarity Fast Accurate Scalable Benefit from my mistakes Scale Unlimited - consulting & training Talking about solutions to real problems 2
  • 3.
    Copyright (c) 2014Scale Unlimited. What are similarity problems? Clustering Grouping similar advertisers Deduplication Joining noisy sets of POI data Recommendations Suggesting pages to users Entity resolution Fuzzy matching of people and companies 3
  • 4.
    Copyright (c) 2014Scale Unlimited. What is "Similarity"? Exact matching is easy(er) Accuracy is a given Fast and scalable can still be hard Lots of key/value systems like Cassandra, HBase, etc. Fuzzy matching is harder Two "things" aren't exactly the same Similarity is based on comparing features 4
  • 5.
    Copyright (c) 2014Scale Unlimited. Between two articles? Features could be a bag of words Are these two articles the same? 5 Bosnia is the largest geographic region of the modern state with a moderate continental climate, marked by hot summers and cold, snowy winters. The inland is a geographically larger region and has a moderate continental climate, bookended by hot summers and cold and snowy winters.
  • 6.
    Copyright (c) 2014Scale Unlimited. What about now? Easy to create challenging situations for a person Which is an impossible problem for a computer Need to distinguish between "conceptually similar" and "derived from" 6 Bosnia is the largest geographic region of the modern state with a moderate continental climate, marked by hot summers and cold, snowy winters. Bosnia has a warm European climate, though the summers can be hot and the winters are often cold and wet.
  • 7.
    Copyright (c) 2014Scale Unlimited. Between two records? Features could be field values Are these two people the same? 7 Name Address City State Zip Bob Bogus Robert Bogus 220 3rd Avenue 220 3rd Avenue Seattle Seattle WA WA 98104-2608 98104
  • 8.
    Copyright (c) 2014Scale Unlimited. What about now? Need to get rid of false differences caused by abbreviations How does a computer know what's a "significant" difference? 8 Name Address City State Zip Bob Bogus Robert H. Bogus Apt 102, 3220 3rd Ave 220 3rd Avenue South Seattle Seattle Washington WA 98104
  • 9.
    Copyright (c) 2014Scale Unlimited. Between two users? Features could be... Items a user has bought Are these two users the same? 9 User 1 User 2
  • 10.
    Copyright (c) 2014Scale Unlimited. What about now? Need more generic features E.g. product categories 10 User 1 User 2
  • 11.
    Copyright (c) 2014Scale Unlimited. How to measure similarity? Assuming you have some features for two "things" How does a program determine their degree of similarity? You want a number that represents their "closeness" Typically 1.0 means exactly the same And 0.0 means completely different 11
  • 12.
    Copyright (c) 2014Scale Unlimited. Jaccard Coefficient Ratio of number of items in common / total number of items Where "items" typical means unique values (sets of things) So 1.0 is exactly the same, and 0.0 is completely different 12 Jaccard(A, B) = A!B A"B
  • 13.
    Copyright (c) 2014Scale Unlimited. Cosine Similarity Assume a document only has three unique words cat, dog, goldfish Set x = frequency of cat Set y = frequency of dog Set z = frequency of goldfish The result is a "term vector" with 3 dimensions Calculate cosine of angle between term vectors This is their "cosine similarity" 13
  • 14.
    Copyright (c) 2014Scale Unlimited. Why is scalability hard? Assume you have 8.5 million businesses in the US There are ≈ N^2/2 pairs to evaluate That's 36 trillion comparisons Sometimes you can quickly trim this problem E.g. if you assume the ZIP code exists, and must match Then this becomes about 4 billion comparisons But often you don't have a "magic" field 14
  • 15.
    Copyright (c) 2011-2014Scale Unlimited. All Rights Reserved. Reproduction or distribution of this document in any form without prior written permission is forbidden. DataStax Web Site Page Recommender 15
  • 16.
    Copyright (c) 2014Scale Unlimited. How to recommend pages? Besides manually adding a bunch of links... Which is tedious, doesn't scale well, and gets busy 16
  • 17.
    Copyright (c) 2014Scale Unlimited. Can we exploit other users? Classic shopping cart analysis "Users who bought X also bought Y" Based on actual activity, versus (noisy, skewed) ratings 17
  • 18.
    Copyright (c) 2014Scale Unlimited. What's the general approach? We have web logs with IP addresses, time, path to page 157.55.33.39 - - [18/Mar/2014:00:01:00 -0500] "GET /solutions/nosql HTTP/1.1" A browsing session is a series of requests from one IP address With some maximum time gap between requests Find sessions "similar to" the current user's session Recommend pages from these similar sessions 18
  • 19.
    Copyright (c) 2014Scale Unlimited. How to find similar sessions? Create a Lucene search index with one document per session Each indexed document contains the page paths for one session session-1 /path/to/page1, /path/to/page2, /path/to/page3 session-2 /path/to/pageX, /path/to/pageY Search for paths from the current user's session 19
  • 20.
    Copyright (c) 2014Scale Unlimited. Why is this a search issue? Solr (search in general) is all about similarity Find documents similar to the words in my query Cosine similarity is used to calculate similarity Between the term vector for my query and the term vector of each document 20
  • 21.
    Copyright (c) 2014Scale Unlimited. What's the algorithm? Find sessions similar to the target (current user's) session Calculate similarity between these sessions and the target session Aggregate similarity scores for all paths from these sessions Remove paths that are already in the target session Recommend the highest scoring path(s) 21
  • 22.
    Copyright (c) 2014Scale Unlimited. Why do you sum similarities? Give more weight to pages from sessions that are more similar Pages from more similar sessions are assumed to be more interesting 22 F D B C A Jaccard = 0.2 (1 / 5) Session 2 vs Target Session E D B C A Jaccard = 0.4 (2 / 5) Session 1 vs Target Session D E F 0.6 (0.4 + 0.2) 0.4 0.2 Page Score
  • 23.
    Copyright (c) 2014Scale Unlimited. What are some problems? The classic problem is that we recommend "common" pages 23
  • 24.
    Copyright (c) 2014Scale Unlimited. What are some problems? The classic problem is that we recommend "common" pages E.g. if you haven't viewed the top-level page in your session 23
  • 25.
    Copyright (c) 2014Scale Unlimited. What are some problems? The classic problem is that we recommend "common" pages E.g. if you haven't viewed the top-level page in your session But this page is very common in most of the other sessions 23
  • 26.
    Copyright (c) 2014Scale Unlimited. What are some problems? The classic problem is that we recommend "common" pages E.g. if you haven't viewed the top-level page in your session But this page is very common in most of the other sessions So then it becomes one of the top recommended page 23
  • 27.
    Copyright (c) 2014Scale Unlimited. What are some problems? The classic problem is that we recommend "common" pages E.g. if you haven't viewed the top-level page in your session But this page is very common in most of the other sessions So then it becomes one of the top recommended page But that generally stinks as a recommendation 23
  • 28.
    Copyright (c) 2014Scale Unlimited. Can RowSimilarityJob help? 24 Page 1 Page 2 Page 3 Page 1 Page 2 Page 3 2.1 0.8 2.1 4.5 0.8 4.5
  • 29.
    Copyright (c) 2014Scale Unlimited. Can RowSimilarityJob help? Part of the Mahout open source project 24 Page 1 Page 2 Page 3 Page 1 Page 2 Page 3 2.1 0.8 2.1 4.5 0.8 4.5
  • 30.
    Copyright (c) 2014Scale Unlimited. Can RowSimilarityJob help? Part of the Mahout open source project Takes as input a table of users (one per row) with lists of items 24 Page 1 Page 2 Page 3 Page 1 Page 2 Page 3 2.1 0.8 2.1 4.5 0.8 4.5
  • 31.
    Copyright (c) 2014Scale Unlimited. Can RowSimilarityJob help? Part of the Mahout open source project Takes as input a table of users (one per row) with lists of items Generates an item-item co-occurrence matrix 24 Page 1 Page 2 Page 3 Page 1 Page 2 Page 3 2.1 0.8 2.1 4.5 0.8 4.5
  • 32.
    Copyright (c) 2014Scale Unlimited. Can RowSimilarityJob help? Part of the Mahout open source project Takes as input a table of users (one per row) with lists of items Generates an item-item co-occurrence matrix Values are weights calculated using log-likelihood ratio (LLR) 24 Page 1 Page 2 Page 3 Page 1 Page 2 Page 3 2.1 0.8 2.1 4.5 0.8 4.5
  • 33.
    Copyright (c) 2014Scale Unlimited. Can RowSimilarityJob help? Part of the Mahout open source project Takes as input a table of users (one per row) with lists of items Generates an item-item co-occurrence matrix Values are weights calculated using log-likelihood ratio (LLR) Unsurprising (common) items get low weights 24 Page 1 Page 2 Page 3 Page 1 Page 2 Page 3 2.1 0.8 2.1 4.5 0.8 4.5
  • 34.
    Copyright (c) 2014Scale Unlimited. Can RowSimilarityJob help? Part of the Mahout open source project Takes as input a table of users (one per row) with lists of items Generates an item-item co-occurrence matrix Values are weights calculated using log-likelihood ratio (LLR) Unsurprising (common) items get low weights If we run it on our data, where users = sessions and items = pages 24 Page 1 Page 2 Page 3 Page 1 Page 2 Page 3 2.1 0.8 2.1 4.5 0.8 4.5
  • 35.
    Copyright (c) 2014Scale Unlimited. Can RowSimilarityJob help? Part of the Mahout open source project Takes as input a table of users (one per row) with lists of items Generates an item-item co-occurrence matrix Values are weights calculated using log-likelihood ratio (LLR) Unsurprising (common) items get low weights If we run it on our data, where users = sessions and items = pages We get page-page co-occurrence matrix 24 Page 1 Page 2 Page 3 Page 1 Page 2 Page 3 2.1 0.8 2.1 4.5 0.8 4.5
  • 36.
    Copyright (c) 2014Scale Unlimited. How to use co-occurrence? 25 Page 1 Page 2 Page 3 Page 1 Page 2 Page 3 2.1 0.8 2.1 4.5 0.8 4.5 Related Pages Page 1 Page 2 Page 3 Page 2 Page 1, Page 3 Page 2
  • 37.
    Copyright (c) 2014Scale Unlimited. How to use co-occurrence? Convert the matrix into an index 25 Page 1 Page 2 Page 3 Page 1 Page 2 Page 3 2.1 0.8 2.1 4.5 0.8 4.5 Related Pages Page 1 Page 2 Page 3 Page 2 Page 1, Page 3 Page 2
  • 38.
    Copyright (c) 2014Scale Unlimited. How to use co-occurrence? Convert the matrix into an index Each row is one Lucene document 25 Page 1 Page 2 Page 3 Page 1 Page 2 Page 3 2.1 0.8 2.1 4.5 0.8 4.5 Related Pages Page 1 Page 2 Page 3 Page 2 Page 1, Page 3 Page 2
  • 39.
    Copyright (c) 2014Scale Unlimited. How to use co-occurrence? Convert the matrix into an index Each row is one Lucene document Drop any low-scoring entries 25 Page 1 Page 2 Page 3 Page 1 Page 2 Page 3 2.1 0.8 2.1 4.5 0.8 4.5 Related Pages Page 1 Page 2 Page 3 Page 2 Page 1, Page 3 Page 2
  • 40.
    Copyright (c) 2014Scale Unlimited. How to use co-occurrence? Convert the matrix into an index Each row is one Lucene document Drop any low-scoring entries Create list of "related" pages 25 Page 1 Page 2 Page 3 Page 1 Page 2 Page 3 2.1 0.8 2.1 4.5 0.8 4.5 Related Pages Page 1 Page 2 Page 3 Page 2 Page 1, Page 3 Page 2
  • 41.
    Copyright (c) 2014Scale Unlimited. How to use co-occurrence? Convert the matrix into an index Each row is one Lucene document Drop any low-scoring entries Create list of "related" pages 25 Page 1 Page 2 Page 3 Page 1 Page 2 Page 3 2.1 0.8 2.1 4.5 0.8 4.5 Related Pages Page 1 Page 2 Page 3 Page 2 Page 1, Page 3 Page 2
  • 42.
    Copyright (c) 2014Scale Unlimited. How to use co-occurrence? Convert the matrix into an index Each row is one Lucene document Drop any low-scoring entries Create list of "related" pages Search in Related Pages field 25 Page 1 Page 2 Page 3 Page 1 Page 2 Page 3 2.1 0.8 2.1 4.5 0.8 4.5 Related Pages Page 1 Page 2 Page 3 Page 2 Page 1, Page 3 Page 2
  • 43.
    Copyright (c) 2014Scale Unlimited. How to use co-occurrence? Convert the matrix into an index Each row is one Lucene document Drop any low-scoring entries Create list of "related" pages Search in Related Pages field Using pages from current session 25 Page 1 Page 2 Page 3 Page 1 Page 2 Page 3 2.1 0.8 2.1 4.5 0.8 4.5 Related Pages Page 1 Page 2 Page 3 Page 2 Page 1, Page 3 Page 2
  • 44.
    Copyright (c) 2014Scale Unlimited. How to use co-occurrence? Convert the matrix into an index Each row is one Lucene document Drop any low-scoring entries Create list of "related" pages Search in Related Pages field Using pages from current session So Page 2 recommends Page 1 & 3 25 Page 1 Page 2 Page 3 Page 1 Page 2 Page 3 2.1 0.8 2.1 4.5 0.8 4.5 Related Pages Page 1 Page 2 Page 3 Page 2 Page 1, Page 3 Page 2
  • 45.
    Copyright (c) 2011-2014Scale Unlimited. All Rights Reserved. Reproduction or distribution of this document in any form without prior written permission is forbidden. EWS Entity Resolution 26
  • 46.
    Copyright (c) 2014Scale Unlimited. What is Early Warning? Early Warning helps banks fight fraud It's owned by the top 5 US banks And gets data from 800+ financial institutions So they have details on most US bank accounts When somebody signs up for an account They need to quickly match the person to "known entities" And derive a risk score based on related account details 27
  • 47.
    Copyright (c) 2014Scale Unlimited. Why do they need similarity? Assume you have information on 100s of millions of entities Name(s), address(es), phone number(s), etc. And often a unique ID (Social Security Number, EIN, etc) Why is this a similarity problem? Data is noisy - typos, abbreviations, partial data People lie - much fraud starts with opening an account using bad data 28
  • 48.
    Copyright (c) 2014Scale Unlimited. How does search help? We can quickly build a list of candidate entities, using search Query contains field data provided by the client bank Significantly less than 1 second for 30 candidate entities Then do more precise, sophisticated and CPU-intensive scoring The end result is a ranked list of entities with similarity scores Which then is used to look up account status, fraud cases, etc. 29
  • 49.
    Copyright (c) 2014Scale Unlimited. What's the data pipeline? Incoming data is cleaned up/normalized in Hadoop Simple things like space stripping Also phone number formatting ZIP+4 expansion into just ZIP plus full Other normalization happens inside of Solr This gets loaded into Cassandra tables And automatically indexed by Solr, via DataStax Enterprise 30 ZIP+4 Terms 95014-2127 95014, 2127 Phone Terms 4805551212 480, 5551212
  • 50.
    Copyright (c) 2014Scale Unlimited. What's the Solr setup? Each field in the index has very specific analysis Simple things like normalization Synonym expansion for names, abbreviations Split up fields so partial matches work At query time we can weight the importance of each field Which helps order the top N candidates similar to their real match scores E.g. an SSN matching means much more than a first name matching 31
  • 51.
    Copyright (c) 2011-2014Scale Unlimited. All Rights Reserved. Reproduction or distribution of this document in any form without prior written permission is forbidden. Batch Similarity 32
  • 52.
    Copyright (c) 2014Scale Unlimited. Can we do batch similarity? Search works well for real-time similarity But batch processing at scale maxes out the search system We can use two different techniques with Hadoop for batch SimHash - good for text document similarity Parallel Set-Similarity Joins - good for record similarity 33
  • 53.
    Copyright (c) 2014Scale Unlimited. What is SimHash? Assume a document is a set of (unique) words Calculate a hash for each word Probability that the minimum hash is the same for two documents... ...is magically equal to the Jaccard Coefficient 34 Term Hash bosnia is the largest geographic 78954874223 53466156768 5064199193 3193621783 -5718349925
  • 54.
    Copyright (c) 2014Scale Unlimited. What is a SimHash workflow? Calculate N hash values Easy way is to use the N smallest hash values Calculate number of matching hash values between doc pairs (M) Then the Jaccard Coefficient is ≈ M/N Only works if N is much smaller than # of unique words in docs Implementation of this in cascading.utils open source project https://github.com/ScaleUnlimited/cascading.utils 35
  • 55.
    Copyright (c) 2014Scale Unlimited. What is Set-Similarity Join? Joining records in two sets that are "close enough" aka "fuzzy join" Requires generation of "tokens" from record field(s) Typically words from text Simple implementation has three phases First calculate counts for each unique token value Then output <token, record> for N most common tokens of each record Group by token, compare records in each group 36
  • 56.
    Copyright (c) 2014Scale Unlimited. How does fuzzy join work? For two records to be "similar enough"... They need to share one of their common tokens Generalization of the ZIP code "magic field" approach Basic implementation has a number of issues Passing around copies of full record is inefficient Too-common tokens create huge groups for comparison Two records compared multiple times 37
  • 57.
    Copyright (c) 2011-2014Scale Unlimited. All Rights Reserved. Reproduction or distribution of this document in any form without prior written permission is forbidden. Summary 38
  • 58.
    Copyright (c) 2014Scale Unlimited. The Net-Net 39
  • 59.
    Copyright (c) 2014Scale Unlimited. The Net-Net Similarity is a common requirement for many applications 39
  • 60.
    Copyright (c) 2014Scale Unlimited. The Net-Net Similarity is a common requirement for many applications Recommendations 39
  • 61.
    Copyright (c) 2014Scale Unlimited. The Net-Net Similarity is a common requirement for many applications Recommendations Entity matching 39
  • 62.
    Copyright (c) 2014Scale Unlimited. The Net-Net Similarity is a common requirement for many applications Recommendations Entity matching Combining Hadoop with search is a powerful combination 39
  • 63.
    Copyright (c) 2014Scale Unlimited. The Net-Net Similarity is a common requirement for many applications Recommendations Entity matching Combining Hadoop with search is a powerful combination Scalability 39
  • 64.
    Copyright (c) 2014Scale Unlimited. The Net-Net Similarity is a common requirement for many applications Recommendations Entity matching Combining Hadoop with search is a powerful combination Scalability Performance 39
  • 65.
    Copyright (c) 2014Scale Unlimited. The Net-Net Similarity is a common requirement for many applications Recommendations Entity matching Combining Hadoop with search is a powerful combination Scalability Performance Flexibility 39
  • 66.
    Copyright (c) 2014Scale Unlimited. Questions? Feel free to contact me http://www.scaleunlimited.com/contact/ Take a look at Pat Ferrel's Hadoop + Solr recommender http://github.com/pferrel/solr-recommender Check out Mahout http://mahout.apache.org Read paper & code for fuzzyjoin project http://asterix.ics.uci.edu/fuzzyjoin/ 40