Fighting fraud: finding duplicates at scale (Highload+ 2019)
•
2 likes•466 views
The document discusses duplicate detection in online marketplaces with large amounts of user-generated content. It describes a two-step framework for finding duplicate listings: candidate selection to identify potentially duplicate pairs, followed by candidate scoring using machine learning to identify true duplicates. Key aspects include using category, location, seller data, and image hashes to select candidate pairs, and training ML models on text and image similarity features to classify pairs as duplicates or not. Elasticsearch is used to index hashes at scale for fuzzy matching of image duplicates.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Fighting fraud: finding duplicates at scale (Highload+ 2019)
Similar to Fighting fraud: finding duplicates at scale (Highload+ 2019) (20)
More from Alexey Grigorev
More from Alexey Grigorev (20)
Recently uploaded
Recently uploaded (20)
Fighting fraud: finding duplicates at scale (Highload+ 2019)
- 1. Борьба с мошенниками Поиск дубликатов среди сотен миллионов объявлений Алексей Григорьев
- 2. Fighting Fraud Finding Duplicates at Scale Alexey Grigorev
- 3. About me
- 16. Disclaimer It’s a simplification and doesn’t show all the details form the actual system we use at OLX
- 18. Plan ● User generated content ○ Fraud and duplicates ○ Content moderation systems ● Duplicate detection framework ○ Step 1: Selecting candidates ○ Step 2: Scoring candidates with Machine Learning ○ Image hashes ● Implementation ○ Elasticsearch ○ Image index system
- 19. User generated content Such description. So much text
- 20. User generated content Such description. So much text Such description. So much text Such description. So much text
- 22. Problems: ● Illegal content ● NSFW content ● Duplicates ● Spam ● Fraud
- 23. FraudDuplicates Two sides of the same coin * I don’t necessarily think that dogecoin is fraud
- 24. Such description So much text
- 25. Such description So much text
- 26. Such description So much text
- 28. Such good description, so better text
- 29. Such goud description, so better text
- 30. Such goud description, so better text
- 32. 100$ deposit
- 43. How to prevent it?
- 44. Content moderation Such description So much text
- 45. Content moderation ML Such description So much text Automatic moderation system
- 46. Content moderation ML Such description So much text AcceptAutomatic moderation system Reject
- 47. Content moderation ML Such description So much text Accept Reject Moderation queue MP Automatic moderation system Moderation panel Accept Reject Moderators
- 48. ML Such description So much text Accept Reject Moderation queue Automatic moderation system Duplicate detection Forbidden items Other ML models
- 49. ML Such description So much text Accept Reject Moderation queue Automatic moderation system Duplicate detection Forbidden items Other ML models
- 50. Plan ● User generated content ○ Fraud and duplicates ○ Content moderation systems ● Duplicate detection framework ○ Step 1: Selecting candidates ○ Step 2: Scoring candidates with Machine Learning ○ Image hashes ● Implementation ○ Elasticsearch ○ Image index system
- 51. Duplicate detection framework Candidate Selection step Candidate Scoring step find candidate duplicates (10-200) get real duplicates (0-50) Step 1 Step 2 https://www.slideshare.net/AlexeyGrigorev/duplicates-everywhere-berlin
- 52. Why? ● Cannot compare each item with each ● 1000 items ⇒ 1000 x 999 / 2 = 499 500 pairs for comparison ● Real datasets: millions! (avito: 58mln, olx.ua: 13mln) Candidate Selection step Candidate Scoring step find candidate duplicates (10-200) get real duplicates (0-50) Step 1 Step 2
- 53. How? ● Domain knowledge (heuristics) ● Information retrieval techniques ● Approximate knn Candidate Selection step Candidate Scoring step find candidate duplicates (10-200) get real duplicates (0-50) Step 1 Step 2
- 54. Candidate selection ● Category ● City / district ● Seller id ● IP address of the seller ● Device signature
- 55. Candidate Selection step Candidate Scoring step find candidate duplicates (10-200) get real duplicates (0-50) Step 1 Step 2 Machine Learning!
- 56. Duplicate Not duplicate Moderation queue MP Moderation panel Accept Reject Moderators
- 57. Machine Learning ID1 ID2 Features Label 1 2 [0, 1, ..., 5] 1 1 4 [2, 0, ..., 3] 0 2 7 [3, 1, ..., 3] 0 k 5 [5, 3, ..., 8] 1 Feature engineering ID1 ID2 Label 1 2 1 1 4 0 2 7 0 k 5 1 Model Tune F1/Precision/Recall
- 58. Features Pairwise distances/similarities ● |price1 - price2| ● min(price1, price2) / max(price1, price2) ● dist(loc1, loc2) ● same(ip1, ip2) ● same(loc_id1, loc_id2) ● same(category1, category2) ● |len(title1) - len(title2)| ● |len(images1) - len(images2)|
- 59. Features Pairwise distances/similarities ● Cosine between titles (Bag of words, TF-IDF) ● Cosine between description (Bag of words, TF-IDF) ● Word2Vec
- 60. Bag of Words Titles: ● “Selling Pixel” ● “iPhone XS” ● “Samsung X9”
- 61. Bag of Words sell, pixel, iphone, xs, samsung, x9 Titles: ● “Selling Pixel” ● “iPhone XS” ● “Samsung X9”
- 62. Titles: ● “Selling Pixel” ● “iPhone XS” ● “Samsung X9” Bag of Words sell pixel iphone xs samsung s9 1 1 0 0 0 0 0 0 1 1 0 0 0 0 0 0 1 1
- 63. Titles: ● “Selling Pixel” ● “iPhone XS” ● “Samsung X9” TF-IDF sell pixel iphone xs samsung s9 0.001 0.3 0 0 0 0 0 0 0.2 0.8 0 0 0 0 0 0 0.1 0.9
- 64. Text features: cosine similarity doc1 doc2 θ angle between vectors buy sell vectorizer = TfidfVectorizer() vectorizer.fit(left + right) X_left = vectorizer.transform(left) X_right = vectorizer.transform(right) sim = X_left.multiply(X_right).sum(axis=1) In Scikit-Learn:
- 65. Image hashes ● md5: cryptographic hash ● dhash, phash, whash: Perceptive hashes 94088af86c038327 14ee7fe587860078a1109033318bd986
- 66. Image hashes ● md5: cryptographic hash ● dhash, phash, whash: Perceptive hashes 94088af86c038327 14ee7fe587860078a1109033318bd986 94088af86c038327 07aaedb9b75e88a6051184f01be5cc50
- 67. Dhash: difference hash Read as b/w image, resize Get numpy array Difference between adjacent cells https://www.kaggle.com/iezepov/get-hash-from-images-slightly-daster/code
- 68. 149 168 145 131 134 111 115 114 108 198 192 162 135 104 137 128 108 97 158 165 151 117 111 133 130 139 115 79 95 132 151 180 212 189 158 124 91 47 57 90 67 81 165 142 110 104 80 63 53 43 34 20 42 101 110 113 109 92 79 53 27 59 102 114 111 110 112 108 67 73 90 103 149 168 145 131 134 111 115 114 108 198 192 162 135 104 137 128 108 97 158 165 151 117 111 133 130 139 115 79 95 132 151 180 212 189 158 124 91 47 57 90 67 81 165 142 110 104 80 63 53 43 34 20 42 101 110 113 109 92 79 53 27 59 102 141 111 110 112 108 67 73 90 103 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8
- 69. 19 -23 -15 4 -23 4 -1 -6 -6 -30 -27 -31 33 -9 -20 -11 7 -14 -34 -6 22 -3 9 -24 16 37 19 29 32 -23 -31 -34 -44 10 33 -23 14 84 -23 -32 -24 -17 -10 -10 -9 -14 22 59 3 -3 -18 -13 -26 -26 32 43 -3 -4 2 -4 -41 6 17 13 19 -23 -15 4 -23 4 -1 -6 -6 -30 -27 -31 33 -9 -20 -11 7 -14 -34 -6 22 -3 9 -24 16 37 19 29 32 -23 -31 -34 -44 10 33 -23 14 84 -23 -32 -24 -17 -10 -10 -9 -14 22 59 3 -3 -18 -13 -26 -26 32 43 -30 -4 2 -4 -41 6 17 13 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8
- 70. TRUE FALSE FALSE TRUE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE TRUE FALSE FALSE FALSE TRUE FALSE TRUE FALSE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE TRUE TRUE FALSE FALSE TRUE FALSE FALSE TRUE TRUE TRUE TRUE FALSE FALSE TRUE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE TRUE FALSE FALSE FALSE TRUE FALSE TRUE FALSE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE TRUE TRUE FALSE FALSE TRUE FALSE FALSE TRUE TRUE TRUE 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8
- 71. TRUE FALSE FALSE TRUE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE TRUE FALSE FALSE FALSE TRUE FALSE TRUE FALSE TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE FALSE FALSE FALSE FALSE FALSE TRUE TRUE FALSE FALSE TRUE FALSE FALSE TRUE TRUE TRUE 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 148 94 8 08 138 8a 248 f8 108 64 3 03 131 83 39 27 94088af86c038327
- 72. Features: hashes ● Number of images with same md5, phash, dhash, etc ● Distances between hashes: min, avg, max 94088af86c038327 94088af86c038328 1 0 0 0 0 0 1 1 0 0 1 0 0 1 1 1 1 0 0 0 0 0 1 1 0 0 1 0 1 0 0 0 Distance is 4 bit
- 73. Candidate selection ● Category ● City / district ● Seller id ● IP address of the seller ● Device signature ● Image hashes
- 75. Image embeddings These embeddings capture semantic similarity between images
- 79. Image embeddings CNN Dim 1k+ Dim 100 SVD* 36a93c34a3abff LSH * TruncatedSVD works best https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html
- 80. LSH: Random Projections ● Close in the original space ⇒ close in the projection ● Far in the original space ⇒ far in the projection
- 81. Generate the projection vectors once and store them somewhere LSH: Random Projections https://www.slideshare.net/AlexeyGrigorev/duplicates-everywhere Use the vectors to reduce the dimensionality and compute the hash Store the hash in the database
- 83. Plan ● User generated content ○ Fraud and duplicates ○ Content moderation systems ● Duplicate detection framework ○ Step 1: Selecting candidates ○ Step 2: Scoring candidates with Machine Learning ○ Image hashes ● Implementation ○ Elasticsearch ○ Image index system
- 84. Why Elasticsearch? ● Well-known, convenient, stable and scalable inverted index (thanks, Lucene!) 1 00fc 2 12ec 3 00fc 4 ebe4 5 7a1f 6 00fc 7 8ef4 8 12ec 00fc 1 3 6 12ec 2 8 ebe4 4 7a1f 5 8ef4 7 Direct index Inverted index ImageID Hash Hash ImageID
- 85. Elasticsearch for hashes { "_id": "cafebabe", "_source": { "title": "new iphone" , "description": "new iphone almost not used" , "hashes": ["94088af86c038327", ... ] } } "94088af86c038327" "query": { "bool": { "must": [{ "term": { "hashes": "94088af86c038327" } ]} } }
- 86. Elasticsearch for hashes "94088af86c038327" "1:9408 2:8af8 3:6c03 4:8327" Fuzzy? Chunk the hash!
- 87. Elasticsearch for hashes "94088af86c038327" "1:9408 2:8af8 3:6c03 4:8327" { "_id": "cafebabe", "_source": { "title": "new iphone" , "description": "new iphone almost not used" , "hashes": ["1:9408 2:8af8 3:6c03 4:8327", ... ] } } Let elasticsearch treat it as usual tokens using e.g. whitespace tokenizer "query": { "match": { "hashes": { "query": "1:9408 2:8af8 3:6c03 4:8327" } } }
- 88. Elasticsearch for hashes "query": { "match": { "hashes": { "query": "1:9408 2:8af8 3:6c03 4:8327" } } } ● 1:9408 2:8af8 3:6c03 4:8327 ● 1:9408 2:8af8 3:6c03 4:8238 ● 1:9408 2:8af8 3:6323 4:8327 ● 1:9408 2:34f4 3:6c03 4:8327 ● 1:9408 2:b3af 3:6c03 4:31eb First, exact matches Then 3 out of 4 Then 2 out of 4
- 89. Elasticsearch for hashes “More like this” queries { "query": { "more_like_this": { "like": { "_index": "listings", "_type": "_doc", "_id": "cafebabe" }, "max_query_terms": 100, "fields": ["title^2", "description" , "hashes"] } } } { "_id": "cafebabe", "_source": { "title": "new iphone" , "description": "new iphone almost not used" , "hashes": ["1:9408 2:8af8 3:6c03 4:8327", ... ] } }
- 90. How to implement it?
- 91. Image index (simplified) s3 Such description So much text
- 93. Image index (simplified) s3 ObjectCreated { "eventName": "ObjectCreated:Put", "s3": { "bucket": { "name": "pictures" }, "object": { "key": "doge.jpg" } } }
- 94. Image index (simplified) s3 ObjectCreated Hash calculation
- 95. Image index (simplified) s3 ObjectCreated Hash calculation
- 96. Image index (simplified) s3 ObjectCreated Hash calculation https://pypi.org/project/ImageHash/
- 97. Image index (simplified) hashes Hash calculation { "dhash": "9687678c367b7b3a", "phash": "ad60ad89b54b0d3d", "whash": "fbf3804003199f9f" }
- 98. Image index (simplified) hashes Ingestor Hash calculation ES
- 99. Image index (simplified) hashes Ingestor Hash calculation ESs3 ObjectCreated
- 100. It scalez! 231 rps ~10 mln~8 mln~8 mln
- 101. How about deletes?
- 102. Image index (still simplified) hashes Ingestor Hash calculation ES s3 ObjectCreated ObjectDeleted Ingestor
- 103. How about neural networks?
- 105. CNN+LSH Options for deploying image models: ● Lambda
- 106. CNN+LSH Options for deploying image models: ● Lambda ○ Tricky to do: HUGE TF binaries ○ 66 USD per 1 mln ○ Worth when load is low (< 1 mln)
- 107. CNN+LSH Options for deploying image models: ● Lambda ○ Tricky to do: HUGE TF binaries ○ 66 USD per 1 mln ○ Worth when load is low (< 1 mln) ● Kubernetes ○ Easier to do: docker + existing cluster ○ More expensive for low load ○ Better for 1+ mln images per day
- 108. CNN+LSH Options for deploying image models: ● Lambda ○ Tricky to do: HUGE TF binaries ○ 66 USD per 1 mln ○ Worth when load is low (< 1 mln) ● Kubernetes ○ Easier to do: docker + existing cluster ○ More expensive for low load ○ Better for 1+ mln images per day
- 109. Putting it all together
- 110. Such description So much text
- 111. Such description So much text s3 ES Hashes
- 112. ML Such description So much text Automatic moderation system s3 ES Hashes
- 113. ML Such description So much text Automatic moderation system s3 ES Duplicate detection system Hashes
- 114. ML Such description So much text Automatic moderation system s3 ES Duplicate detection system Hashes Accept Reject
- 115. ML Such description So much text MP Automatic moderation system Moderation panel Accept Reject Moderators s3 ES Duplicate detection system Hashes Accept Reject Moderation queue
- 116. ML Such description So much text MP Automatic moderation system Moderation panel Accept Reject Moderators s3 ES Duplicate detection system Hashes Accept Reject Moderation queue
- 117. Plan ● User generated content ○ Fraud and duplicates ○ Content moderation systems ● Duplicate detection framework ○ Step 1: Selecting candidates ○ Step 2: Scoring candidates with Machine Learning ○ Image hashes ● Implementation ○ Elasticsearch ○ Image index system
- 118. Summary ● Fraud and duplicates often come together ● Use heuristics to find duplicate candidates and ML to find duplicates ● Image hashes is a good and easy way to find duplicate images ● Neural networks can be used for hashing as well ● Elasticsearch is good for finding duplicates (inverted index!) ● AWS Lambda can scale up and down with no human involvement ● Simple things (e.g. hashes) - better in AWS Lambda ● Complex heavy things (e.g. neural nets) - Kubernetes
- 119. Thanks!
- 120. Contact information ● http://alexeygrigorev.com & contact@alexeygrigorev.com ● https://github.com/alexeygrigorev ● https://www.linkedin.com/in/agrigorev ● https://www.slideshare.net/AlexeyGrigorev
- 121. Feedback (and link to the slides!) https://forms.gle/XnVJk8QCeW9gTebP9
- 122. Questions?