Real Time Fuzzy Matching With Spark and Elasticsearch
•
0 likes•106 views
Presentation at GIDS 2016 - Great Indian Developer Summit
Recommended
More Related Content
Viewers also liked
Similar to Real Time Fuzzy Matching With Spark and Elasticsearch
Similar to Real Time Fuzzy Matching With Spark and Elasticsearch (20)
Recently uploaded
Recently uploaded (20)
Real Time Fuzzy Matching With Spark and Elasticsearch
- 1. Real Time Fuzzy Matching With Spark and ElasticSearch
- 2. BFSI
- 4. Sanctions Screening PEP HMT OFAC SDN ..and many others
- 5. However ... 7TH OF TIR 7TH OF TIR COMPLEX 7TH OF TIR INDUSTRIAL COMPLEX 7TH OF TIR INDUSTRIES 7TH OF TIR INDUSTRIES OF ISFAHAN/ESFAHAN SEVENTH of Tir
- 7. Directory Listings De Dew Drops, Shop no - A-152, super mart 1, Gurgaon - 122001, DLF Phase 4 DewDrop Florist, A 152, DLF City Phase 4, Near Galleria Market, Super Mart 1
- 8. Ecommerce Cherry Mobile Amethyst Android 4.2 Jelly Bean (Black) with Free Smart and Globe SIM Cherry Mobile Amethyst (White) with 1 Smart SIM CHERRY MOBILE AMETHYST + 1 SMART SIM Cherry Mobile Amethyst Android 4.2 Jelly Bean Cherry Mobile Amethyst (White) with 1 Samsung Galaxy V CHERRY MOBILE AMETHYST + 1 SAMSUNG GALAXY V. + 1 SMART AND GLOBE SIM
- 9. Government of .. ● Benefit rollouts ● Surveillance ● Licenses ● Linking NPR with Passport
- 10. 360 view ID Company Name Project 12345 UBM Asia Dave Chan HK - Fine Jewellery 13222 UBM A Dave C HK - Fashion Jewellery 15656 UBM Davechan HK - Beauty 14456 ubmAsia Mr. Dave CChan HK - Fine Jewellery
- 11. “In order to be irreplaceable, one must always be different.” ― Coco Chanel
- 12. Other uses ● Cross selling ● Data Quality ● Vendor consolidation ● Master Data Management ● CRM Deduplication
- 13. Challenges ● Discovering and maintaining rules is extremely tough ● Custom coding and domain specific logic makes maintenance a nightmare ● No one size fits all, big custom implementations needed every time even after using existing tools
- 14. Challenges.. ● High Data volumes ● Each record has multiple dimensions ● Exact matches are rare ● Comparing each record with every other is not possible ● Languages have unique issues
- 15. Lets start wishing... ● Data variety ● Scalable ● No manual configuration of rules or algorithms ● Multi language ● Real time
- 16. Our Approach - Learn from the data - Divide the load
- 17. Reifier Workflow Configure data Reifier Interactive Learner Linked Result Have training data? Reifier Match Yes No
- 18. 1. Select Data
- 19. 2. Field Selection and Stop Words
- 20. Strata Hadoop World Singapore 2015 3. Choose Training Set
- 21. Strata Hadoop World Singapore 2015 4. Run the Spark Job
- 22. Strata Hadoop World Singapore 2015 5. Enjoy the results
- 23. Strata Hadoop World Singapore 2015 At the beginning: (Without Chinese Stopped words) 亚洲博闻有限公司 Dave Chan 亚洲华乐有限公司 David Chan In this case, the similarity between 2 records is very high What if we include the stopped word? (亚 洲,有限公司) 博闻 Dave Chan 华乐 David Chan Company names for these records now are not matched at all and the system will not group them together. Fuzzy Match in Reifier – Stopped word
- 28. Spark Benefits ● Distributed ● Scalable ● Fast ● Machine Learning ● Sampling ● No need to orchestrate multiple jobs
- 29. Real Time Spark + ElasticSearch
- 30. Advantages ● Point and Shoot - Zero config ● Learning similarity definitions from data ■ - No hard coding of business rules ■ - Domain agnostic ■ - Handle multiple languages (English, Chinese, Japanese, Thai)
- 31. Advantages ● Scalability ● Real time as well as batch