Outline catalogue fifth_elephant_2019_deepak_sharma_v0.2
•Download as PPTX, PDF•
2 likes•60 views
This document summarizes an approach to building catalogues for small and medium enterprises by aggregating and structuring product descriptions from existing customers. It describes three iterations to improve the approach: 1) using blocking and learnable similarity functions, 2) extracting attributes like brand and category for improved blocking, and 3) combining lexical and distributed features to calculate similarity within blocks. Active learning is used to label sample clusters and refine the clustering algorithms. Key challenges include addressing variability in product descriptions and scaling the approach to handle large volumes of data.
Recommended
Recommended
More Related Content
Similar to Outline catalogue fifth_elephant_2019_deepak_sharma_v0.2
Similar to Outline catalogue fifth_elephant_2019_deepak_sharma_v0.2 (20)
Recently uploaded
Recently uploaded (20)
Outline catalogue fifth_elephant_2019_deepak_sharma_v0.2
- 1. Building Catalogues for MSMEs Deepak Sharma, Bhanu Pratap
- 2. Goal • Provide millions of small enterprises access to structured catalogue • Faster improved search • Aggregation • Services like HSN, Tax Rate etc
- 3. Approach • Bootstrap catalogue from raw product descriptions available with existing customers • Existing customers create “masters” for inventory management and invoicing • These masters are Product Descriptions and Hierarchies but highly specific to the customer (Since Tally imposes minimal structure on this ontology) • GK Adv mat 10 pcs (Good Knight Advance Mat) • Ks - Deo on. 225 (Kamasutra Deodorant) • Dab Real Pin Rs99 (Dabur Real Pineapple) • Extremely rich data covering breadth and depth of SKUs. Things we can not find elsewhere • Guru Essence EDP 100ml, • Lal Prive Rose Royale 100ml
- 4. Building Catalogue GK Adv Fast Card mat 10 pcs Good Night Adv mat 10 pcs Good Knight Advance 10 pcs Good Knight Advance mosquito mat 10 pcs Good Knight Advance refill 45ml GK Adv active+ liquid refill 45ml Title: Good Knight Advanced Active+ Liquid Refill Category: Mosquito Repellent > Liquid > Refill Brand : Good Knight Unit of Measurement: {45, ml} Title: Good Knight Advanced Mosquito Mat Category: Mosquito Repellent > Mat Brand : Good Knight Unit of Measurement: {10, pcs} Published Stock Item in Catalogue Published Stock Item in Catalogue
- 5. Problem and Challenges • Summarizing the problem • Cluster/ Deduplicate raw input data and resolve key attributes to Publish a Stock Item • Deduplicate raw product data • Resolve key attributes in the deduplicated cluster • Challenges • Highly contextual Product Descriptions with very little common grammar • Uncommon abbreviations, transliterations, misspellings etc of attributes • Lack of reliable attribute information unlike product data from ecommerce websites • High volume with high product variation • The reference data against which we need to resolve also needs to be built simultaneously
- 6. First Iteration • Use the dedupe algorithm provided in the PhD dissertation of Mikhail Bilenko Learnable Similarity Functions and Their Application to Record Linkage and Clustering • Essentially dedupe using edit distance based learnable similarity methods • The first step is to roughly group data into similar SKUs and only learn and calculate similarity within groups/blocks • A different similarity function for each block is learnt • We tried various mechanisms to block, a challenge is that no other attribute (brand, category) is explicitly present in the data which cannot be used for blocking • Token based blocking • Word vector based clustering and using that to do blocking
- 7. First Pipeline Segment Identification Blocking/High Level Clustering Learn Similarity Model within Each Linkage Dedupe Output Sample and Manually Label Active Learning Raw SKUs • Token Based • Clustering using Word Vectors
- 8. GK Adv Fast Card mat 10 pcs Good Night Adv mat 10 pcs Good Knight Advance 10 pcs Good Knight Advance mosquito mat 10 pcs Good Knight Advance refill 45ml GK Adv active+ liquid refill 45ml Good Knight mosquito mat 10 pcs A GOOD POTENTIAL BLOCK In reality what we get Good Knight Advance 10 pcs Good Knight Advance mosquito mat 10 pcs Good Knight Advance refill 45ml Hair and Care Advance 45ml Challenges with Blocking
- 9. Lexical Similarity • Another key challenge is that a lexical paired HMM model was insufficient as it can not take into account the context required to find similarity Good Knight Advance refill Good Knight Advance mat Quite close in edit distance but quite different SKUs GK Adv mat Good Knight Advance mat Quite different in edit distance but actually same SKUS
- 10. A few things we learned • Blocking requires to be more precise • Since attributes like Category can be naturally grouping we started thinking about extracting attributes • Blocking step needs to scale up. For example, the clustering we were working on uses clusters as blocking but this requires O(N2) computations • Learnable lexical similarity is not able to model the variations in the SKUs. Similarity requires a representation of the SKU which can take into account context
- 11. Second Iteration – Improving blocking with attribute extraction • Extract attributes like brand and category from Product Title and use them for blocking • We built our own NER model using BiLSTM-CRF model to detect attributes • https://arxiv.org/pdf/1806.01264.pdf, Open Attribute Value Extraction from Product Profiles, Guineng Zheng et. Al. • We use word2vec representation of the attributes to cluster SKUs to be used for blocking • Use hashing techniques to allow faster neighbourhood search from which clusters are evolved. • We use LSH to cluster and block
- 12. GK Adv Fast Card mat 10 pcs Good Night Adv mat 10 pcs Good Knight Advance 10 pcs Good Knight Advance mosquito mat 10 pcs Good Knight Advance refill 45ml GK Adv active+ liquid refill 45ml Good Knight mosquito mat 10 pcs GK Adv Fast Card mat 10 pcs Brand Category measurement Good Night Adv mat 10 pcs Brand Category measurement Good Knight Advance mosquito mat 10 pcs Brand Category measurement
- 13. Second Pipeline Learn Similarity Model within Each Linkage Dedupe Output Sample and Manually Label Active Learning Raw SKUs Segment Identifica tion Attribute Extraction Clustering /Blocking using Attributes
- 14. GK Adv Fast Card mat 10 pcs Good Night Adv mat 10 pcs Good Knight Advance 10 pcs Good Knight Advance mosquito mat 10 pcs Good Knight Advance refill 45ml GK Adv active+ liquid refill 45ml Good Knight mosquito mat 10 pcs A GOOD POTENTIAL BLOCK Blocking with attributes Good Night Adv mat 10 pcs Good Knight Advance 10 pcs Good Knight Advance mosquito mat 10 pcs Good Knight mosquito mat 10 pcs Good Knight Advance refill 45ml GK Adv active+ liquid refill 45ml GK Adv Fast Card mat 10 pcs
- 15. Third Iteration – Improving similarity within a block • Learning similarity using an affine edit distance model was not sufficient • We combine multiple features to calculate similarity • Word2Vec representation • Attribute features • Lexical features • Soundex • Abbreviations of Brands/Categories • Non vowel string match • We are experimenting with supervised models to combine these different features
- 16. Third Pipeline Similarity Calculation using Distributed and Lexical Features Dedupe Output Sample and Manually Label Active Learning Raw SKUs Segment Identifica tion Attribute Extraction Clustering /Blocking using Attributes
- 17. ['00lifebuoy soap care 125g', 'lifebuoy care soap 100gm', 'lifebuoy care soap 62g', 'lifebuoy soap care 125gm', '100lifebuoy soap care 60g', 'lb soap care 144*59 rs.10', 'lifebuoy soap care 125g', 'lifebuoy care soap 26 mrp', 'lifebuoy care soap 125 g', 'lifebuoy care soap', 'lifebuoy total soap 62g with save 20%', 'lifebuoy soap 26rs', 'lifebuoy soap 27rs', 'lifebuoy soap ₹26/-', 'lifebuoy soap 5/-', 'lifebuoy soap 3*100', 'lifebuoy soap ₹10/-', 'lifebuoy soap 30rs', 'lifebuoy soap', 'lifebuoy soap 24rs', 'lifebuoy soap ₹66/-', 'lifebuoy soap', 'lifebuoy soap ₹27/-', 'lifebuoy soap 94rs', 'lifebuoy soap 29rs', 'lifebuoy soap 28rs', 'lifebuoy soap 25', 'lifebuoy soap care rs.10', 'lifebuoy soap 4pcs 125g', 'lifebuoy care 59g', 'lifebuoy soap nature 125g', 'lifebuoy care 125g', 'lifebuoy care 100g', 'lifebuoy nature soap 125g', 'doy care soap', 'acnelak pimple care soap 75g', 'lifebuoy soap set 18%'] 'lifebuoy soap 94rs', 'lifebuoy soap 29rs', 'lifebuoy soap 28rs', 00lifebuoy soap care 125g lifebuoy care soap 62g 100lifebuoy soap care 60g lifebuoy soap care 125g lifebuoy care soap 125 g lifebuoy care soap lifebuoy soap ₹26/- -- sp lifebuoy soap 5/- -- sp lifebuoy soap lifebuoy soap 25 Some of the final clusters found Block
- 18. Active Learning • We use active learning approach to improve our clustering algorithms • Here we provide samples from our clusters for manual evaluation • We provide paired products for labelling • Dissimilar products from same cluster • Similar products from different cluster
- 19. Challenges/Next Steps • Attribute extraction coverage has to be improved • Improvement in supervised learning model for calculating similarity