Finding Products on the Internet Using Neural Networks
•
3 likes•717 views
Datafiniti crawls the entire Internet to find product data. We use a machine learning technique called neural networks to automatically identify product listings and various heuristic techniques to extract product data.
Recommended
Recommended
More Related Content
Similar to Finding Products on the Internet Using Neural Networks
Similar to Finding Products on the Internet Using Neural Networks (20)
Recently uploaded
Recently uploaded (20)
Finding Products on the Internet Using Neural Networks
- 1. Finding Products on the Internet using Neural Networks http://www.datafiniti.net
- 2. ● Goals ○ Collect vast amounts of data through web crawling ○ Normalize and deduplicate data ○ Make it searchable and meaningful
- 3. Motivation
- 4. Challenges 48 billion pages on the Internet ○ Crawled 6 billion+ pages in the past year Mostly unstructured data Limitations of customized crawls ○ Non-scalable ○ Less robust
- 5. Solution: Intelligent Classifiers Advantages ○ Generic code: Scalability ○ More robust Challenges ○ More difficult to parse data of interest
- 10. Some Other Page
- 11. Problem Solution
- 12. Minimize dependency on HTML Supervised learning for page classification ○ Neural networks Heuristic algorithms for data parsing Our Approach
- 13. Hidden Layer Input Layer Output Layer AV(Product) AV(Product Category) AV(Other) PageFeatures AV: Activation Value : {0, 1} Neural Network Classification_Type = Type with max. AV
- 14. Page Features Buy Widget Price Image Num Clickable Images with Price Shipping Info
- 16. Trained offline Dataset Feature Vector Normalization Neural Network Input Layer Parameter Set (P) Hidden Layer Parameter Set (Q) Training
- 17. Web page Feature Vector Normalized Feature Vector (x) Neural Network Input Layer Parameter Set (P) Hidden Layer Parameter Set (Q) AV(Prod) AV(ProdCat) AV(Other) Page_Type = max{ AV(Prod), AV(ProdCat), AV(Other) } Output of hidden layer: L1 = sigmoid(PT X) Final output: L2 = sigmoid(QT L1 ) L2 = { AV(Prod), AV(ProdCat), AV(Other) } sigmoid(s) = 1 / (1 + e-s ) Deployment
- 18. Notation ○ True Positive (TP) ○ False Positive (FP) ○ False Negative (FN) Precision : TP / (TP + FP) Recall : TP / (TP + FN) F-score: 2PR / (P + R) Known Dataset ○ Precision = 1.0 ○ Recall = 0.985 ○ F-score = 0.9925 Live System/Unknown Data ○ Precision = 0.854 ○ Recall = Difficult to calculate Evaluation
- 20. Product Name Product Price Product Code ○ UPC, EAN, ISBN, ASIN Fields to Collect
- 21. Product PageGetting Product Name Potential Names <title>Pebble Smart Watch for Select Apple and Android Devices 301RD - Best Buy</title> Match Found
- 22. Product PageGetting Product Price Price values with text - discard Old Price - discard Current Price - Accept
- 23. Improve classification accuracy Increase/improve collection of data fields Future Work
- 25. Price Image(s) # clickable images adjacent to price values "Add to cart", "Buy" widget # words in page text Keywords ○ Product detail, specifications, features, size, color, weight, shipping, availability, SKU, UPC, ISBN, ASIN Page Features
- 26. Some Other Page