Web2Text: Deep Structured Boilerplate Removal
Report
Carsten EickhoffFollow
Apr. 5, 2018•0 likes•1,258 views
Web2Text: Deep Structured Boilerplate Removal
Apr. 5, 2018•0 likes•1,258 views
Download to read offline
Report
Science
Web pages are a valuable source of information for many natural language processing and information retrieval tasks. Extracting the main content from those documents is essential for the performance of derived applications. To address this issue, we introduce a novel model that performs sequence labeling to collectively classify all text elements in an HTML page as either boilerplate or main content. Our method uses convolutional networks on top of DOM tree features to learn unary classification potentials for each block of text on the page and pairwise potentials for each pair of neighboring text blocks. We find the most likely labeling according to these potentials using the Viterbi algorithm. The proposed method improves page cleaning performance on the CleanEval benchmark compared to the state-of-the-art. As a component of information retrieval pipelines it improves retrieval performance on the ClueWeb12 collection.
Carsten EickhoffFollow
Recommended
Mapping metabolites against pathway databases Dinesh Barupal
How to be a bioinformaticianChristian Frech
Instalar PENTAHO 5 en CentOS 6Moisés Elías Araya
Gene Ontology Network Enrichment AnalysisUC Davis
Alpha fold 2Vishwas N
An optimal and progressive algorithm for skyline queries slideWooSung Choi
More Related Content
Similar to Web2Text: Deep Structured Boilerplate Removal
Mercurial introRealNitro
Dat 305 dat305 dat 305 education for service uopstudy.comULLPTT
Data compression using python draftAshok Govindarajan
Count-min sketch to Infinity.pdfStephen Lorello
Logic Equation SimplificationSchool of Design Engineering Fashion & Technology (DEFT), University of Wales, Newport
Learning Deep Learningsimaokasonse
![[246]QANet: Towards Efficient and Human-Level Reading Comprehension on SQuAD](https://cdn.slidesharecdn.com/ss_thumbnails/246qanetdeview2018-181012000849-thumbnail.jpg?width=1600&height=908&fit=bounds)
More from Carsten Eickhoff
Unsupervised Learning of General-Purpose Embeddings for User and Location Mod...Carsten Eickhoff
Cognitive Biases in CrowdsourcingCarsten Eickhoff
Evaluating Music Recommender Systems for GroupsCarsten Eickhoff
Active Content-Based Crowdsourcing Task SelectionCarsten Eickhoff
Efficient Parallel Learning of Word2VecCarsten Eickhoff
An Eye-Tracking Study of Query ReformulationCarsten Eickhoff
Recently uploaded
Practices and drivers of antibiotic use in Kenyan smallholder dairy farmsILRI
IMAGES OF PESTS OF ONIONJAY KUMAR
8. Dr Rao.pdfInternational Food Policy Research Institute- South Asia Office
Exploring Estrogen’s Role in Metabolism and the Use of 13C-Labeled Nutrients ...InsideScientific
Everyday Neurobiology Example.pptxTithiRoy37
2023 Oxford Resources for IB DP Chemistry Course Book (Sergey Bylikin, Gary H...BONNIEGUTIERREZBOSSA
Web2Text: Deep Structured Boilerplate Removal
- 1. Web2Text: Deep Structured Boilerplate Removal Thijs Vogels1 , Octavian-Eugen Ganea2 , Carsten Eickhoff3 1 Disney Research 2 ETH Zurich 3 Brown University
- 2. Outline 1. Background & Goals 2. Feature Extraction 3. Representation Learning 4. Inference 5. Experiments
- 11. Beyond HTML Cleaning ● Main content ● Boilerplate ○ Ads ○ Banners ○ Navigation ○ Feeds ○ Next article preview ○ Link lists ○ etc.
- 12. Beyond HTML Cleaning ● Main content ● Boilerplate ○ Ads ○ Banners ○ Navigation ○ Feeds ○ Next article preview ○ Link lists ○ etc.
- 13. Downstream Benefits ● More accurate content extraction ● Improved quality of derived systems ● More compact dataset/index sizes
- 14. Method
- 15. DOM Parsing ● JSoup ● Remove empty nodes: ○ Empty content ○ Whitespace only ● Remove non-content nodes: ○ <br> ○ <checkbox> ○ <hr> ○ Etc.
- 16. Collapsing the DOM Tree ● Inflated node hierarchies can hinder expressiveness ● Especially troublesome for distance calculation
- 18. Collapsing DOM Trees ● Merge single-child nodes with their respective children ● Repeat until collapse
- 19. Collapsed DOM Trees (CDOM)
- 20. Webpages as Sequences of Blocks
- 21. Unary Features (128) per Block ● Avg. word length ● Stopword ratio ● Numeric character ratio ● Relative distance from root ● parent/grandparent information ● ...
- 22. Pairwise Features (25) per Neighboring Pair ● Tree distance in CDOM ● Are blocks separated by line break? ● Node features of common ancestor...
- 23. Representation Learning ● 128/25 hand crafted features ● Consolidate “raw” features in 2 CNNs:
- 24. Representation Learning ● 128/25 hand crafted features ● Consolidate “raw” features in 2 CNNs: ● Unary ○ 5 layers ○ ReLU ○ Softmax ○ Cross entropy loss ○ Outputs: ■ pi (li = 1) ■ pi (li = 0)
- 25. Representation Learning ● 128/25 hand crafted features ● Consolidate “raw” features in 2 CNNs: ● Pairwise ○ 5 layers ○ ReLU ○ Softmax ○ Cross entropy loss ○ Outputs: ■ pi,i+1 (li = 1, li+1 = 1) ■ pi,i+1 (li = 1, li+1 = 0) ■ pi,i+1 (li = 0, li+1 = 1) ■ pi,i+1 (li = 0, li+1 = 0) ● Unary ○ 5 layers ○ ReLU ○ Softmax ○ Cross entropy loss ○ Outputs: ■ pi (li = 1) ■ pi (li = 0)
- 29. Inference p1 (l1 =1) b1 p1 (l1 =0) l1 = ? p2 (l2 =1) b2 p2 (l2 =0) l2 = ? p1,2 (l1 = 1, l2 = 1) p1,2 (l1 = 0, l2 = 0) p 1,2(l1=0,l2=1) p1,2 (l1 =1,l2 =0)
- 30. Inference p1 (l1 =1) b1 p1 (l1 =0) l1 = ? p2 (l2 =1) b2 p2 (l2 =0) l2 = ? p1,2 (l1 = 1, l2 = 1) p1,2 (l1 = 0, l2 = 0) p 1,2(l1=0,l2=1) p1,2 (l1 =1,l2 =0) ... ...
- 31. Inference p1 (l1 =1) b1 p1 (l1 =0) l1 = ? p2 (l2 =1) b2 p2 (l2 =0) l2 = ? pn (ln =1) bn pn (ln =0) ln = ? p1,2 (l1 = 1, l2 = 1) p1,2 (l1 = 0, l2 = 0) p 1,2(l1=0,l2=1) p1,2 (l1 =1,l2 =0) ... ...
- 32. Overview
- 33. Experiments
- 34. Experiment I: Boilerplate Removal ● CleanEval data ● 736 manually annotated Web pages ● On average 188 blocks per page ● Measure Acc, P, R, F ● Baselines: ○ BTE heuristic ○ CRF ○ Unfluff ○ Boilerpipe
- 35. Experiment I: Boilerplate Removal Acc P R F1 BTE 0.75 0.76 0.84 0.80 CRF 0.82 0.88 0.81 0.84 Boilerpipe (def) 0.79 0.89 0.74 0.81 Boilerpipe (art) 0.67 0.89 0.50 0.64 Boilerpipe (text) 0.59 0.93 0.33 0.48 Unfluff 0.68 0.90 0.51 0.65 Web2Text 0.86 0.87 0.90 0.88
- 36. Experiment II: Document Retrieval ● Effect on ad-hoc retrieval ● ClueWeb’12 collection (733M docs) ● 50 TREC 2013 Web track queries ● Indri Query likelihood (QL) and relevance-based language model (RM)
- 37. Experiment II: Document Retrieval
- 38. Experiment II: Document Retrieval ● Low-recall extractors hurt retrieval performance (BTE, art, text, Unfluff)
- 39. Experiment II: Document Retrieval ● Low-recall extractors hurt retrieval performance (BTE, art, text, Unfluff) ● CRF and Web2Text extraction sig. better than raw content indexing
- 40. Experiment II: Document Retrieval ● Low-recall extractors hurt retrieval performance (BTE, art, text, Unfluff) ● CRF and Web2Text extraction sig. better than raw content indexing ● Web2Text extraction sig. better than all compared methods
- 41. Run Times ● Average runtime per Web page ● Macbook with 2.8 GHz Intel Core i5 processor ● Global: 54 ms ○ DOM parsing & feature extraction: 35 ms ○ NN forward pass & Viterbi: 19 ms
- 42. Conclusion ● Deep structure prediction pipeline for Web content extraction ○ Collapsed DOMs ○ Unary and pairwise potentials ○ CNN representation learning ○ HMM-based inference ● Solid content extraction performance ● Can translate into increased downstream effectiveness
- 43. Thank You! Code available at: https://github.com/dalab/web2text