• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Benchmarking the Extraction and Disambiguation of Named Entities on the Semantic Web
 

Benchmarking the Extraction and Disambiguation of Named Entities on the Semantic Web

on

  • 482 views

"Benchmarking the Extraction and Disambiguation of Named Entities on the Semantic Web" talk given at LREC'14, Reykjavik, Iceland

"Benchmarking the Extraction and Disambiguation of Named Entities on the Semantic Web" talk given at LREC'14, Reykjavik, Iceland

Statistics

Views

Total Views
482
Views on SlideShare
475
Embed Views
7

Actions

Likes
1
Downloads
4
Comments
0

1 Embed 7

https://twitter.com 7

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Benchmarking the Extraction and Disambiguation of Named Entities on the Semantic Web Benchmarking the Extraction and Disambiguation of Named Entities on the Semantic Web Presentation Transcript

    • Benchmarking the Extraction andBenchmarking the Extraction and Disambiguation of Named EntitiesDisambiguation of Named Entities on the Semantic Webon the Semantic Web Giuseppe Rizzo, Marieke van Erp, Raphaël Troncy @merpeltje @rtroncy@giusepperizzo
    • May 30, 2014 2/219th Edition of the Language Resources and Evaluation Conference (LREC'14) Benchmarking NER & NED ➢ NER ➢ [newswire] CoNLL, ACE, MUC ➢ [microposts] Microposts Concept Extraction ➢ NED ➢ [newswire] TAC KBP ➢ [microposts] Microposts NEEL ➢ Numerous academic and commercial NER and NED tools ➢ To name a few: AlchemyAPI, DBpedia Spotlight, GATE, OpeNER, Stanford
    • May 30, 2014 3/219th Edition of the Language Resources and Evaluation Conference (LREC'14) This Work ➢ Evaluation and comparison of 11 NER and NED tools through the NERD API ➢ Combination of the 11 NER tools in NERD-ML ➢ Experiments on two types of corpora: newswire and microposts
    • May 30, 2014 4/219th Edition of the Language Resources and Evaluation Conference (LREC'14) ➢ http://nerd.eurecom.fr ➢ Ontology, REST API & Web Application ➢ Uniform access to 11 NER/NED external tools ➢ commercial: AlchemyAPI, dataTXT, OpenCalais, Saplo, TextRazor, Wikimeta, Yahoo!, Zemanta ➢ academic: DBpedia Spotlight, Lupedia, THD
    • May 30, 2014 5/219th Edition of the Language Resources and Evaluation Conference (LREC'14) Theoretical limit ➢ Any of these systems have strengths in entity typing ➢ An ideal combination will use the best in typing among all ➢ Estimate the upper bound where each type is ttarget=selectte=tGS (te1 ,te2 ,...,ten )
    • May 30, 2014 6/219th Edition of the Language Resources and Evaluation Conference (LREC'14) ➢ Try to perform better than each individual NER tool ➢ Learning: ➢ NERD tool predictions ➢ Stanford CRF predictions ➢ Linguistic features ➢ Naive Bayes (NB), k-nearest neighbors (k-NN), Support Vector Machines (SVM, RBF kernel) N ERD-ML
    • May 30, 2014 7/219th Edition of the Language Resources and Evaluation Conference (LREC'14) Feature Vector extractor2 type extractor1 type linguistic vector ... extractorN type GS type training vector
    • May 30, 2014 8/219th Edition of the Language Resources and Evaluation Conference (LREC'14) Linguistic Features POS initial cap (*) all caps (*) capitalized ratio (**) prefix suffix begin or end (*) linguistic vector * Boolean value ** Double value token
    • May 30, 2014 9/219th Edition of the Language Resources and Evaluation Conference (LREC'14) Experiments - NER ➢ CoNLL2003 English, testb set [newswire] ➢ 231 Articles ➢ 46,435 Tokens ➢ 5,648 NEs ➢ MSM2013, test set [microposts] ➢ 1,450 Posts ➢ 29,085 Tokens ➢ 1,538 NEs
    • May 30, 2014 10/219th Edition of the Language Resources and Evaluation Conference (LREC'14) Results on CoNLL2003
    • May 30, 2014 11/219th Edition of the Language Resources and Evaluation Conference (LREC'14) Results on MSM2013
    • May 30, 2014 12/219th Edition of the Language Resources and Evaluation Conference (LREC'14) CoNLL2003 NERD-ML Incremental Learning (1/2) Experimental settings: ➢ Feature Vector: token, AlchemyAPI, DBpedia Spotlight, Cicero, Lupedia, OpenCalais, Saplo, Yahoo!, Textrazor, Wikimeta, Stanford, GS type ➢ Classifier = NB
    • May 30, 2014 13/219th Edition of the Language Resources and Evaluation Conference (LREC'14) MSM2013 NERD-ML Incremental Learning (2/2) Experimental settings: ➢ Feature Vector: token, pos, initialcaps, allcaps, prefix, suffix, capitalfreq, start, AlchemyAPI, DBpedia Spotlight, Cicero, Lupedia, Opencalais, Textrazor, Ritter, Stanford, GS type ➢ Classifier = SVM
    • May 30, 2014 14/219th Edition of the Language Resources and Evaluation Conference (LREC'14) Experiments - NED ➢ AIDA CoNLL-YAGO links to Wikipedia, testb set [newswire] ➢ 231 Articles ➢ 46,435 Tokens ➢ 4,485 Links ➢ Microposts2014 links to DBpedia, test set [microposts] ➢ 1,165 Posts ➢ 23,815 Tokens ➢ 1,330 Links
    • May 30, 2014 15/219th Edition of the Language Resources and Evaluation Conference (LREC'14) Results on AIDA CoNLL-YAGO Wikipeda is the reference Knowledge Base
    • May 30, 2014 16/219th Edition of the Language Resources and Evaluation Conference (LREC'14) Results on Microposts2014 DBpedia v3.9 is the reference Knowledge Base
    • May 30, 2014 17/219th Edition of the Language Resources and Evaluation Conference (LREC'14) Discussion NER ➢ Newswire ➢ Robust performance on recognizing common types ➢ But MISC class is hard to detect (always will be?) ➢ Microposts ➢ Fairly robust for PER ➢ Weak in recognizing LOC and ORG ➢ MISC is around 30% of F1
    • May 30, 2014 18/219th Edition of the Language Resources and Evaluation Conference (LREC'14) Discussion NED ➢ Newswire ➢ Unreliable performance on linking, with the peak in F1 of 50.41% for TextRazor ➢ Linkers use different reference knowledge bases. Source of bias is the link normalization part ➢ Microposts ➢ Linking shows a big drop in performance ➢ TextRazor has the best score with a 32.65% F1
    • May 30, 2014 19/219th Edition of the Language Resources and Evaluation Conference (LREC'14) Future Work ➢ NER ➢ Improving the taxonomy alignment ➢ NED ➢ Better harmonization of the linking stage ➢ NERD-ML ➢ Getting closer to the theoretical limit in NER ➢ Use of gazetteers for MISC types ➢ Combining the outputs of the NEL tools to predict the links
    • May 30, 2014 20/219th Edition of the Language Resources and Evaluation Conference (LREC'14) Acknowledgments The research leading to this paper was partially supported by the European Union’s 7th Framework Programme via the projects LinkedTV (GA 287911) and NewsReader (ICT-316404)
    • May 30, 2014 21/219th Edition of the Language Resources and Evaluation Conference (LREC'14) Thank You For Listening http://www.slideshare.net/giusepperizzo https://github.com/giusepperizzo/nerdml