Your SlideShare is downloading. ×
Evaluating Named Entity Recognition and Disambiguation in News and Tweets
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Evaluating Named Entity Recognition and Disambiguation in News and Tweets

1,447

Published on

Named entity recognition and disambiguation are important for information extraction and populating knowledge bases. Detecting and classifying named entities has traditionally been taken on by the …

Named entity recognition and disambiguation are important for information extraction and populating knowledge bases. Detecting and classifying named entities has traditionally been taken on by the natural language processing community, whilst linking of entities to external resources, such as DBpedia and GeoNames, has been the domain of the Semantic Web community. As these tasks are treated in different communities, it is difficult to assess the performance of these tasks combined.

We present results on an evaluation of the NERD-ML approach on newswire and tweets for both Named Entity Recognition and Named Entity Disambiguation.

Presented at CLIN 24: http://clin24.inl.nl/

http://nerd.eurecom.fr
https://github.com/giusepperizzo/nerdml

Published in: Technology, Education
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,447
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
20
Comments
0
Likes
4
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. EVALUATING NAMED ENTITY RECOGNITION AND DISAMBIGUATION IN NEWS AND TWEETS Giuseppe Rizzo Università degli studi di Torino Marieke van Erp VU University Amsterdam Raphaël Troncy EURECOM
  • 2. EVALUATING NER & NED • NER typically an NLP task (MUC, CoNLL, ACE) • NED took flight with availability of large structured resources (Wikipedia, DBpedia, Freebase) • Tools for NER and NED have started popping up outside regular research outlets (TextRazor, DBpedia Spotlight, AlchemyAPI) • Unclear how well these tools perform
  • 3. THIS WORK • Evaluation & comparison of 10 out-of-the-box NER and NED tools through NERD API as well as a combination of the tools in NERD-ML • Two types of data: Newswire & Tweets
  • 4. • http://nerd.eurecom.fr • Ontology, REST API & UI • Uniform access to 12 different extractors/linkers: AlchemyAPI, DBpedia Spotlight, Extractiv, Lupedia, OpenCalais, Saplo, SemiTags, TextRazor, THD, Wikimeta, Yahoo! Content Analysis, Zemanta
  • 5. NERD-ML • The aim of NERD-ML is to combine the knowledge of the different extractors into a better named entity recogniser • Uses NERD predictions, Stanford NER & extra features • Naive Bayes, k-NN, SMO
  • 6. DATA • CoNLL 2003 English NER with AIDA CoNLL-YAGO links to Wikipedia (5,648 NEs/4,485 links in test set) • Making Sense of Microposts 2013 (MSM’13) for NER on Twitter domain + 62 randomly selected tweets from Ritter et al.’s corpus with links to DBpedia resources (MSM: 1,538 NEs/Ritter: 177 links in test set)
  • 7. PER LOC ORG Precision Precision MISC OVERALL 100 90 80 70 60 50 40 30 20 10 0 0 10 20 30 40 50 60 70 80 90 100 RESULTS NER NEWSWIRE PER LOC ORG Recall Recall MISC OVERALL PER LOC ORG MISC OVERALL F1 F1 AlchemyAPI DBpedia Spotlight Extractiv Lupedia OpenCalais Saplo Textrazor Yahoo Wikimeta Zemanta Stanford NER NERD-ML Run01 NERD-ML Run02 NERD-ML Run03 Upper Limit
  • 8. PER LOC ORG Precision Precision MISC OVERALL 100 0 10 20 30 40 50 60 70 80 90 100 90 80 70 60 50 40 30 20 10 0 0 10 20 30 40 50 60 70 80 90 100 RESULTS NER MSM PER LOC ORG Precision Recall MISC OVERALL PER LOC ORG MISC OVERALL Precision F1 AlchemyAPI DBpedia Spotlight Extractiv Lupedia OpenCalais Saplo Textrazor Wikimeta Zemanta Ritter et al. Stanford NER NERD-ML Run01 NERD-ML Run02 NERD-ML Run03 Upper Limit
  • 9. RESULTS NED DBpedia AlchemyAPI Extractiv Lupedia Textrazor Yahoo Zemanta Spotlight AIDAYAGO 70.63 26.93 51.31 57.98 49.21 0.0 35.58 TWEETS 53.85 25.13 74.07 65.38 58.14 76.00 48.57
  • 10. DISCUSSION • Still a ways to go, but for certain classes NER is getting close to really good results • MISC class is (and probably always will be?) hard • Bigger datasets needed (for tweets and NED) • NED task can use standardisation
  • 11. THANK YOU FOR LISTENING • Try out our code at: https://github.com/giusepperizzo/nerdml
  • 12. ACKNOWLEDGEMENTS This research is funded through the LinkedTV and NewsReader projects, both funded by the European Union’s 7th Framework Programme grants GA 287911 and ICT-316404).

×