0
EVALUATING NAMED ENTITY
RECOGNITION AND DISAMBIGUATION
IN NEWS AND TWEETS
Giuseppe Rizzo	

 	

 	

 Università degli studi...
EVALUATING NER & NED
•

NER typically an NLP task (MUC, CoNLL, ACE)	


•

NED took flight with availability of large struct...
THIS WORK
•

Evaluation & comparison of 10 out-of-the-box
NER and NED tools through NERD API as well as
a combination of t...
•

http://nerd.eurecom.fr 	


•

Ontology, REST API & UI 	


•

Uniform access to 12 different extractors/linkers:
Alchemy...
NERD-ML
•

The aim of NERD-ML is to combine the
knowledge of the different extractors into a better
named entity recognise...
DATA
•

CoNLL 2003 English NER with AIDA CoNLL-YAGO
links to Wikipedia (5,648 NEs/4,485 links in test set)	


•

Making Se...
PER

LOC

ORG

Precision
Precision

MISC

OVERALL

100
90
80
70
60
50
40
30
20
10
0

0

10

20

30

40

50

60

70

80

90...
PER

LOC

ORG
Precision

Precision

MISC

OVERALL

100
0

10

20

30

40

50

60

70

80

90

100
90
80
70
60
50
40
30
20
...
RESULTS NED
DBpedia
AlchemyAPI
Extractiv Lupedia Textrazor Yahoo Zemanta
Spotlight
AIDAYAGO

70.63	


26.93

51.31

57.98
...
DISCUSSION
•

Still a ways to go, but for certain classes NER is
getting close to really good results	


•

MISC class is ...
THANK YOU FOR LISTENING

•

Try out our code at:
https://github.com/giusepperizzo/nerdml
ACKNOWLEDGEMENTS
This research is funded through the LinkedTV
and NewsReader projects, both funded by the
European Union’s...
Upcoming SlideShare
Loading in...5
×

Evaluating Named Entity Recognition and Disambiguation in News and Tweets

1,511

Published on

Named entity recognition and disambiguation are important for information extraction and populating knowledge bases. Detecting and classifying named entities has traditionally been taken on by the natural language processing community, whilst linking of entities to external resources, such as DBpedia and GeoNames, has been the domain of the Semantic Web community. As these tasks are treated in different communities, it is difficult to assess the performance of these tasks combined.

We present results on an evaluation of the NERD-ML approach on newswire and tweets for both Named Entity Recognition and Named Entity Disambiguation.

Presented at CLIN 24: http://clin24.inl.nl/

http://nerd.eurecom.fr
https://github.com/giusepperizzo/nerdml

Published in: Technology, Education
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,511
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
21
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Transcript of "Evaluating Named Entity Recognition and Disambiguation in News and Tweets"

  1. 1. EVALUATING NAMED ENTITY RECOGNITION AND DISAMBIGUATION IN NEWS AND TWEETS Giuseppe Rizzo Università degli studi di Torino Marieke van Erp VU University Amsterdam Raphaël Troncy EURECOM
  2. 2. EVALUATING NER & NED • NER typically an NLP task (MUC, CoNLL, ACE) • NED took flight with availability of large structured resources (Wikipedia, DBpedia, Freebase) • Tools for NER and NED have started popping up outside regular research outlets (TextRazor, DBpedia Spotlight, AlchemyAPI) • Unclear how well these tools perform
  3. 3. THIS WORK • Evaluation & comparison of 10 out-of-the-box NER and NED tools through NERD API as well as a combination of the tools in NERD-ML • Two types of data: Newswire & Tweets
  4. 4. • http://nerd.eurecom.fr • Ontology, REST API & UI • Uniform access to 12 different extractors/linkers: AlchemyAPI, DBpedia Spotlight, Extractiv, Lupedia, OpenCalais, Saplo, SemiTags, TextRazor, THD, Wikimeta, Yahoo! Content Analysis, Zemanta
  5. 5. NERD-ML • The aim of NERD-ML is to combine the knowledge of the different extractors into a better named entity recogniser • Uses NERD predictions, Stanford NER & extra features • Naive Bayes, k-NN, SMO
  6. 6. DATA • CoNLL 2003 English NER with AIDA CoNLL-YAGO links to Wikipedia (5,648 NEs/4,485 links in test set) • Making Sense of Microposts 2013 (MSM’13) for NER on Twitter domain + 62 randomly selected tweets from Ritter et al.’s corpus with links to DBpedia resources (MSM: 1,538 NEs/Ritter: 177 links in test set)
  7. 7. PER LOC ORG Precision Precision MISC OVERALL 100 90 80 70 60 50 40 30 20 10 0 0 10 20 30 40 50 60 70 80 90 100 RESULTS NER NEWSWIRE PER LOC ORG Recall Recall MISC OVERALL PER LOC ORG MISC OVERALL F1 F1 AlchemyAPI DBpedia Spotlight Extractiv Lupedia OpenCalais Saplo Textrazor Yahoo Wikimeta Zemanta Stanford NER NERD-ML Run01 NERD-ML Run02 NERD-ML Run03 Upper Limit
  8. 8. PER LOC ORG Precision Precision MISC OVERALL 100 0 10 20 30 40 50 60 70 80 90 100 90 80 70 60 50 40 30 20 10 0 0 10 20 30 40 50 60 70 80 90 100 RESULTS NER MSM PER LOC ORG Precision Recall MISC OVERALL PER LOC ORG MISC OVERALL Precision F1 AlchemyAPI DBpedia Spotlight Extractiv Lupedia OpenCalais Saplo Textrazor Wikimeta Zemanta Ritter et al. Stanford NER NERD-ML Run01 NERD-ML Run02 NERD-ML Run03 Upper Limit
  9. 9. RESULTS NED DBpedia AlchemyAPI Extractiv Lupedia Textrazor Yahoo Zemanta Spotlight AIDAYAGO 70.63 26.93 51.31 57.98 49.21 0.0 35.58 TWEETS 53.85 25.13 74.07 65.38 58.14 76.00 48.57
  10. 10. DISCUSSION • Still a ways to go, but for certain classes NER is getting close to really good results • MISC class is (and probably always will be?) hard • Bigger datasets needed (for tweets and NED) • NED task can use standardisation
  11. 11. THANK YOU FOR LISTENING • Try out our code at: https://github.com/giusepperizzo/nerdml
  12. 12. ACKNOWLEDGEMENTS This research is funded through the LinkedTV and NewsReader projects, both funded by the European Union’s 7th Framework Programme grants GA 287911 and ICT-316404).
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×