SlideShare a Scribd company logo
1 of 21
Download to read offline
Benchmarking the Extraction andBenchmarking the Extraction and
Disambiguation of Named EntitiesDisambiguation of Named Entities
on the Semantic Webon the Semantic Web
Giuseppe Rizzo, Marieke van Erp, Raphaël Troncy
@merpeltje @rtroncy@giusepperizzo
May 30, 2014 2/219th
Edition of the Language Resources and Evaluation Conference (LREC'14)
Benchmarking NER & NED
➢ NER
➢ [newswire] CoNLL, ACE, MUC
➢ [microposts] Microposts Concept Extraction
➢ NED
➢ [newswire] TAC KBP
➢ [microposts] Microposts NEEL
➢ Numerous academic and commercial NER and NED tools
➢ To name a few: AlchemyAPI, DBpedia Spotlight, GATE,
OpeNER, Stanford
May 30, 2014 3/219th
Edition of the Language Resources and Evaluation Conference (LREC'14)
This Work
➢ Evaluation and comparison of 11 NER and NED
tools through the NERD API
➢ Combination of the 11 NER tools in NERD-ML
➢ Experiments on two types of corpora: newswire and
microposts
May 30, 2014 4/219th
Edition of the Language Resources and Evaluation Conference (LREC'14)
➢
http://nerd.eurecom.fr
➢
Ontology, REST API & Web Application
➢
Uniform access to 11 NER/NED external tools
➢
commercial: AlchemyAPI, dataTXT, OpenCalais,
Saplo, TextRazor, Wikimeta, Yahoo!, Zemanta
➢
academic: DBpedia Spotlight, Lupedia, THD
May 30, 2014 5/219th
Edition of the Language Resources and Evaluation Conference (LREC'14)
Theoretical limit
➢ Any of these systems have strengths in entity typing
➢ An ideal combination will use the best in typing
among all
➢ Estimate the upper bound where each type is
ttarget=selectte=tGS
(te1
,te2
,...,ten
)
May 30, 2014 6/219th
Edition of the Language Resources and Evaluation Conference (LREC'14)
➢
Try to perform better than each individual NER tool
➢
Learning:
➢
NERD tool predictions
➢
Stanford CRF predictions
➢
Linguistic features
➢
Naive Bayes (NB), k-nearest neighbors (k-NN),
Support Vector Machines (SVM, RBF kernel)
N ERD-ML
May 30, 2014 7/219th
Edition of the Language Resources and Evaluation Conference (LREC'14)
Feature Vector
extractor2
type
extractor1
type
linguistic
vector
...
extractorN
type
GS
type
training
vector
May 30, 2014 8/219th
Edition of the Language Resources and Evaluation Conference (LREC'14)
Linguistic Features
POS
initial
cap (*)
all
caps (*)
capitalized
ratio (**)
prefix suffix
begin or
end (*)
linguistic
vector
* Boolean value
** Double value
token
May 30, 2014 9/219th
Edition of the Language Resources and Evaluation Conference (LREC'14)
Experiments - NER
➢ CoNLL2003 English, testb set [newswire]
➢ 231 Articles
➢ 46,435 Tokens
➢ 5,648 NEs
➢ MSM2013, test set [microposts]
➢ 1,450 Posts
➢ 29,085 Tokens
➢ 1,538 NEs
May 30, 2014 10/219th
Edition of the Language Resources and Evaluation Conference (LREC'14)
Results on CoNLL2003
May 30, 2014 11/219th
Edition of the Language Resources and Evaluation Conference (LREC'14)
Results on MSM2013
May 30, 2014 12/219th
Edition of the Language Resources and Evaluation Conference (LREC'14)
CoNLL2003
NERD-ML Incremental Learning (1/2)
Experimental settings:
➢
Feature Vector: token, AlchemyAPI, DBpedia Spotlight, Cicero, Lupedia,
OpenCalais, Saplo, Yahoo!, Textrazor, Wikimeta, Stanford, GS type
➢
Classifier = NB
May 30, 2014 13/219th
Edition of the Language Resources and Evaluation Conference (LREC'14)
MSM2013
NERD-ML Incremental Learning (2/2)
Experimental settings:
➢
Feature Vector: token, pos, initialcaps, allcaps, prefix, suffix, capitalfreq, start,
AlchemyAPI, DBpedia Spotlight, Cicero, Lupedia, Opencalais, Textrazor, Ritter,
Stanford, GS type
➢
Classifier = SVM
May 30, 2014 14/219th
Edition of the Language Resources and Evaluation Conference (LREC'14)
Experiments - NED
➢ AIDA CoNLL-YAGO links to Wikipedia, testb set
[newswire]
➢ 231 Articles
➢ 46,435 Tokens
➢ 4,485 Links
➢ Microposts2014 links to DBpedia, test set [microposts]
➢ 1,165 Posts
➢ 23,815 Tokens
➢ 1,330 Links
May 30, 2014 15/219th
Edition of the Language Resources and Evaluation Conference (LREC'14)
Results on AIDA CoNLL-YAGO
Wikipeda is the reference Knowledge Base
May 30, 2014 16/219th
Edition of the Language Resources and Evaluation Conference (LREC'14)
Results on Microposts2014
DBpedia v3.9 is the reference Knowledge Base
May 30, 2014 17/219th
Edition of the Language Resources and Evaluation Conference (LREC'14)
Discussion NER
➢ Newswire
➢ Robust performance on recognizing common types
➢ But MISC class is hard to detect (always will be?)
➢ Microposts
➢ Fairly robust for PER
➢ Weak in recognizing LOC and ORG
➢ MISC is around 30% of F1
May 30, 2014 18/219th
Edition of the Language Resources and Evaluation Conference (LREC'14)
Discussion NED
➢ Newswire
➢ Unreliable performance on linking, with the peak in
F1 of 50.41% for TextRazor
➢ Linkers use different reference knowledge bases.
Source of bias is the link normalization part
➢ Microposts
➢ Linking shows a big drop in performance
➢ TextRazor has the best score with a 32.65% F1
May 30, 2014 19/219th
Edition of the Language Resources and Evaluation Conference (LREC'14)
Future Work
➢ NER
➢ Improving the taxonomy alignment
➢ NED
➢ Better harmonization of the linking stage
➢ NERD-ML
➢ Getting closer to the theoretical limit in NER
➢
Use of gazetteers for MISC types
➢ Combining the outputs of the NEL tools to predict the links
May 30, 2014 20/219th
Edition of the Language Resources and Evaluation Conference (LREC'14)
Acknowledgments
The research leading to this
paper was partially supported
by the European Union’s 7th
Framework Programme via the
projects LinkedTV (GA
287911) and NewsReader
(ICT-316404)
May 30, 2014 21/219th
Edition of the Language Resources and Evaluation Conference (LREC'14)
Thank You For Listening
http://www.slideshare.net/giusepperizzo
https://github.com/giusepperizzo/nerdml

More Related Content

Viewers also liked

20 J An N As
20 J An N As20 J An N As
20 J An N Asepaper
 
Edisi 15 Nov
Edisi 15 NovEdisi 15 Nov
Edisi 15 Novepaper
 
Edisi 22 Feb Aceh
Edisi 22 Feb AcehEdisi 22 Feb Aceh
Edisi 22 Feb Acehepaper
 
Patrones Hipoxicos
Patrones HipoxicosPatrones Hipoxicos
Patrones Hipoxicosjenniefer
 
8 Des N As
8 Des N As8 Des N As
8 Des N Asepaper
 
Waspada Nasional3 Sep
Waspada Nasional3 SepWaspada Nasional3 Sep
Waspada Nasional3 Sepepaper
 
Treasury Finance Audit Tj Sparrow Jan 2011
Treasury Finance Audit  Tj Sparrow Jan 2011Treasury Finance Audit  Tj Sparrow Jan 2011
Treasury Finance Audit Tj Sparrow Jan 2011Trishjo
 
Waspada Aceh3sep
Waspada Aceh3sepWaspada Aceh3sep
Waspada Aceh3sepepaper
 
Waspada 8 Ags Nas
Waspada 8 Ags NasWaspada 8 Ags Nas
Waspada 8 Ags Nasepaper
 
10desnasyg Bner
10desnasyg Bner10desnasyg Bner
10desnasyg Bnerepaper
 
Ovarianreservepamphlet4
Ovarianreservepamphlet4Ovarianreservepamphlet4
Ovarianreservepamphlet4aleeoc
 
[Infographic] Online Sales Rev Up in the Automotive Aftermarket
[Infographic] Online Sales Rev Up in the Automotive Aftermarket[Infographic] Online Sales Rev Up in the Automotive Aftermarket
[Infographic] Online Sales Rev Up in the Automotive AftermarketKathy McCoy, MBA
 
Edisi 6 Des 2009 Nas
Edisi 6 Des 2009 NasEdisi 6 Des 2009 Nas
Edisi 6 Des 2009 Nasepaper
 
Edisi 12 Aceh Sep
Edisi 12 Aceh SepEdisi 12 Aceh Sep
Edisi 12 Aceh Sepepaper
 
Aspects of Reproducibility in Earth Science
Aspects of Reproducibility in Earth ScienceAspects of Reproducibility in Earth Science
Aspects of Reproducibility in Earth ScienceRaul Palma
 
Edisi 6 Des Nas
Edisi 6 Des NasEdisi 6 Des Nas
Edisi 6 Des Nasepaper
 
Edisi5novnas
Edisi5novnasEdisi5novnas
Edisi5novnasepaper
 

Viewers also liked (20)

20 J An N As
20 J An N As20 J An N As
20 J An N As
 
Edisi 15 Nov
Edisi 15 NovEdisi 15 Nov
Edisi 15 Nov
 
Edisi 22 Feb Aceh
Edisi 22 Feb AcehEdisi 22 Feb Aceh
Edisi 22 Feb Aceh
 
Patrones Hipoxicos
Patrones HipoxicosPatrones Hipoxicos
Patrones Hipoxicos
 
8 Des N As
8 Des N As8 Des N As
8 Des N As
 
2006 Iarep Parigi Paper
2006 Iarep Parigi Paper2006 Iarep Parigi Paper
2006 Iarep Parigi Paper
 
Waspada Nasional3 Sep
Waspada Nasional3 SepWaspada Nasional3 Sep
Waspada Nasional3 Sep
 
Treasury Finance Audit Tj Sparrow Jan 2011
Treasury Finance Audit  Tj Sparrow Jan 2011Treasury Finance Audit  Tj Sparrow Jan 2011
Treasury Finance Audit Tj Sparrow Jan 2011
 
Waspada Aceh3sep
Waspada Aceh3sepWaspada Aceh3sep
Waspada Aceh3sep
 
Waspada 8 Ags Nas
Waspada 8 Ags NasWaspada 8 Ags Nas
Waspada 8 Ags Nas
 
10desnasyg Bner
10desnasyg Bner10desnasyg Bner
10desnasyg Bner
 
2016 NMA
2016 NMA2016 NMA
2016 NMA
 
Ovarianreservepamphlet4
Ovarianreservepamphlet4Ovarianreservepamphlet4
Ovarianreservepamphlet4
 
Blooms taxonomy
Blooms taxonomyBlooms taxonomy
Blooms taxonomy
 
[Infographic] Online Sales Rev Up in the Automotive Aftermarket
[Infographic] Online Sales Rev Up in the Automotive Aftermarket[Infographic] Online Sales Rev Up in the Automotive Aftermarket
[Infographic] Online Sales Rev Up in the Automotive Aftermarket
 
Edisi 6 Des 2009 Nas
Edisi 6 Des 2009 NasEdisi 6 Des 2009 Nas
Edisi 6 Des 2009 Nas
 
Edisi 12 Aceh Sep
Edisi 12 Aceh SepEdisi 12 Aceh Sep
Edisi 12 Aceh Sep
 
Aspects of Reproducibility in Earth Science
Aspects of Reproducibility in Earth ScienceAspects of Reproducibility in Earth Science
Aspects of Reproducibility in Earth Science
 
Edisi 6 Des Nas
Edisi 6 Des NasEdisi 6 Des Nas
Edisi 6 Des Nas
 
Edisi5novnas
Edisi5novnasEdisi5novnas
Edisi5novnas
 

Similar to Benchmarking the Extraction and Disambiguation of Named Entities on the Semantic Web

NEEL2015 challenge summary
NEEL2015 challenge summaryNEEL2015 challenge summary
NEEL2015 challenge summaryGiuseppe Rizzo
 
NLP Data Cleansing Based on Linguistic Ontology Constraints
NLP Data Cleansing Based on Linguistic Ontology ConstraintsNLP Data Cleansing Based on Linguistic Ontology Constraints
NLP Data Cleansing Based on Linguistic Ontology ConstraintsDimitris Kontokostas
 
Morphological Analyzer and Generator for Tamil Language
Morphological Analyzer and Generator for Tamil LanguageMorphological Analyzer and Generator for Tamil Language
Morphological Analyzer and Generator for Tamil LanguageLushanthan Sivaneasharajah
 
Natural language processing for requirements engineering: ICSE 2021 Technical...
Natural language processing for requirements engineering: ICSE 2021 Technical...Natural language processing for requirements engineering: ICSE 2021 Technical...
Natural language processing for requirements engineering: ICSE 2021 Technical...alessio_ferrari
 
Inductive Entity Typing Alignment
Inductive Entity Typing AlignmentInductive Entity Typing Alignment
Inductive Entity Typing AlignmentGiuseppe Rizzo
 
Introduction of the SLE'17 conference
Introduction of the SLE'17 conferenceIntroduction of the SLE'17 conference
Introduction of the SLE'17 conferenceBenoit Combemale
 
Eclipse Day India 2015 - Keynote - Stephan Herrmann
Eclipse Day India 2015 - Keynote - Stephan HerrmannEclipse Day India 2015 - Keynote - Stephan Herrmann
Eclipse Day India 2015 - Keynote - Stephan HerrmannEclipse Day India
 
65 - An Empirical Simulation-based Study of Real-Time Speech Translation for ...
65 - An Empirical Simulation-based Study of Real-Time Speech Translation for ...65 - An Empirical Simulation-based Study of Real-Time Speech Translation for ...
65 - An Empirical Simulation-based Study of Real-Time Speech Translation for ...ESEM 2014
 
An Introduction to ANTLR
An Introduction to ANTLRAn Introduction to ANTLR
An Introduction to ANTLRMorteza Zakeri
 
ESWC-2011: S-Match received 7 years award
ESWC-2011: S-Match received 7 years award ESWC-2011: S-Match received 7 years award
ESWC-2011: S-Match received 7 years award Pavel Shvaiko
 
Combining Textual and Graph-based Features for Entity Disambiguation
Combining Textual and Graph-based Features for Entity DisambiguationCombining Textual and Graph-based Features for Entity Disambiguation
Combining Textual and Graph-based Features for Entity Disambiguationshakimov
 

Similar to Benchmarking the Extraction and Disambiguation of Named Entities on the Semantic Web (11)

NEEL2015 challenge summary
NEEL2015 challenge summaryNEEL2015 challenge summary
NEEL2015 challenge summary
 
NLP Data Cleansing Based on Linguistic Ontology Constraints
NLP Data Cleansing Based on Linguistic Ontology ConstraintsNLP Data Cleansing Based on Linguistic Ontology Constraints
NLP Data Cleansing Based on Linguistic Ontology Constraints
 
Morphological Analyzer and Generator for Tamil Language
Morphological Analyzer and Generator for Tamil LanguageMorphological Analyzer and Generator for Tamil Language
Morphological Analyzer and Generator for Tamil Language
 
Natural language processing for requirements engineering: ICSE 2021 Technical...
Natural language processing for requirements engineering: ICSE 2021 Technical...Natural language processing for requirements engineering: ICSE 2021 Technical...
Natural language processing for requirements engineering: ICSE 2021 Technical...
 
Inductive Entity Typing Alignment
Inductive Entity Typing AlignmentInductive Entity Typing Alignment
Inductive Entity Typing Alignment
 
Introduction of the SLE'17 conference
Introduction of the SLE'17 conferenceIntroduction of the SLE'17 conference
Introduction of the SLE'17 conference
 
Eclipse Day India 2015 - Keynote - Stephan Herrmann
Eclipse Day India 2015 - Keynote - Stephan HerrmannEclipse Day India 2015 - Keynote - Stephan Herrmann
Eclipse Day India 2015 - Keynote - Stephan Herrmann
 
65 - An Empirical Simulation-based Study of Real-Time Speech Translation for ...
65 - An Empirical Simulation-based Study of Real-Time Speech Translation for ...65 - An Empirical Simulation-based Study of Real-Time Speech Translation for ...
65 - An Empirical Simulation-based Study of Real-Time Speech Translation for ...
 
An Introduction to ANTLR
An Introduction to ANTLRAn Introduction to ANTLR
An Introduction to ANTLR
 
ESWC-2011: S-Match received 7 years award
ESWC-2011: S-Match received 7 years award ESWC-2011: S-Match received 7 years award
ESWC-2011: S-Match received 7 years award
 
Combining Textual and Graph-based Features for Entity Disambiguation
Combining Textual and Graph-based Features for Entity DisambiguationCombining Textual and Graph-based Features for Entity Disambiguation
Combining Textual and Graph-based Features for Entity Disambiguation
 

More from Giuseppe Rizzo

Artificial intelligence for social good
Artificial intelligence for social goodArtificial intelligence for social good
Artificial intelligence for social goodGiuseppe Rizzo
 
COMPRENDE, PERSONALIZZA, INTERAGISCE E IMPARA: L’AI COGNITIVA PER L’HR
COMPRENDE, PERSONALIZZA, INTERAGISCE E  IMPARA: L’AI COGNITIVA PER L’HRCOMPRENDE, PERSONALIZZA, INTERAGISCE E  IMPARA: L’AI COGNITIVA PER L’HR
COMPRENDE, PERSONALIZZA, INTERAGISCE E IMPARA: L’AI COGNITIVA PER L’HRGiuseppe Rizzo
 
Understand, Answer and Argument: Conversational Agents
Understand, Answer and Argument: Conversational AgentsUnderstand, Answer and Argument: Conversational Agents
Understand, Answer and Argument: Conversational AgentsGiuseppe Rizzo
 
AI For Profiling Your Customers
AI For Profiling Your CustomersAI For Profiling Your Customers
AI For Profiling Your CustomersGiuseppe Rizzo
 
AI for Personalized Chatbot
AI for Personalized ChatbotAI for Personalized Chatbot
AI for Personalized ChatbotGiuseppe Rizzo
 
Tourist Knowledge Graph Creation to Automating Travel Bookings
Tourist Knowledge Graph Creation to Automating Travel BookingsTourist Knowledge Graph Creation to Automating Travel Bookings
Tourist Knowledge Graph Creation to Automating Travel BookingsGiuseppe Rizzo
 
The SentiME System at the SSA Challenge Task 1
The SentiME System at the SSA Challenge Task 1The SentiME System at the SSA Challenge Task 1
The SentiME System at the SSA Challenge Task 1Giuseppe Rizzo
 
Context-Enhanced Adaptive Entity Linking
Context-Enhanced Adaptive Entity LinkingContext-Enhanced Adaptive Entity Linking
Context-Enhanced Adaptive Entity LinkingGiuseppe Rizzo
 
From Data to Knowledge for Tourists
From Data to Knowledge for TouristsFrom Data to Knowledge for Tourists
From Data to Knowledge for TouristsGiuseppe Rizzo
 
Enabling Visitors to Explore a Smart City
Enabling Visitors to Explore a Smart CityEnabling Visitors to Explore a Smart City
Enabling Visitors to Explore a Smart CityGiuseppe Rizzo
 
CrossLanguageSpotter: A Library for Detecting Relations in Polyglot Frameworks
CrossLanguageSpotter: A Library for Detecting Relations in Polyglot FrameworksCrossLanguageSpotter: A Library for Detecting Relations in Polyglot Frameworks
CrossLanguageSpotter: A Library for Detecting Relations in Polyglot FrameworksGiuseppe Rizzo
 
Learning with the Web. Structuring data to ease machine understanding
Learning with the Web. Structuring data to ease  machine understandingLearning with the Web. Structuring data to ease  machine understanding
Learning with the Web. Structuring data to ease machine understandingGiuseppe Rizzo
 
Learning with the Web: Spotting Named Entities on the intersection of NERD an...
Learning with the Web: Spotting Named Entities on the intersection of NERD an...Learning with the Web: Spotting Named Entities on the intersection of NERD an...
Learning with the Web: Spotting Named Entities on the intersection of NERD an...Giuseppe Rizzo
 
NERD meets NIF: Lifting NLP Extraction Results to the Linked Data Cloud
NERD meets NIF:  Lifting NLP Extraction Results to the Linked Data CloudNERD meets NIF:  Lifting NLP Extraction Results to the Linked Data Cloud
NERD meets NIF: Lifting NLP Extraction Results to the Linked Data CloudGiuseppe Rizzo
 
L'enorme archivio di dati: il Web
L'enorme archivio di dati: il WebL'enorme archivio di dati: il Web
L'enorme archivio di dati: il WebGiuseppe Rizzo
 
NERD: Evaluating Named Entity Recognition Tools in the Web of Data
NERD: Evaluating Named Entity Recognition Tools in the Web of DataNERD: Evaluating Named Entity Recognition Tools in the Web of Data
NERD: Evaluating Named Entity Recognition Tools in the Web of DataGiuseppe Rizzo
 
Zenaminer: driving the SCORM tandard towards the Web of Data
Zenaminer: driving the SCORM tandard towards the Web of DataZenaminer: driving the SCORM tandard towards the Web of Data
Zenaminer: driving the SCORM tandard towards the Web of DataGiuseppe Rizzo
 

More from Giuseppe Rizzo (19)

Artificial intelligence for social good
Artificial intelligence for social goodArtificial intelligence for social good
Artificial intelligence for social good
 
AI in 60 minutes
AI in 60 minutesAI in 60 minutes
AI in 60 minutes
 
COMPRENDE, PERSONALIZZA, INTERAGISCE E IMPARA: L’AI COGNITIVA PER L’HR
COMPRENDE, PERSONALIZZA, INTERAGISCE E  IMPARA: L’AI COGNITIVA PER L’HRCOMPRENDE, PERSONALIZZA, INTERAGISCE E  IMPARA: L’AI COGNITIVA PER L’HR
COMPRENDE, PERSONALIZZA, INTERAGISCE E IMPARA: L’AI COGNITIVA PER L’HR
 
Understand, Answer and Argument: Conversational Agents
Understand, Answer and Argument: Conversational AgentsUnderstand, Answer and Argument: Conversational Agents
Understand, Answer and Argument: Conversational Agents
 
AI For Profiling Your Customers
AI For Profiling Your CustomersAI For Profiling Your Customers
AI For Profiling Your Customers
 
AI for Personalized Chatbot
AI for Personalized ChatbotAI for Personalized Chatbot
AI for Personalized Chatbot
 
Tourist Knowledge Graph Creation to Automating Travel Bookings
Tourist Knowledge Graph Creation to Automating Travel BookingsTourist Knowledge Graph Creation to Automating Travel Bookings
Tourist Knowledge Graph Creation to Automating Travel Bookings
 
The SentiME System at the SSA Challenge Task 1
The SentiME System at the SSA Challenge Task 1The SentiME System at the SSA Challenge Task 1
The SentiME System at the SSA Challenge Task 1
 
Context-Enhanced Adaptive Entity Linking
Context-Enhanced Adaptive Entity LinkingContext-Enhanced Adaptive Entity Linking
Context-Enhanced Adaptive Entity Linking
 
From Data to Knowledge for Tourists
From Data to Knowledge for TouristsFrom Data to Knowledge for Tourists
From Data to Knowledge for Tourists
 
Enabling Visitors to Explore a Smart City
Enabling Visitors to Explore a Smart CityEnabling Visitors to Explore a Smart City
Enabling Visitors to Explore a Smart City
 
CrossLanguageSpotter: A Library for Detecting Relations in Polyglot Frameworks
CrossLanguageSpotter: A Library for Detecting Relations in Polyglot FrameworksCrossLanguageSpotter: A Library for Detecting Relations in Polyglot Frameworks
CrossLanguageSpotter: A Library for Detecting Relations in Polyglot Frameworks
 
Learning with the Web. Structuring data to ease machine understanding
Learning with the Web. Structuring data to ease  machine understandingLearning with the Web. Structuring data to ease  machine understanding
Learning with the Web. Structuring data to ease machine understanding
 
Learning with the Web: Spotting Named Entities on the intersection of NERD an...
Learning with the Web: Spotting Named Entities on the intersection of NERD an...Learning with the Web: Spotting Named Entities on the intersection of NERD an...
Learning with the Web: Spotting Named Entities on the intersection of NERD an...
 
NERD meets NIF: Lifting NLP Extraction Results to the Linked Data Cloud
NERD meets NIF:  Lifting NLP Extraction Results to the Linked Data CloudNERD meets NIF:  Lifting NLP Extraction Results to the Linked Data Cloud
NERD meets NIF: Lifting NLP Extraction Results to the Linked Data Cloud
 
The NERD project
The NERD projectThe NERD project
The NERD project
 
L'enorme archivio di dati: il Web
L'enorme archivio di dati: il WebL'enorme archivio di dati: il Web
L'enorme archivio di dati: il Web
 
NERD: Evaluating Named Entity Recognition Tools in the Web of Data
NERD: Evaluating Named Entity Recognition Tools in the Web of DataNERD: Evaluating Named Entity Recognition Tools in the Web of Data
NERD: Evaluating Named Entity Recognition Tools in the Web of Data
 
Zenaminer: driving the SCORM tandard towards the Web of Data
Zenaminer: driving the SCORM tandard towards the Web of DataZenaminer: driving the SCORM tandard towards the Web of Data
Zenaminer: driving the SCORM tandard towards the Web of Data
 

Benchmarking the Extraction and Disambiguation of Named Entities on the Semantic Web

  • 1. Benchmarking the Extraction andBenchmarking the Extraction and Disambiguation of Named EntitiesDisambiguation of Named Entities on the Semantic Webon the Semantic Web Giuseppe Rizzo, Marieke van Erp, Raphaël Troncy @merpeltje @rtroncy@giusepperizzo
  • 2. May 30, 2014 2/219th Edition of the Language Resources and Evaluation Conference (LREC'14) Benchmarking NER & NED ➢ NER ➢ [newswire] CoNLL, ACE, MUC ➢ [microposts] Microposts Concept Extraction ➢ NED ➢ [newswire] TAC KBP ➢ [microposts] Microposts NEEL ➢ Numerous academic and commercial NER and NED tools ➢ To name a few: AlchemyAPI, DBpedia Spotlight, GATE, OpeNER, Stanford
  • 3. May 30, 2014 3/219th Edition of the Language Resources and Evaluation Conference (LREC'14) This Work ➢ Evaluation and comparison of 11 NER and NED tools through the NERD API ➢ Combination of the 11 NER tools in NERD-ML ➢ Experiments on two types of corpora: newswire and microposts
  • 4. May 30, 2014 4/219th Edition of the Language Resources and Evaluation Conference (LREC'14) ➢ http://nerd.eurecom.fr ➢ Ontology, REST API & Web Application ➢ Uniform access to 11 NER/NED external tools ➢ commercial: AlchemyAPI, dataTXT, OpenCalais, Saplo, TextRazor, Wikimeta, Yahoo!, Zemanta ➢ academic: DBpedia Spotlight, Lupedia, THD
  • 5. May 30, 2014 5/219th Edition of the Language Resources and Evaluation Conference (LREC'14) Theoretical limit ➢ Any of these systems have strengths in entity typing ➢ An ideal combination will use the best in typing among all ➢ Estimate the upper bound where each type is ttarget=selectte=tGS (te1 ,te2 ,...,ten )
  • 6. May 30, 2014 6/219th Edition of the Language Resources and Evaluation Conference (LREC'14) ➢ Try to perform better than each individual NER tool ➢ Learning: ➢ NERD tool predictions ➢ Stanford CRF predictions ➢ Linguistic features ➢ Naive Bayes (NB), k-nearest neighbors (k-NN), Support Vector Machines (SVM, RBF kernel) N ERD-ML
  • 7. May 30, 2014 7/219th Edition of the Language Resources and Evaluation Conference (LREC'14) Feature Vector extractor2 type extractor1 type linguistic vector ... extractorN type GS type training vector
  • 8. May 30, 2014 8/219th Edition of the Language Resources and Evaluation Conference (LREC'14) Linguistic Features POS initial cap (*) all caps (*) capitalized ratio (**) prefix suffix begin or end (*) linguistic vector * Boolean value ** Double value token
  • 9. May 30, 2014 9/219th Edition of the Language Resources and Evaluation Conference (LREC'14) Experiments - NER ➢ CoNLL2003 English, testb set [newswire] ➢ 231 Articles ➢ 46,435 Tokens ➢ 5,648 NEs ➢ MSM2013, test set [microposts] ➢ 1,450 Posts ➢ 29,085 Tokens ➢ 1,538 NEs
  • 10. May 30, 2014 10/219th Edition of the Language Resources and Evaluation Conference (LREC'14) Results on CoNLL2003
  • 11. May 30, 2014 11/219th Edition of the Language Resources and Evaluation Conference (LREC'14) Results on MSM2013
  • 12. May 30, 2014 12/219th Edition of the Language Resources and Evaluation Conference (LREC'14) CoNLL2003 NERD-ML Incremental Learning (1/2) Experimental settings: ➢ Feature Vector: token, AlchemyAPI, DBpedia Spotlight, Cicero, Lupedia, OpenCalais, Saplo, Yahoo!, Textrazor, Wikimeta, Stanford, GS type ➢ Classifier = NB
  • 13. May 30, 2014 13/219th Edition of the Language Resources and Evaluation Conference (LREC'14) MSM2013 NERD-ML Incremental Learning (2/2) Experimental settings: ➢ Feature Vector: token, pos, initialcaps, allcaps, prefix, suffix, capitalfreq, start, AlchemyAPI, DBpedia Spotlight, Cicero, Lupedia, Opencalais, Textrazor, Ritter, Stanford, GS type ➢ Classifier = SVM
  • 14. May 30, 2014 14/219th Edition of the Language Resources and Evaluation Conference (LREC'14) Experiments - NED ➢ AIDA CoNLL-YAGO links to Wikipedia, testb set [newswire] ➢ 231 Articles ➢ 46,435 Tokens ➢ 4,485 Links ➢ Microposts2014 links to DBpedia, test set [microposts] ➢ 1,165 Posts ➢ 23,815 Tokens ➢ 1,330 Links
  • 15. May 30, 2014 15/219th Edition of the Language Resources and Evaluation Conference (LREC'14) Results on AIDA CoNLL-YAGO Wikipeda is the reference Knowledge Base
  • 16. May 30, 2014 16/219th Edition of the Language Resources and Evaluation Conference (LREC'14) Results on Microposts2014 DBpedia v3.9 is the reference Knowledge Base
  • 17. May 30, 2014 17/219th Edition of the Language Resources and Evaluation Conference (LREC'14) Discussion NER ➢ Newswire ➢ Robust performance on recognizing common types ➢ But MISC class is hard to detect (always will be?) ➢ Microposts ➢ Fairly robust for PER ➢ Weak in recognizing LOC and ORG ➢ MISC is around 30% of F1
  • 18. May 30, 2014 18/219th Edition of the Language Resources and Evaluation Conference (LREC'14) Discussion NED ➢ Newswire ➢ Unreliable performance on linking, with the peak in F1 of 50.41% for TextRazor ➢ Linkers use different reference knowledge bases. Source of bias is the link normalization part ➢ Microposts ➢ Linking shows a big drop in performance ➢ TextRazor has the best score with a 32.65% F1
  • 19. May 30, 2014 19/219th Edition of the Language Resources and Evaluation Conference (LREC'14) Future Work ➢ NER ➢ Improving the taxonomy alignment ➢ NED ➢ Better harmonization of the linking stage ➢ NERD-ML ➢ Getting closer to the theoretical limit in NER ➢ Use of gazetteers for MISC types ➢ Combining the outputs of the NEL tools to predict the links
  • 20. May 30, 2014 20/219th Edition of the Language Resources and Evaluation Conference (LREC'14) Acknowledgments The research leading to this paper was partially supported by the European Union’s 7th Framework Programme via the projects LinkedTV (GA 287911) and NewsReader (ICT-316404)
  • 21. May 30, 2014 21/219th Edition of the Language Resources and Evaluation Conference (LREC'14) Thank You For Listening http://www.slideshare.net/giusepperizzo https://github.com/giusepperizzo/nerdml