Are Linked Datasets fit for Open-domain Question Answering? A Quality Assessment
•
1 like•264 views
Presentation for the paper accepted at The 6th International Conference on Web Intelligence, Mining and Semantics (WIMS) 2016. [http://harshthakkar.in/wp-content/uploads/2016/02/wims.pdf]
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Are Linked Datasets fit for Open-domain Question Answering? A Quality Assessment
Similar to Are Linked Datasets fit for Open-domain Question Answering? A Quality Assessment (20)
Recently uploaded
Recently uploaded (20)
Are Linked Datasets fit for Open-domain Question Answering? A Quality Assessment
- 13. QA systems Quality assessment of the LOD datasets The answer lies here!
- 14. • •
- 15. Digging into the QA system Typical IR system performances measures ● Overall Performance ○ F1 ○ Precision ○ Recall
- 16. Digging into the QA system Data & Component/Module oriented measures ● Search & retrieval module ○ Indexer ○ Retriever ● Preprocessing / Linguistic ○ NLP - POS tags, NER, etc ○ Entity linking & annotation - semantics ○ Relation extraction & annotation ● Query formulation ○ SPARQL conversion ● Datasource/knowledge base ○ Data Typical IR system performances measures ● Overall Performance ○ F1 ○ Precision ○ Recall
- 17. Digging into the QA system Data & Component/Module oriented measures ● Search & retrieval module ○ Indexer ■ Top K words accuracy; P@10, P@1000, etc ○ Retriever ■ Ranking, Re-ranking, MRR, etc ● Preprocessing / Linguistic ○ NLP - POS tags, NER, etc ○ Entity linking & annotation - semantics ○ Relation extraction & annotation ■ annotation accuracy/precision ■ consistency, interlinking, etc ● Query formulation ○ SPARQL conversion ■ conversion accuracy/precision ● Datasource/knowledge base ○ Completeness ○ Data diversity ○ Trust and Provenance ○ Coverage ○ Timeliness (up to date) ○ etc Typical IR system performances measures ● Overall Performance ○ F1 ○ Precision ○ Recall
- 18. Digging into the QA system Data & Component/Module oriented measures ● Search & retrieval module ○ Indexer ■ Top K words accuracy; P@10, P@1000, etc ○ Retriever ■ Ranking, Re-ranking, MRR, etc ● Preprocessing / Linguistic ○ NLP - POS tags, NER, etc ○ Entity linking & annotation - semantics ○ Relation extraction & annotation ■ annotation accuracy/precision ■ consistency, interlinking, etc ● Query formulation ○ SPARQL conversion ■ conversion accuracy/precision ● Datasource/Knowledge base ○ Completeness ○ Data diversity ○ Trust and Provenance ○ Coverage ○ Timeliness (up to date) ○ etc Typical IR system performances measures ● Overall Performance ○ F1 ○ Precision ○ Recall
- 19. • •
- 20. Evaluated in this study
- 25. •
- 30. ● ○ ○ ○ ●
- 31. DBpedia data slice sizes (in MB)Wikidata data slice sizes (in MB)
- 32. Dimension Metric DB_Rest DB_Poli DB_Film DB_Soc Availability EstimatedDereferenceabilityMetric 0.013 0.013 0.012 0.012 EstimatedDereferenceabilityForwardLinksMetric 0.027 0.027 0.027 0.027 NoMisreportedContentTypesMetric 0 1 1 1 RDFAvailabilityMetric 0 0 0 0 EndPointAvailabilityMetric 0 0 0 0 Interlinking EstimatedInterlinkDetectionMetric - - - - EstimatedLinkExternalDataProviders - - - - EstimatedDereferenceBackLinks 0.012 0.014 0.015 0.022 Semantic accuracy OntologyHijacking 1 1 1 1 MisusedOwlDatatypeOrObjectProperties 1 1 1 1 Data diversity HumanReadableLabelling 0.953 0.985 0.997 1 MultipleLanguageUsageMteric 1 2 3 3 Trust and Provenance Basic Provenance 0 0 0 0 Extended Provenance 0 0 0 0 Provenance Richness 0 0 0 0 DBPEDIA SLICE ASSESSMENT RESULTS
- 33. WIKIDATA SLICE ASSESSMENT RESULTS Dimension Metric Wiki_Rest Wiki_Poli Wiki_Film Wiki_Soc Availability EstimatedDereferenceabilityMetric 0.051 0.063 0.048 0.062 EstimatedDereferenceabilityForwardLinksMetric 0.093 0.053 0.050 0.064 NoMisreportedContentTypesMetric 0 1 0 1 RDFAvailabilityMetric 0 0 0 0 EndPointAvailabilityMetric 0 0 0 0 Interlinking EstimatedInterlinkDetectionMetric - - - - EstimatedLinkExternalDataProviders 5 11 9 8 EstimatedDereferenceBackLinks 0.013 0.098 0.089 0.083 Semantic accuracy OntologyHijacking 1 1 1 1 MisusedOwlDatatypeOrObjectProperties 1 1 1 1 Data diversity HumanReadableLabelling 0.175 0.076 0.091 0.102 MultipleLanguageUsageMteric 2 3 2 3 Trust and Provenance Basic Provenance 0 0 0 0 Extended Provenance 0 0 0 0 Provenance Richness 0.055 0.083 0.010 0.025