Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Systematic Study of Long Tail Phenomena in Entity Linking

110 views

Published on

Presented at COLING'18 (21-08-2018)

Published in: Science
  • Be the first to comment

  • Be the first to like this

Systematic Study of Long Tail Phenomena in Entity Linking

  1. 1. Systematic Study of Long Tail Phenomena in Entity Linking Filip Ilievski, Piek Vossen, Stefan Schlobach
  2. 2. Entity Linking (EL) “Washington announces Alex Smith trade It seems like months ago that the Chiefs traded Alex Smith to Washington... Smith, 33, originally entered ...” (https://profootballtalk.nbcsports.com/2018/03/14/washington-announces-alex-smith-trade/) surface form instance interpretation
  3. 3. State-of-the-art Entity Linking SotA: High F1-scores by probabilistic optimization F1-score => system skills ?? => errors ?? ~ data properties ?? “Washington announces Alex Smith trade It seems like months ago that the Chiefs traded Alex Smith to Washington... Smith, 33, originally entered ...” (https://profootballtalk.nbcsports.com/2018/03/14/washington-announces-alex-smith-trade/)
  4. 4. Head and tail of Entity Linking Claim: performance (head) >> performance (tail) (Ilievski et al., 2016; van Erp et al., 2016; Esquivel et al., 2017) head =? ∧ tail=? => performance (head) >> performance (tail) ?? => how to improve performance (tail) ??
  5. 5. Contributions of this work 1. Description and hypotheses on the long tail properties of EL 2. Analysis of EL datasets WRT the long tail properties 3. Analysis of system performance WRT the long tail properties 4. Recommended actions
  6. 6. Ambiguity of forms (number of different instances that a form refers to) “Washington “ Variance of instances (number of distinct forms that refer to an instance) “... U.S. federal government” “Washington” “... government of U.S. ...” Frequency of forms/instances (number of occurrences in a corpus) “Washington announces Alex Smith trade It seems like months ago that the Chiefs traded Alex Smith to Washington. Smith, 33, originally entered ...” Popularity of instances (PageRank in a knowledge graph) Definition of long tail properties
  7. 7. Hypotheses and setup 16 hypotheses 2 data collections (CoNLL-AIDA and N3), 5 corpora in total 3 SotA systems: AGDISTIS MAG, DBpedia Spotlight, and WAT Precision, recall and F1-score
  8. 8. Hypotheses on the data properties Positive correlation between ambiguity and frequency of forms amb(f) ~ freq(f) Positive correlation between variance, frequency, and popularity of instances var(i) ~ freq(i) var(i) ~ pop(i) freq(i) ~ pop(i) Zipfian frequency distribution within all forms that refer to an instance freq(f|I) ~ zipfian Zipfian frequency distribution within all instances that refer to a form freq(i|F) ~ zipfian
  9. 9. amb(f) ~ freq(f) var(i) ~ freq(i) freq(i) ~ pop(i)var(i) ~ pop(i)
  10. 10. freq(f) ~ zipfian freq(i) ~ zipfian
  11. 11. Hypotheses on system performance Systems perform worse on forms that are ambiguous than overall. f1(AMF) << f1(ALL) Best performance on frequent, non-ambiguous forms; worst performance on infrequent, highly ambiguous forms. f1(freq, ⅂amb) = MAX(f1) f1(⅂freq, amb) = MIN(f1) Performance is inversely proportional with entropy. f1(AMF) ~ ⅂entropy(AMF) Systems perform better on frequent/popular instances of ambiguous forms, compared to their infrequent/unpopular instances. f1(i|F) ~ freq(i|F) f1(i|F) ~ pop(i|F)
  12. 12. f1(AMF) << f1(ALL)
  13. 13. f1(freq, ⅂amb) = MAX(f1) f1(⅂freq, amb) = MIN(f1)
  14. 14. S4: Systems perform better on ambiguous forms with imbalanced, compared to balanced, instance distribution f1(AMF) ~ ⅂entropy(AMF)
  15. 15. f1(amb)<< f1(all) f1(i|F) ~ freq(i|F) f1(i|F) ~ pop(i|F)
  16. 16. Recommendations [Dataset creation] ● statistics on the head and the tail ● most-frequent-value baseline [Evaluation] ● evaluate on the head and the tail ● use macro F1-score [System development] ● which heuristics target which cases ● which resources optimize for the head/tail
  17. 17. Conclusions First work that systematically describes the relation of surface forms in EL corpora and their instances in DBpedia, through long tail properties. We measured expected inter-correlations between long tail phenomena in EL datasets. System performance correlates positively with frequency and popularity of instances, and negatively with ambiguity of forms. We listed recommended actions to influence future designs of systems and datasets in EL.
  18. 18. Thanks for your attention! Questions? Github: cltl/EL-long-tail-phenomena Twitter: @earthling91

×