Offspring from Reproduction Problems: what replication failure teaches us


Published on

Slides of ACL 2013 presentation of:
Antske Fokkens, Marieke van Erp, Marten Postma, Ted Pedersen, Piek Vossen and Nuno Freire (2013) Offspring from Reproduction Problems: what replication failure teaches us. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 1691–1701, Sofia, Bulgaria, August 4-9 2013

Published in: Technology, Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Offspring from Reproduction Problems: what replication failure teaches us

  1. 1. NewsReader is funded by the European Union’s 7th Framework Programme (ICT-316404) BiographyNet is funded by the Netherlands eScience Center. Partners in BiographyNet are Huygens/ING Institute of the Dutch Academy of Sciences andVU University Amsterdam. Wednesday, August 7, 13
  2. 2. OFFSPRING FROM REPRODUCTION PROBLEMS: WHAT REPLICATION FAILURETEACHES US Antske Fokkens, Marieke van Erp, Marten Postma,Ted Pedersen, PiekVossen and Nuno Freire Wednesday, August 7, 13
  3. 3. NER EXPERIMENTS • Nuno Freire, José Borbinha, and Pável Calado (2012) present an approach to recognise named entities in the cultural heritage domain with a small amount of training data • The dataset used in Freire et al., (2012) is available • Their software was not available • The paper describes the feature set used as well as the (open source) machine learning package and the experimental setup Wednesday, August 7, 13
  4. 4. • Nuno Freire provided additional feedback, but had finished his PhD and changed jobs: he had no access to his original experimental setup or his code • The paper did not provide information on tokenisation, exact preprocessing steps, cuts for 10-fold cross validation, number of decimals used for rounding weights, etc. NER EXPERIMENTS Wednesday, August 7, 13
  5. 5. NER EXPERIMENTS Freire et al. (2012) Precision Recall F-score P Freire et al. (2012) Precision Recall F-score P Freire et al. (2012) Precision Recall F-score P Van Erp and Van der Meij (2013) Precision Recall F-score Van Erp and Van der Meij (2013) Precision Recall F-score Van Erp and Van der Meij (2013) Precision Recall F-score LOC (388) 92% 55% 69 77.8% 39.2% 52.1 ORG (157) 90% 57% 70 65.8% 30.6% 41.7 PER (614) 91% 56% 69 73.3% 37.6% 49.7 Overall (1,159) 91% 55% 69 73.3% 37.1% 49.5 Wednesday, August 7, 13
  6. 6. • Variations on tokenisation yielded a 15 point drop in overall F- score • Results on individual folds different up to 25 points in F-score • Experimenting with a different implementation of the CRF algorithm yielded significantly different scores (almost attaining those of Freire et al. (2012) without the complex features) NER EXPERIMENTS Wednesday, August 7, 13
  7. 7. • Preprocessing of the data set and the extra resources used probably influenced our experiments • Encoding the features as multivariate vs. boolean as input for the machine learner may have made a difference • Without additional information (exact output, data/resources after preprocessing), it is hard to find out what causes these differences or which experiment provides the most indicative results of the potential of the approach. NER EXPERIMENTS Wednesday, August 7, 13
  8. 8. WORDNET SIMILARITY EXPERIMENTS • Marten Postma & PiekVossen wanted to run a WordNet similarity experiment for Dutch previously done for English by Patwardhan & Pedersen (2006) and Pedersen (2010) • This experiment ranks similarities between words based on WordNet similarity measures and compares it to human rankings • Step 1: replicate the WordNet calculations of the original experiments Wednesday, August 7, 13
  9. 9. WORDNET SIMILARITY EXPERIMENTS • The code used in the original experiments is open source, but still results were different from the original • Pedersen pointed out that the version of WordNet (and possibly Perl packages) may influence results • Experiments were repeated with the exact same versions as the original: results still differed Wednesday, August 7, 13
  10. 10. WORDNET SIMILARITY EXPERIMENTS • Together withTed Pedersen, we ran the experiment step by step comparing outcome until we obtained the same results • We identified the following factors that had lead to differences: • Restriction on PoS tags • Gold Standard used • Ranking coefficient used • How much can these factors actually influence results? Wednesday, August 7, 13
  11. 11. WORDNET SIMILARITY EXPERIMENTS Wednesday, August 7, 13
  12. 12. WORDNET SIMILARITY EXPERIMENTS VARIATIONS IN OUTPUT Measure rho tau rank path 0.08 0.07 1-8 wup 0.09 0.08 1-6 lch 0.08 0.07 1-7 res 0.10 0.31 4-11 lin 0.24 0.17 6-10 jcn 0.27 0.23 5,7-11 hso 0.07 0.05 1-3,5-10 vpairs 0.30 0.24 7-11 vector 0.44 0.43 1,2,4,6-11 lesk 0.17 0.63 1-8,11,12 Wednesday, August 7, 13
  13. 13. WORDNET SIMILARITY EXPERIMENTS Variation Spearman rho Kendall tau Different rank WordNet version 0.44 0.42 88% Gold Standard 0.24 0.21 71% PoS-tag 0.09 0.08 41% Configuration 0.08 0.60 41% Wednesday, August 7, 13
  14. 14. WORDNET SIMILARITY EXPERIMENTS • Performance of similarity measures can vary significantly • Influential factors interact differently with individual scores (i.e. comparative performance changes) • Apart from WordNet version, these factors have (to our knowledge) not been discussed in previous literature, despite the fact that similarity scores are used very frequently • Open question: what is the impact of influential factors when similarity measures are used for other tasks? Wednesday, August 7, 13
  15. 15. DISCUSSION • The results in this paper point to two main issues: (1) Our methodological descriptions often do not contain the details needed to reproduce our results (2)These details can have such high impact on our results that it is hard to distinguish the contribution of the approach from the contribution of preprocessing, the exact versions of tools/resources used, the evaluation set chosen etc. Wednesday, August 7, 13
  16. 16. CONCLUSION • It is easier to find out how an approach really works, if you have: • the original code (even if containing hacks, unclean and/or undocumented) • a clear description of each individual step • the exact output on evaluation data (not just the overall numbers) • the preprocessed/modified/improved versions of standard resources Wednesday, August 7, 13
  17. 17. CONCLUSION • Systematic testing can help to gain insight into the expected variation of a specific approach: • what is the performance on individual tools • what are the best and worst result using different parameters? • how does performance compare using different evaluation metrics? • As a community, we should know where our approaches fail as much -if not more- as where they succeed Wednesday, August 7, 13
  18. 18. THANKYOU &THANKS • ToTed Pedersen and Nuno Freire for writing this paper with us! • To Ruben Izquierdo, Lourens van der Meij, Christoph Zwirello, Rebecca Dridan and the Semantic Web group atVU university for their help and feedback • To the anonymous reviewers who really helped to make this a better paper Wednesday, August 7, 13
  19. 19. Wednesday, August 7, 13