Downloaded 14 times
![@eLifeInnovation
Hypothesis: Retraction ~ Negative sentiment
● 2009: Paper B1 published
● 2010: Neutral citation: “These deficiencies in NK
activity may increase viral load in CFS, incidentally a
recent study observed increases in xenotropic murine
leukemia virus-related virus (XMRV) in peripheral blood
samples of CFS patients [B1]”
● 2011: Paper B1 retracted
● 2016: Criticising citation: “The recent link between
ME/CFS and xenotropic murine leukemia virus-related
virus (XMRV) [B1] has largely been disproven and
attributed to laboratory contaminants [B2]”
30](https://image.slidesharecdn.com/citationsentiment-2018-09-04-180903094017/85/Citation-Sentiment-30-320.jpg)
![ANIMAL_CELL_,_TISSUE_AND_ORGAN_CULTURE[1].pptx](https://cdn.slidesharecdn.com/ss_thumbnails/animalcelltissueandorganculture1-260204172026-4462b440-thumbnail.jpg?width=640&height=640&fit=bounds)
Citation Sentiment
- 1. @eLifeInnovation Citation Sentiment Workshop onOpen Citations (Sep 2018) Work by David Ciudad, presented by Daniel Ecer Data Scientist
- 2. @eLifeInnovation Outline About eLife Project Overview AtharDataset Inferring Sentiment from Retractions Summary Next elifesciences.org 2
- 3.
- 4. @eLifeInnovation -- Sir MarkWalport, UK Science Advisor Founding member, eLife Board of Directors
- 5. 5 eLife is anon-profit organisation inspired by research funders and led by scientists Title
- 6. Illustration by DavideBonazzi Get involved elifesci.org/innovate #eLifeSprint. Credit: Orquidea Real Photobook - Julieta Sarmiento Photography @orquidea.real.pho Funding, exposure and mentorship Supporting the open source community through the eLife Innovation Initiative
- 7.
- 8. @eLifeInnovation Work by DavidCiudad ● ASI Fellowship project: Investigating the context of citations ● Project continuation: Investigate criticism / sentiment of citations (preliminary results) 8
- 9. @eLifeInnovation Example citation networkvisualisation 9 http://citationgecko.com/
- 10. @eLifeInnovation Example citation “Our resultscontrast with the high rate of XMRV detection reported by Lombard et al. among both CFS patients and controls, but are in agreement with recent data reported in two large studies in the UK and in the Netherlands...” (Switzer et al., 2010) 10
- 11. @eLifeInnovation Why does thesentiment matter? 11 Pos Pos Neg 3 Citation count “impact” “important” Negative citations count towards citation count ?
- 12. @eLifeInnovation What do wemean by sentiment? ● Polarity: ○ Positive ○ Neutral ○ Negative (criticism) 12
- 13. @eLifeInnovation Off-the-shelf sentiment analysis... 13 Textblob+0.16+0.07 “Our results contrast …” “but are in agreement …” NLTK Vader 0.0 +0.18 Expected< 0 > 0 https://github.com/elifesciences/citation-sentiment-analysis/blob/develop/notebooks/off-the-shelf-sentiment-analysis.ipynb
- 14.
- 15. @eLifeInnovation Athar Dataset ● “SentimentAnalysis of Citations using Sentence Structure-Based Features” (June 2011, Awais Athar) ● http://cl.awaisathar.com/citation-sentiment-corpus/ ● 8736 manually annotated citations 15 https://github.com/elifesciences/citation-sentiment-analysis/blob/develop/notebooks/athar-dataset.ipynb
- 16. @eLifeInnovation Athar dataset cleaning ●There are 37 records with a citation text > 1000 characters (text shows missing sentence separator) ● Some citation texts are missing word separations or include single character tokens ● Filter tokens, keep only: ○ known English words ○ that are at least three characters ○ except the word "no" (we don’t remove most nltk stop words as they might have an influence on sentiment, e.g. “won't”) 16
- 17. @eLifeInnovation Preliminary Results -predict negative vs neutral ● Naive Bayes (bag of words): ○ ~0.77 (0.84 for more biased subset) ○ Words are stemmed 17 Negative Neutral outperform bilingu while similar no updat although constitu limit posit small follow show describ Table: Most indicative word (stems)
- 18. @eLifeInnovation Athar Dataset Conclusion ●Dataset too small (only 280 negative citations) ● “Messy” data (could be cleaned up via source ACL Anthology Network corpus?) 18
- 19.
- 20.
- 21.
- 22.
- 23. @eLifeInnovation Can we harvestcriticising citations from retraction notes? 23
- 24. @eLifeInnovation Retraction note -not linking to criticising papers 24 https://arthritis-research.biomedcentral.com/articles/10.1186/s13075-015-0847-3
- 25. @eLifeInnovation Retraction note -not including any references 25 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4225460/
- 26. @eLifeInnovation Retraction Notes inPMC 26 499 Retraction Watch Database error(s) in data, results and/or conclusions 60in PMC
- 27. @eLifeInnovation Retraction Notes inPMC with Criticism 27 499 Retraction Watch Database error(s) in data, results and/or conclusions 60in PMC 18 with references 9 criticising references
- 28. @eLifeInnovation Retraction Notes (reality) ●Most retraction notes do not link to criticising papers ● Retraction Notes not always tagged accurately in PMC ● Retraction reason may not be machine readable 28
- 29. @eLifeInnovation Can we stillinfer sentiment from retractions? 29
- 30. @eLifeInnovation Hypothesis: Retraction ~Negative sentiment ● 2009: Paper B1 published ● 2010: Neutral citation: “These deficiencies in NK activity may increase viral load in CFS, incidentally a recent study observed increases in xenotropic murine leukemia virus-related virus (XMRV) in peripheral blood samples of CFS patients [B1]” ● 2011: Paper B1 retracted ● 2016: Criticising citation: “The recent link between ME/CFS and xenotropic murine leukemia virus-related virus (XMRV) [B1] has largely been disproven and attributed to laboratory contaminants [B2]” 30
- 31. @eLifeInnovation Are citations afterretraction more likely negative? 31 Pos Pos Neg Cited Paper Citations time Retraction Before After Pos NegNeg
- 32. @eLifeInnovation Predict: before /after retraction 32 Citation Text 1 Learn Before Citation Text 2 Learn After Citation Text 3 Learn Before Citation Text 4 Predict ?
- 33. @eLifeInnovation Retraction Notes inPMC (reminder) 33 499 Retraction Watch Database error(s) in data, results and/or conclusions 60in PMC
- 34. @eLifeInnovation Retraction Notes inPMC with known retraction date 53 with known retraction date 142citing papers 34 499 Retraction Watch Database error(s) in data, results and/or conclusions 60in PMC
- 35. @eLifeInnovation Citing papers before/afterdate of retraction 142 104 38 published after published before 35 citing papers
- 36. @eLifeInnovation Citation Text -Citing Sentence “Conversely, a recent article asserted that the variant was associated with breast cancer, based on a relatively limited case control association study in the Norwegian population (Møller & Hovig, 2017).” 36 citingsentence
- 37. @eLifeInnovation Citation Text -Citing Sentence + 3 (4 sentences) “Conversely, a recent article asserted that the variant was associated with breast cancer, based on a relatively limited case control association study in the Norwegian population (Møller & Hovig, 2017). As a consequence, to date the classification of c.68‐7T > A reported in databases aggregating information on genomic variations has remained inconclusive. In particular, ClinVar reports conflicting interpretations classifying the variant as benign (seven entries), likely benign (nine entries) and of uncertain significance (four entries). Moreover, the BIC database presently annotates the variant as of unknown clinical importance, pending classification, while the BRCA Share™ classifies it as likely benign.” 37 citing sentence+1+2+3 https://dl.acm.org/citation.cfm?id=2382125 (2012, Context-enhanced citation sentiment detection)
- 38. @eLifeInnovation Different models withciting sentences + context Citing Sentence Model Citing Sentence +3 (4) Model Citing Sentence +5 (6) Model
- 39. @eLifeInnovation Balance dataset (controlfor subject area etc.) 142 citing 104 before 38 after 53 retracted 54 citing 27 before 27 after 12 retracted balance
- 40. @eLifeInnovation Detailed results* 40 Undersampled Undersampled + reducedbias Citing Sentence 0.846 0.750 Citing Sentence +3 (4) 0.538 0.833 Citing Sentence +5 (6) 0.615 0.583 On closer inspection, used words like “cancer” (indication of biased dataset) More useful *preliminary results
- 41. @eLifeInnovation Top 7 stemmedwords for 3-sentence model* 41 After Before remain model present subject assess function fail lack conflict discoveri induct research individu no Table: Most indicative word (stems) *preliminary results
- 42.
- 43.
- 44.
- 45.
- 46. @eLifeInnovation Next ● Getting moretraining/test data: ○ Pay for manual annotation? ○ Other ideas to automatically infer sentiment? ● Use cases for reliable sentiment detection of a citation: ○ Better metric? (Colin will talk more about metrics) ○ Identify contentious work? ○ <your suggestion here> ● More suggestions, feedback, collaboration please 46
- 47. @eLifeInnovation Links ● https://github.com/dciudadr/eLife_retractions ● https://github.com/elifesciences/citation-sentiment-analysis ●Blog: Investigating the context of citations 47
- 48.