SlideShare a Scribd company logo
Towards Knowledge Maintenance in Scientific
Digital Libraries with the Keystone Framework
Yuanxi Fu & Jodi Schneider
School of Information Sciences
University of Illinois at Urbana-Champaign
Presentation for JCDL 2020, Virtual, 2020-08-02
Motivation: Need for KNOWLEDGE MAINTENANCE
• When you’re not an expert, how do you judge papers?
– Recency
– Citation count
– Other heuristics
• Literature becomes obsolete but usually that’s not explicit.
• These errors can be passed on & can lead to misinterpretations.
– Most research needs information from cognate fields.
How big is retraction & citation to retraction
• Over 600,000 articles directly cite a retracted paper.
• The Retraction Watch Database lists over 19,000 retracted
publications.
• In biomedicine 94% of retracted papers have received at least
one citation, with an average citation count of 35 (Dinh, …, Schneider 2019).
Motivating Questions
• Are papers citing a retracted paper
necessarily wrong?
• Does it matter when citing authors make
use of a paper whose findings are no longer
considered valid?
• When DOES the citation matter?
• Could we selectively alert authors who cite a
retracted or abandoned paper?
Introducing the Keystone Framework
Under our framework:
1) A scientific research paper puts forward at
least one main finding, along with a
logical argument, giving reasons and
evidence to support the main finding.
2) The main finding is accepted (or not) on
the basis of the logical argument.
3) Evidence from earlier literature may be
incorporated into the argument by citing a
paper and presenting it as support, using a
citation context.
Main Finding
Arguments
support
support
Data Methods Citations …
Applying the Framework
Step 1, model the argument
Citing ArticleCited Article
Citation Context
“Many papers with
known validity problems
are highly cited [3].”
Our
paper
[3] = Bar-Ilan, J.
and Halevi, G.
2018. Temporal
characteristics
of retracted
articles
Applying the Framework:
Step 2, find the citation contexts that contribute to
the argument
Applying the Framework:
Step 3, analyze the citation context:
How many items are cited?
Citation Context
“Many papers with
known validity problems
are highly cited [3].”
Singleton
Cluster
“Digital library
applications of
argumentation theory
include argument-based
retrieval [21, 30]”
Cited Article
Citation Context
“Many papers with
known validity problems
are highly cited [3].”
[3] = Bar-Ilan, J.
and Halevi, G.
2018. Temporal
characteristics
of retracted
articles
Applying the Framework:
Step 4, analyze the cited article:
What kind of support does it give?
Main Findings Support
Applying the Keystone Framework to
Knowledge Maintenance Tasks
Case Study 1: Citing Non-reproducible Code
Assess the real impact of
citing an unreliable paper
Case Study 2: Citations Supporting One Paper’s Argument
Curate a high impact paper
(citation count > 1000) to find its
keystone citations
(de Calignon et al., 2012)
Experts & Non-experts
Step 4
Flag those articles
that are potentially
impacted.
Workflow for Assessing the Citations of an Unreliable Paper
Step 2
The domain expert
develops a list of screening
questions.
Step 3
Experts/non-
experts/text mining
tools screen target
articles using the check
list.
Step 1
The domain expert
develops a generalized
argument model.
Experts
Workflow for Targeted Curation of Important Papers
Step 1
Identify main findings of the
article from abstract,
conclusion section, and
section titles.
Step 2
Construct an argument
diagram for each main
finding.
Experts
Step 3
Align citations and
citation contexts to
components in the
argument diagrams.
Step 4
Identify and categorize
keystone citation
contexts and keystone
citations.
Results from Case Study 1
Our assessment # of papers Why? What to do?
Unaffected by the code glitch 6 Didn’t directly use the protocol.
Cited it to either support
decisions they made in their
calculations or use it as
background information.
Nothing!
Affected by the code glitch 4 Followed the protocol.
Calculations supported claims
that went into abstracts or the
conclusion section.
Authors of the 4 potentially
affected papers should
double-check their results
and either amend their
claims or document how
the claims are sustained
despite the code glitch.
Results from Case Study 2
• 51 citations in the whole paper
• 5 singleton keystone citations, including
• 3 main finding singleton keystone citations support the choice of experimental
materials
• 1 pass-through keystone citation support the choice of experimental materials
• 1 singleton main finding keystone citation support the interpretation of data
• One keystone citation cluster with 3 reference items supports the
choice of experimental method.
An ad-hoc literature review from
pass-through keystone citation 1
was used to support the choice of
PSD-95 as a synaptic marker.
Research Agenda
• Scale up: Large-scale identification of keystone citations
– Test argument-based curation approaches that have already been
automated (e.g. rhetorical-based approaches, Teufel & Kan, 2009)
– Develop text mining tools to aid manual curation and screening
• Understand citation behaviors, esp. pass-through citations
• Develop a taxonomy of validity for indicating the confidence of
a reader can have in relying on or reusing the methods and
findings of a paper
References
Bar-Ilan, J. and Halevi, G. 2018. Temporal characteristics of retracted articles. Scientometrics. 116, 3 (Jun.
2018), 1771–1783. https://doi.org/10.1007/s11192-018-2802-y
Clark, T., Ciccarese, P., & Goble, C.A. (2014). Micropublications: a semantic model for claims, evidence,
arguments and annotations in biomedical communications. Journal of Biomedical Semantics, 5, 28.
https://doi.org/10.1186/2041-1480-5-28
Dinh, L., Sarol, J., Cheng, Y., Hsiao, T., Parulian, N.N., & Schneider, J. (2019). Systematic Examination of Pre- and
Post-Retraction Citations. Proceedings of ASIST, 56(1), 390-394 https://doi.org/10.1002/pra2.35
Green, N.L. (2017). Argumentation Mining in Scientific Discourse. CMNA@ICAIL.
http://ceur-ws.org/Vol-2048/paper02.pdf
Greenberg, S.A. (2009). How citation distortions create unfounded authority: analysis of a citation network.
The BMJ, 339, b2680. https://doi.org/10.1136/bmj.b2680
Teufel, S., & Kan, M. (2009). Robust Argumentative Zoning for Sensemaking in Scholarly Documents.
NLP4DL/AT4DL, 154-170. https://doi.org/10.1007/978-3-642-23160-5_10
de Calignon, A., Polydoro, M., Suárez-Calvet, M., William, C., Adamowicz, D. H., Kopeikina, K. J., Pitstick, R.,
Sahara, N., Ashe, K. H., Carlson, G. A., Spires-Jones, T. L., & Hyman, B. T. (2012). Propagation of tau
pathology in a model of early Alzheimer's disease. Neuron, 73(4), 685–697.
https://doi.org/10.1016/j.neuron.2011.11.033
Thank You!
Appendix A
Keystone Citations Found in Case Study 2
Three Types of Keystone Citation Contexts Observed
Properties of the
keystone citation
context
Removing the citation
context would weaken the
argument supporting a
main finding
Only one
paper is
cited.
The main findings of the
cited paper(s) provide
evidence to support the
argument.
Corresponding
cited article
Singleton, main-
findings support
+ + + Main-finding
keystone citation
Cluster, main-findings
support
+ - + Main-finding
keystone citation
cluster
Singleton, pass-
through support
+ + - Pass-through
keystone citation
Three main-finding keystone
citations supported the choice of
experimental materials.
Main-finding keystone citation to
support choice of material
The fourth main-finding
keystone citation supports
the interpretation of
experimental data.
Main-finding keystone citation to
support data interpretation
An ad-hoc literature review from
pass-through keystone citation 1
was used to support the choice of
PSD-95 as a synaptic marker.
Main-finding keystone citation cluster
to support material
A main-finding keystone citation
cluster supports the choice of
experimental method.
Main-finding keystone citation cluster
to support method
Appendix B
The Keystone Framework
Concepts in the Framework
• Keystone statement: any statement whose unreliability
threatens the argument for a main finding of a paper.
• Keystone citation context: citation contexts supporting
keystone statements.
– singleton vs. cluster citation context.
• A singleton citation context cites one item, e.g. ‘[2]’
• A cluster citation context cites multiple items, e.g., ‘[2, 16]’ or ‘(DeKosky and
Scheff, 1990; Scheff and Price, 2006; Terry et al., 1991)’.
Concepts in the Framework
• Keystone statement: any statement whose unreliability
threatens the argument for a main finding of a paper.
• Keystone citation context: citation contexts supporting
keystone statements.
– Number of supporting items
• A singleton citation context cites one item, e.g. ‘[2]’
• A cluster citation context cites multiple items, e.g., ‘[2, 16]’ or
‘(DeKosky and Scheff, 1990; Scheff and Price, 2006; Terry et al., 1991)’.
Concepts in the Framework
• Keystone statement: any statement whose unreliability threatens
the argument for a main finding of a paper.
• Keystone citation context: citation contexts supporting keystone
statements.
– Strength of support
• Main-findings support, if the citation context closely relates to a main finding of the
cited item.
• Pass-through support, if support can be found within the cited item but only in an
unsupported statement or a statement referencing one or more other work(s).
• No clear support, if the citation context does not clearly relate to the cited item,
either its main findings, or other statements it makes.
Concepts in the Framework
• Keystone Statement (KS):
any statement whose
unreliability threatens the
argument for a main finding
of a paper.
Main Finding
Tau pathology results in neurodegeneration
Data
support
Method
Synaptic marker
support
Keystone Statement
XXX is a good synaptic biomarker for detecting
neurodegeneration
Concepts in the Framework
• Keystone Citation Context
(KCC): citation contexts
that supports keystone
statements.
Keystone Statement
XXX is a good synaptic biomarker for detecting
neurodegeneration
Keystone Citation Context
support
Keystone Citation Contexts Continued
How many items are cited?
Singleton
Cluster
Whether the cited item’s
main findings support the
citation context?
Main-finding
support
Pass-through
support
No
support
Distinction between KS and KCC
• KSs are summation of KCCs. Domain
experts can distill the same KS from
several KCCs found in different papers.
• One study found that several citation
contexts can be distilled to a statement
about β amyloid accumulation, but the
cited papers mentioned the statement
only as “hypothesis” or not at all.
Greenberg, 2009
Statement: The accumulation of β amyloid occurs early and
precedes other abnormalities.
Citing Article Cited Article
The appearance of Aβ-
positive, noncongophilic
deposits precedes
vacuolization in IBM
muscle fibers.8 (as fact)
Some muscle fibers had
Aβ-positive
accumulations,… Those
muscle fibers,…, may
represent early changes of
IBM. (as hypothesis)
support
Appendix C
Argument-based Curation
(Teufel & Kan, 2011)
Rhetoric-based Approaches
• Extract information based on
rhetorical feature
• Limited need for domain
knowledge
• Provide a coarser argumentative
structure
• Well-automated
Argument-scheme based Approaches
Premises:
• A group of individuals G have atypical
phenotype P
• All of the individuals in G have atypical
genotype M
• Another group of individuals (controls) do not
have P
• None of controls have M
Conclusion: M may be the cause of P (in G)
(Green, 2017)
• Require deep domain analysis
• Show the logic of how a discipline
justifies its findings
• Identify potential weakness through
critical questions
• Currently depend on manual curation,
but potentially automatable through
text mining
Provenance-based Method
(Clark et al., 2014)
• Model scientists’ work process
• Most suitable for modeling
empirical research articles
• Require manual curation

More Related Content

What's hot

Data reuse and scholarly reward: understanding practice and building infrastr...
Data reuse and scholarly reward: understanding practice and building infrastr...Data reuse and scholarly reward: understanding practice and building infrastr...
Data reuse and scholarly reward: understanding practice and building infrastr...Todd Vision
 
Answering More Questions with Provenance and Query Patterns
Answering More Questions with Provenance and Query PatternsAnswering More Questions with Provenance and Query Patterns
Answering More Questions with Provenance and Query PatternsBertram Ludäscher
 
Research data and scholarly publications: going from casual acquaintances to ...
Research data and scholarly publications: going from casual acquaintances to ...Research data and scholarly publications: going from casual acquaintances to ...
Research data and scholarly publications: going from casual acquaintances to ...Todd Vision
 
The Right Metrics for Generation Open [Open Access Week 2014]
The Right Metrics for Generation Open [Open Access Week 2014]The Right Metrics for Generation Open [Open Access Week 2014]
The Right Metrics for Generation Open [Open Access Week 2014]Impactstory Team
 
Leveraging publication metadata to help overcome the data ingest bottleneck
Leveraging publication metadata to help overcome the data ingest bottleneck Leveraging publication metadata to help overcome the data ingest bottleneck
Leveraging publication metadata to help overcome the data ingest bottleneck Todd Vision
 
Research Skills Session 8: Avoid Scientific Misconduct
Research Skills Session 8: Avoid Scientific MisconductResearch Skills Session 8: Avoid Scientific Misconduct
Research Skills Session 8: Avoid Scientific MisconductNader Ale Ebrahim
 
Digital Scholar Webinar: Open reproducible research
Digital Scholar Webinar: Open reproducible researchDigital Scholar Webinar: Open reproducible research
Digital Scholar Webinar: Open reproducible researchSC CTSI at USC and CHLA
 
Wimmics seminar--drug interaction knowledge base, micropublication, open anno...
Wimmics seminar--drug interaction knowledge base, micropublication, open anno...Wimmics seminar--drug interaction knowledge base, micropublication, open anno...
Wimmics seminar--drug interaction knowledge base, micropublication, open anno...jodischneider
 
The Dryad Digital Repository: Published evolutionary data as part of the gre...
The Dryad Digital Repository: Published evolutionary data as part of the gre...The Dryad Digital Repository: Published evolutionary data as part of the gre...
The Dryad Digital Repository: Published evolutionary data as part of the gre...Todd Vision
 
Collaborative Research: Scopus & RefWorks
Collaborative Research: Scopus & RefWorksCollaborative Research: Scopus & RefWorks
Collaborative Research: Scopus & RefWorksRobin Featherstone
 
Improving the quality of my journal and increase its impact factor
Improving the quality of my journal and increase its impact factorImproving the quality of my journal and increase its impact factor
Improving the quality of my journal and increase its impact factorPortal de Periódicos UFSC
 
Digital Scholar Webinar: Transparent, Open, and Reproducible Research
Digital Scholar Webinar: Transparent, Open, and Reproducible ResearchDigital Scholar Webinar: Transparent, Open, and Reproducible Research
Digital Scholar Webinar: Transparent, Open, and Reproducible ResearchSC CTSI at USC and CHLA
 
Sense About Science: Peer Review Survey
Sense About Science: Peer Review SurveySense About Science: Peer Review Survey
Sense About Science: Peer Review SurveySenseAboutScience
 
Research Skills Session 8: Avoid Scientific Misconduct
Research Skills Session 8: Avoid Scientific MisconductResearch Skills Session 8: Avoid Scientific Misconduct
Research Skills Session 8: Avoid Scientific MisconductNader Ale Ebrahim
 

What's hot (18)

Data reuse and scholarly reward: understanding practice and building infrastr...
Data reuse and scholarly reward: understanding practice and building infrastr...Data reuse and scholarly reward: understanding practice and building infrastr...
Data reuse and scholarly reward: understanding practice and building infrastr...
 
Answering More Questions with Provenance and Query Patterns
Answering More Questions with Provenance and Query PatternsAnswering More Questions with Provenance and Query Patterns
Answering More Questions with Provenance and Query Patterns
 
An introduction to Statistical Analysis Plans
An introduction to Statistical Analysis PlansAn introduction to Statistical Analysis Plans
An introduction to Statistical Analysis Plans
 
Anonymisation 101
Anonymisation 101Anonymisation 101
Anonymisation 101
 
Research data and scholarly publications: going from casual acquaintances to ...
Research data and scholarly publications: going from casual acquaintances to ...Research data and scholarly publications: going from casual acquaintances to ...
Research data and scholarly publications: going from casual acquaintances to ...
 
The Right Metrics for Generation Open [Open Access Week 2014]
The Right Metrics for Generation Open [Open Access Week 2014]The Right Metrics for Generation Open [Open Access Week 2014]
The Right Metrics for Generation Open [Open Access Week 2014]
 
Leveraging publication metadata to help overcome the data ingest bottleneck
Leveraging publication metadata to help overcome the data ingest bottleneck Leveraging publication metadata to help overcome the data ingest bottleneck
Leveraging publication metadata to help overcome the data ingest bottleneck
 
Research Skills Session 8: Avoid Scientific Misconduct
Research Skills Session 8: Avoid Scientific MisconductResearch Skills Session 8: Avoid Scientific Misconduct
Research Skills Session 8: Avoid Scientific Misconduct
 
Fact Checking from Text
Fact Checking from TextFact Checking from Text
Fact Checking from Text
 
Digital Scholar Webinar: Open reproducible research
Digital Scholar Webinar: Open reproducible researchDigital Scholar Webinar: Open reproducible research
Digital Scholar Webinar: Open reproducible research
 
Wimmics seminar--drug interaction knowledge base, micropublication, open anno...
Wimmics seminar--drug interaction knowledge base, micropublication, open anno...Wimmics seminar--drug interaction knowledge base, micropublication, open anno...
Wimmics seminar--drug interaction knowledge base, micropublication, open anno...
 
The Dryad Digital Repository: Published evolutionary data as part of the gre...
The Dryad Digital Repository: Published evolutionary data as part of the gre...The Dryad Digital Repository: Published evolutionary data as part of the gre...
The Dryad Digital Repository: Published evolutionary data as part of the gre...
 
Collaborative Research: Scopus & RefWorks
Collaborative Research: Scopus & RefWorksCollaborative Research: Scopus & RefWorks
Collaborative Research: Scopus & RefWorks
 
Improving the quality of my journal and increase its impact factor
Improving the quality of my journal and increase its impact factorImproving the quality of my journal and increase its impact factor
Improving the quality of my journal and increase its impact factor
 
Developing a Replicable Methodology for Automated Identification of Emerging ...
Developing a Replicable Methodology for Automated Identification of Emerging ...Developing a Replicable Methodology for Automated Identification of Emerging ...
Developing a Replicable Methodology for Automated Identification of Emerging ...
 
Digital Scholar Webinar: Transparent, Open, and Reproducible Research
Digital Scholar Webinar: Transparent, Open, and Reproducible ResearchDigital Scholar Webinar: Transparent, Open, and Reproducible Research
Digital Scholar Webinar: Transparent, Open, and Reproducible Research
 
Sense About Science: Peer Review Survey
Sense About Science: Peer Review SurveySense About Science: Peer Review Survey
Sense About Science: Peer Review Survey
 
Research Skills Session 8: Avoid Scientific Misconduct
Research Skills Session 8: Avoid Scientific MisconductResearch Skills Session 8: Avoid Scientific Misconduct
Research Skills Session 8: Avoid Scientific Misconduct
 

Similar to Towards knowledge maintenance in scientific digital libraries with the keystone framework -jcdl2020--2020-08-02

Literature review niroj dahal-2017
Literature review niroj dahal-2017Literature review niroj dahal-2017
Literature review niroj dahal-2017Niroj Dahal
 
SPE ESUT TECHNICAL SERIES (ACADEMIC PROJECT WRITING AND PRESENTATION) BY ENGR...
SPE ESUT TECHNICAL SERIES (ACADEMIC PROJECT WRITING AND PRESENTATION) BY ENGR...SPE ESUT TECHNICAL SERIES (ACADEMIC PROJECT WRITING AND PRESENTATION) BY ENGR...
SPE ESUT TECHNICAL SERIES (ACADEMIC PROJECT WRITING AND PRESENTATION) BY ENGR...SPEESUTChapter
 
British Library
British LibraryBritish Library
British Libraryclarivate
 
Highly cited references in PLOS ONE and their in-text usage over time
Highly cited references in PLOS ONE and their in-text usage over timeHighly cited references in PLOS ONE and their in-text usage over time
Highly cited references in PLOS ONE and their in-text usage over timeGESIS
 
Bibliometrics, Journal Impact Factors and Maximising the Cite-ability of Jour...
Bibliometrics, Journal Impact Factors and Maximising the Cite-ability of Jour...Bibliometrics, Journal Impact Factors and Maximising the Cite-ability of Jour...
Bibliometrics, Journal Impact Factors and Maximising the Cite-ability of Jour...Jamie Bisset
 
Epistemic networks for Epistemic Commitments
Epistemic networks for Epistemic CommitmentsEpistemic networks for Epistemic Commitments
Epistemic networks for Epistemic CommitmentsSimon Knight
 
Systematic literature review technique.pptx
Systematic literature review technique.pptxSystematic literature review technique.pptx
Systematic literature review technique.pptxTANMAY DAS GUPTA
 
Ab case study assignment help Australia
Ab case study assignment help  AustraliaAb case study assignment help  Australia
Ab case study assignment help Australiaasmits kharel
 
Ab case study assignment help Australia
Ab case study assignment help AustraliaAb case study assignment help Australia
Ab case study assignment help Australiaasmits kharel
 
Scientific research and publication walk through
Scientific research and publication walk throughScientific research and publication walk through
Scientific research and publication walk throughRoshni Mehta
 
research_methodology_presentation_final.pptx
research_methodology_presentation_final.pptxresearch_methodology_presentation_final.pptx
research_methodology_presentation_final.pptxran17april2001
 
Scientific Data overview of Data Descriptors - WT Data-Literature integration...
Scientific Data overview of Data Descriptors - WT Data-Literature integration...Scientific Data overview of Data Descriptors - WT Data-Literature integration...
Scientific Data overview of Data Descriptors - WT Data-Literature integration...Susanna-Assunta Sansone
 
researchmethodologyi-140707092303-phpapp02.pdf
researchmethodologyi-140707092303-phpapp02.pdfresearchmethodologyi-140707092303-phpapp02.pdf
researchmethodologyi-140707092303-phpapp02.pdfMdali657802
 
Research Methodology Part I
Research Methodology Part IResearch Methodology Part I
Research Methodology Part IAnwar Siddiqui
 
Scientific Data and peer review session at Dryad event, May 2015
Scientific Data and peer review session at Dryad event, May 2015 Scientific Data and peer review session at Dryad event, May 2015
Scientific Data and peer review session at Dryad event, May 2015 Susanna-Assunta Sansone
 
Research Method for Business chapter 3
Research Method for Business chapter 3Research Method for Business chapter 3
Research Method for Business chapter 3Mazhar Poohlah
 

Similar to Towards knowledge maintenance in scientific digital libraries with the keystone framework -jcdl2020--2020-08-02 (20)

Mathew.ppt
Mathew.pptMathew.ppt
Mathew.ppt
 
Literature review niroj dahal-2017
Literature review niroj dahal-2017Literature review niroj dahal-2017
Literature review niroj dahal-2017
 
SPE ESUT TECHNICAL SERIES (ACADEMIC PROJECT WRITING AND PRESENTATION) BY ENGR...
SPE ESUT TECHNICAL SERIES (ACADEMIC PROJECT WRITING AND PRESENTATION) BY ENGR...SPE ESUT TECHNICAL SERIES (ACADEMIC PROJECT WRITING AND PRESENTATION) BY ENGR...
SPE ESUT TECHNICAL SERIES (ACADEMIC PROJECT WRITING AND PRESENTATION) BY ENGR...
 
Warren-Jones "Using text-mining and summarisation technology to manage the gr...
Warren-Jones "Using text-mining and summarisation technology to manage the gr...Warren-Jones "Using text-mining and summarisation technology to manage the gr...
Warren-Jones "Using text-mining and summarisation technology to manage the gr...
 
British Library
British LibraryBritish Library
British Library
 
Highly cited references in PLOS ONE and their in-text usage over time
Highly cited references in PLOS ONE and their in-text usage over timeHighly cited references in PLOS ONE and their in-text usage over time
Highly cited references in PLOS ONE and their in-text usage over time
 
Bibliometrics, Journal Impact Factors and Maximising the Cite-ability of Jour...
Bibliometrics, Journal Impact Factors and Maximising the Cite-ability of Jour...Bibliometrics, Journal Impact Factors and Maximising the Cite-ability of Jour...
Bibliometrics, Journal Impact Factors and Maximising the Cite-ability of Jour...
 
Epistemic networks for Epistemic Commitments
Epistemic networks for Epistemic CommitmentsEpistemic networks for Epistemic Commitments
Epistemic networks for Epistemic Commitments
 
Systematic literature review technique.pptx
Systematic literature review technique.pptxSystematic literature review technique.pptx
Systematic literature review technique.pptx
 
Ab case study assignment help Australia
Ab case study assignment help  AustraliaAb case study assignment help  Australia
Ab case study assignment help Australia
 
Ab case study assignment help Australia
Ab case study assignment help AustraliaAb case study assignment help Australia
Ab case study assignment help Australia
 
Citation metrics
Citation metricsCitation metrics
Citation metrics
 
Scientific research and publication walk through
Scientific research and publication walk throughScientific research and publication walk through
Scientific research and publication walk through
 
research_methodology_presentation_final.pptx
research_methodology_presentation_final.pptxresearch_methodology_presentation_final.pptx
research_methodology_presentation_final.pptx
 
Scientific Data overview of Data Descriptors - WT Data-Literature integration...
Scientific Data overview of Data Descriptors - WT Data-Literature integration...Scientific Data overview of Data Descriptors - WT Data-Literature integration...
Scientific Data overview of Data Descriptors - WT Data-Literature integration...
 
researchmethodologyi-140707092303-phpapp02.pdf
researchmethodologyi-140707092303-phpapp02.pdfresearchmethodologyi-140707092303-phpapp02.pdf
researchmethodologyi-140707092303-phpapp02.pdf
 
Writing report
Writing reportWriting report
Writing report
 
Research Methodology Part I
Research Methodology Part IResearch Methodology Part I
Research Methodology Part I
 
Scientific Data and peer review session at Dryad event, May 2015
Scientific Data and peer review session at Dryad event, May 2015 Scientific Data and peer review session at Dryad event, May 2015
Scientific Data and peer review session at Dryad event, May 2015
 
Research Method for Business chapter 3
Research Method for Business chapter 3Research Method for Business chapter 3
Research Method for Business chapter 3
 

More from jodischneider

Continued citation of bad science and what we can do about it--2021-04-20
Continued citation of bad science and what we can do about it--2021-04-20Continued citation of bad science and what we can do about it--2021-04-20
Continued citation of bad science and what we can do about it--2021-04-20jodischneider
 
Continued citation of bad science and what we can do about it--2021-02-19
Continued citation of bad science and what we can do about it--2021-02-19Continued citation of bad science and what we can do about it--2021-02-19
Continued citation of bad science and what we can do about it--2021-02-19jodischneider
 
Methods Pyramids as an Organizing Structure for Evidence-Based Medicine--SIGC...
Methods Pyramids as an Organizing Structure for Evidence-Based Medicine--SIGC...Methods Pyramids as an Organizing Structure for Evidence-Based Medicine--SIGC...
Methods Pyramids as an Organizing Structure for Evidence-Based Medicine--SIGC...jodischneider
 
Argumentation mining--an introduction for linguists--Fribourg--2019-09-02
Argumentation mining--an introduction for linguists--Fribourg--2019-09-02Argumentation mining--an introduction for linguists--Fribourg--2019-09-02
Argumentation mining--an introduction for linguists--Fribourg--2019-09-02jodischneider
 
Beyond Randomized Clinical Trials: emerging innovations in reasoning about he...
Beyond Randomized Clinical Trials: emerging innovations in reasoning about he...Beyond Randomized Clinical Trials: emerging innovations in reasoning about he...
Beyond Randomized Clinical Trials: emerging innovations in reasoning about he...jodischneider
 
Problem-citations--CrossrefLive18--2018-11-13
Problem-citations--CrossrefLive18--2018-11-13Problem-citations--CrossrefLive18--2018-11-13
Problem-citations--CrossrefLive18--2018-11-13jodischneider
 
Problematic citations--Workshop-on-Open-Citations--2018-09-03
Problematic citations--Workshop-on-Open-Citations--2018-09-03Problematic citations--Workshop-on-Open-Citations--2018-09-03
Problematic citations--Workshop-on-Open-Citations--2018-09-03jodischneider
 
Modeling Alzheimer’s Disease research claims, evidence, and arguments from a ...
Modeling Alzheimer’s Disease research claims, evidence, and arguments from a ...Modeling Alzheimer’s Disease research claims, evidence, and arguments from a ...
Modeling Alzheimer’s Disease research claims, evidence, and arguments from a ...jodischneider
 
Innovations in reasoning about health: the case of the Randomized Clinical Tr...
Innovations in reasoning about health: the case of the Randomized Clinical Tr...Innovations in reasoning about health: the case of the Randomized Clinical Tr...
Innovations in reasoning about health: the case of the Randomized Clinical Tr...jodischneider
 
Viewing universities as landscapes of scholarship, VIVO keynote, 2017-08-04
Viewing universities as landscapes of scholarship, VIVO keynote, 2017-08-04Viewing universities as landscapes of scholarship, VIVO keynote, 2017-08-04
Viewing universities as landscapes of scholarship, VIVO keynote, 2017-08-04jodischneider
 
Rhetorical moves and audience considerations in the discussion sections of ra...
Rhetorical moves and audience considerations in the discussion sections of ra...Rhetorical moves and audience considerations in the discussion sections of ra...
Rhetorical moves and audience considerations in the discussion sections of ra...jodischneider
 
Citation practices and the construction of scientific fact--ECA-facts-preconf...
Citation practices and the construction of scientific fact--ECA-facts-preconf...Citation practices and the construction of scientific fact--ECA-facts-preconf...
Citation practices and the construction of scientific fact--ECA-facts-preconf...jodischneider
 
What WikiCite can learn from biomedical citation networks--Wikicite2017--2017...
What WikiCite can learn from biomedical citation networks--Wikicite2017--2017...What WikiCite can learn from biomedical citation networks--Wikicite2017--2017...
What WikiCite can learn from biomedical citation networks--Wikicite2017--2017...jodischneider
 
Medication safety as a use case for argumentation mining, Dagstuhl seminar 16...
Medication safety as a use case for argumentation mining, Dagstuhl seminar 16...Medication safety as a use case for argumentation mining, Dagstuhl seminar 16...
Medication safety as a use case for argumentation mining, Dagstuhl seminar 16...jodischneider
 
Acquiring and representing drug-drug interaction knowledge and evidence, Litm...
Acquiring and representing drug-drug interaction knowledge and evidence, Litm...Acquiring and representing drug-drug interaction knowledge and evidence, Litm...
Acquiring and representing drug-drug interaction knowledge and evidence, Litm...jodischneider
 
Acquiring and representing drug-drug interaction knowledge and evidence, TRIA...
Acquiring and representing drug-drug interaction knowledge and evidence, TRIA...Acquiring and representing drug-drug interaction knowledge and evidence, TRIA...
Acquiring and representing drug-drug interaction knowledge and evidence, TRIA...jodischneider
 
Persons, documents, models: organising and structuring information for the We...
Persons, documents, models: organising and structuring information for the We...Persons, documents, models: organising and structuring information for the We...
Persons, documents, models: organising and structuring information for the We...jodischneider
 
Synthesizing knowledge from disagreement -- Manchester -- 2015-05-06
Synthesizing knowledge from disagreement -- Manchester -- 2015-05-06Synthesizing knowledge from disagreement -- Manchester -- 2015-05-06
Synthesizing knowledge from disagreement -- Manchester -- 2015-05-06jodischneider
 
Synthesizing knowledge from disagreement -cwi-2015-04-23
Synthesizing knowledge from disagreement -cwi-2015-04-23Synthesizing knowledge from disagreement -cwi-2015-04-23
Synthesizing knowledge from disagreement -cwi-2015-04-23jodischneider
 
Packaging ideas--nanopublications-in-the-humanities--Europeana--2015-04-21
Packaging ideas--nanopublications-in-the-humanities--Europeana--2015-04-21Packaging ideas--nanopublications-in-the-humanities--Europeana--2015-04-21
Packaging ideas--nanopublications-in-the-humanities--Europeana--2015-04-21jodischneider
 

More from jodischneider (20)

Continued citation of bad science and what we can do about it--2021-04-20
Continued citation of bad science and what we can do about it--2021-04-20Continued citation of bad science and what we can do about it--2021-04-20
Continued citation of bad science and what we can do about it--2021-04-20
 
Continued citation of bad science and what we can do about it--2021-02-19
Continued citation of bad science and what we can do about it--2021-02-19Continued citation of bad science and what we can do about it--2021-02-19
Continued citation of bad science and what we can do about it--2021-02-19
 
Methods Pyramids as an Organizing Structure for Evidence-Based Medicine--SIGC...
Methods Pyramids as an Organizing Structure for Evidence-Based Medicine--SIGC...Methods Pyramids as an Organizing Structure for Evidence-Based Medicine--SIGC...
Methods Pyramids as an Organizing Structure for Evidence-Based Medicine--SIGC...
 
Argumentation mining--an introduction for linguists--Fribourg--2019-09-02
Argumentation mining--an introduction for linguists--Fribourg--2019-09-02Argumentation mining--an introduction for linguists--Fribourg--2019-09-02
Argumentation mining--an introduction for linguists--Fribourg--2019-09-02
 
Beyond Randomized Clinical Trials: emerging innovations in reasoning about he...
Beyond Randomized Clinical Trials: emerging innovations in reasoning about he...Beyond Randomized Clinical Trials: emerging innovations in reasoning about he...
Beyond Randomized Clinical Trials: emerging innovations in reasoning about he...
 
Problem-citations--CrossrefLive18--2018-11-13
Problem-citations--CrossrefLive18--2018-11-13Problem-citations--CrossrefLive18--2018-11-13
Problem-citations--CrossrefLive18--2018-11-13
 
Problematic citations--Workshop-on-Open-Citations--2018-09-03
Problematic citations--Workshop-on-Open-Citations--2018-09-03Problematic citations--Workshop-on-Open-Citations--2018-09-03
Problematic citations--Workshop-on-Open-Citations--2018-09-03
 
Modeling Alzheimer’s Disease research claims, evidence, and arguments from a ...
Modeling Alzheimer’s Disease research claims, evidence, and arguments from a ...Modeling Alzheimer’s Disease research claims, evidence, and arguments from a ...
Modeling Alzheimer’s Disease research claims, evidence, and arguments from a ...
 
Innovations in reasoning about health: the case of the Randomized Clinical Tr...
Innovations in reasoning about health: the case of the Randomized Clinical Tr...Innovations in reasoning about health: the case of the Randomized Clinical Tr...
Innovations in reasoning about health: the case of the Randomized Clinical Tr...
 
Viewing universities as landscapes of scholarship, VIVO keynote, 2017-08-04
Viewing universities as landscapes of scholarship, VIVO keynote, 2017-08-04Viewing universities as landscapes of scholarship, VIVO keynote, 2017-08-04
Viewing universities as landscapes of scholarship, VIVO keynote, 2017-08-04
 
Rhetorical moves and audience considerations in the discussion sections of ra...
Rhetorical moves and audience considerations in the discussion sections of ra...Rhetorical moves and audience considerations in the discussion sections of ra...
Rhetorical moves and audience considerations in the discussion sections of ra...
 
Citation practices and the construction of scientific fact--ECA-facts-preconf...
Citation practices and the construction of scientific fact--ECA-facts-preconf...Citation practices and the construction of scientific fact--ECA-facts-preconf...
Citation practices and the construction of scientific fact--ECA-facts-preconf...
 
What WikiCite can learn from biomedical citation networks--Wikicite2017--2017...
What WikiCite can learn from biomedical citation networks--Wikicite2017--2017...What WikiCite can learn from biomedical citation networks--Wikicite2017--2017...
What WikiCite can learn from biomedical citation networks--Wikicite2017--2017...
 
Medication safety as a use case for argumentation mining, Dagstuhl seminar 16...
Medication safety as a use case for argumentation mining, Dagstuhl seminar 16...Medication safety as a use case for argumentation mining, Dagstuhl seminar 16...
Medication safety as a use case for argumentation mining, Dagstuhl seminar 16...
 
Acquiring and representing drug-drug interaction knowledge and evidence, Litm...
Acquiring and representing drug-drug interaction knowledge and evidence, Litm...Acquiring and representing drug-drug interaction knowledge and evidence, Litm...
Acquiring and representing drug-drug interaction knowledge and evidence, Litm...
 
Acquiring and representing drug-drug interaction knowledge and evidence, TRIA...
Acquiring and representing drug-drug interaction knowledge and evidence, TRIA...Acquiring and representing drug-drug interaction knowledge and evidence, TRIA...
Acquiring and representing drug-drug interaction knowledge and evidence, TRIA...
 
Persons, documents, models: organising and structuring information for the We...
Persons, documents, models: organising and structuring information for the We...Persons, documents, models: organising and structuring information for the We...
Persons, documents, models: organising and structuring information for the We...
 
Synthesizing knowledge from disagreement -- Manchester -- 2015-05-06
Synthesizing knowledge from disagreement -- Manchester -- 2015-05-06Synthesizing knowledge from disagreement -- Manchester -- 2015-05-06
Synthesizing knowledge from disagreement -- Manchester -- 2015-05-06
 
Synthesizing knowledge from disagreement -cwi-2015-04-23
Synthesizing knowledge from disagreement -cwi-2015-04-23Synthesizing knowledge from disagreement -cwi-2015-04-23
Synthesizing knowledge from disagreement -cwi-2015-04-23
 
Packaging ideas--nanopublications-in-the-humanities--Europeana--2015-04-21
Packaging ideas--nanopublications-in-the-humanities--Europeana--2015-04-21Packaging ideas--nanopublications-in-the-humanities--Europeana--2015-04-21
Packaging ideas--nanopublications-in-the-humanities--Europeana--2015-04-21
 

Recently uploaded

Gliese 12 b, a temperate Earth-sized planet at 12 parsecs discovered with TES...
Gliese 12 b, a temperate Earth-sized planet at 12 parsecs discovered with TES...Gliese 12 b, a temperate Earth-sized planet at 12 parsecs discovered with TES...
Gliese 12 b, a temperate Earth-sized planet at 12 parsecs discovered with TES...Sérgio Sacani
 
Cell Immobilization Methods and Applications.pptx
Cell Immobilization Methods and Applications.pptxCell Immobilization Methods and Applications.pptx
Cell Immobilization Methods and Applications.pptxCherry
 
Erythropoiesis- Dr.E. Muralinath-C Kalyan
Erythropoiesis- Dr.E. Muralinath-C KalyanErythropoiesis- Dr.E. Muralinath-C Kalyan
Erythropoiesis- Dr.E. Muralinath-C Kalyanmuralinath2
 
In vitro androgenesis ...............pptx
In vitro androgenesis ...............pptxIn vitro androgenesis ...............pptx
In vitro androgenesis ...............pptxCherry
 
Structures and textures of metamorphic rocks
Structures and textures of metamorphic rocksStructures and textures of metamorphic rocks
Structures and textures of metamorphic rockskumarmathi863
 
THE IMPORTANCE OF MARTIAN ATMOSPHERE SAMPLE RETURN.
THE IMPORTANCE OF MARTIAN ATMOSPHERE SAMPLE RETURN.THE IMPORTANCE OF MARTIAN ATMOSPHERE SAMPLE RETURN.
THE IMPORTANCE OF MARTIAN ATMOSPHERE SAMPLE RETURN.Sérgio Sacani
 
Triploidy ...............................pptx
Triploidy ...............................pptxTriploidy ...............................pptx
Triploidy ...............................pptxCherry
 
Ostiguy & Panizza & Moffitt (eds.) - Populism in Global Perspective. A Perfor...
Ostiguy & Panizza & Moffitt (eds.) - Populism in Global Perspective. A Perfor...Ostiguy & Panizza & Moffitt (eds.) - Populism in Global Perspective. A Perfor...
Ostiguy & Panizza & Moffitt (eds.) - Populism in Global Perspective. A Perfor...frank0071
 
Aerodynamics. flippatterncn5tm5ttnj6nmnynyppt
Aerodynamics. flippatterncn5tm5ttnj6nmnynypptAerodynamics. flippatterncn5tm5ttnj6nmnynyppt
Aerodynamics. flippatterncn5tm5ttnj6nmnynypptsreddyrahul
 
Multi-source connectivity as the driver of solar wind variability in the heli...
Multi-source connectivity as the driver of solar wind variability in the heli...Multi-source connectivity as the driver of solar wind variability in the heli...
Multi-source connectivity as the driver of solar wind variability in the heli...Sérgio Sacani
 
Hemoglobin metabolism: C Kalyan & E. Muralinath
Hemoglobin metabolism: C Kalyan & E. MuralinathHemoglobin metabolism: C Kalyan & E. Muralinath
Hemoglobin metabolism: C Kalyan & E. Muralinathmuralinath2
 
Unveiling The Crucial Role Of Cobalt In Plant
Unveiling The Crucial Role Of Cobalt In PlantUnveiling The Crucial Role Of Cobalt In Plant
Unveiling The Crucial Role Of Cobalt In PlantHimanshu Pandey
 
electrochemical gas sensors and their uses.pptx
electrochemical gas sensors and their uses.pptxelectrochemical gas sensors and their uses.pptx
electrochemical gas sensors and their uses.pptxlCWU
 
Climate extremes likely to drive land mammal extinction during next supercont...
Climate extremes likely to drive land mammal extinction during next supercont...Climate extremes likely to drive land mammal extinction during next supercont...
Climate extremes likely to drive land mammal extinction during next supercont...Sérgio Sacani
 
Plasmapheresis - Dr. E. Muralinath - Kalyan . C.pptx
Plasmapheresis - Dr. E. Muralinath - Kalyan . C.pptxPlasmapheresis - Dr. E. Muralinath - Kalyan . C.pptx
Plasmapheresis - Dr. E. Muralinath - Kalyan . C.pptxmuralinath2
 
METHODS OF TRANSCRIPTOME ANALYSIS....pptx
METHODS OF TRANSCRIPTOME ANALYSIS....pptxMETHODS OF TRANSCRIPTOME ANALYSIS....pptx
METHODS OF TRANSCRIPTOME ANALYSIS....pptxCherry
 
Mitosis...............................pptx
Mitosis...............................pptxMitosis...............................pptx
Mitosis...............................pptxCherry
 
Extensive Pollution of Uranus and Neptune’s Atmospheres by Upsweep of Icy Mat...
Extensive Pollution of Uranus and Neptune’s Atmospheres by Upsweep of Icy Mat...Extensive Pollution of Uranus and Neptune’s Atmospheres by Upsweep of Icy Mat...
Extensive Pollution of Uranus and Neptune’s Atmospheres by Upsweep of Icy Mat...Sérgio Sacani
 
Plasma proteins_ Dr.Muralinath_Dr.c. kalyan
Plasma proteins_ Dr.Muralinath_Dr.c. kalyanPlasma proteins_ Dr.Muralinath_Dr.c. kalyan
Plasma proteins_ Dr.Muralinath_Dr.c. kalyanmuralinath2
 
Phytogeography........................pptx
Phytogeography........................pptxPhytogeography........................pptx
Phytogeography........................pptxCherry
 

Recently uploaded (20)

Gliese 12 b, a temperate Earth-sized planet at 12 parsecs discovered with TES...
Gliese 12 b, a temperate Earth-sized planet at 12 parsecs discovered with TES...Gliese 12 b, a temperate Earth-sized planet at 12 parsecs discovered with TES...
Gliese 12 b, a temperate Earth-sized planet at 12 parsecs discovered with TES...
 
Cell Immobilization Methods and Applications.pptx
Cell Immobilization Methods and Applications.pptxCell Immobilization Methods and Applications.pptx
Cell Immobilization Methods and Applications.pptx
 
Erythropoiesis- Dr.E. Muralinath-C Kalyan
Erythropoiesis- Dr.E. Muralinath-C KalyanErythropoiesis- Dr.E. Muralinath-C Kalyan
Erythropoiesis- Dr.E. Muralinath-C Kalyan
 
In vitro androgenesis ...............pptx
In vitro androgenesis ...............pptxIn vitro androgenesis ...............pptx
In vitro androgenesis ...............pptx
 
Structures and textures of metamorphic rocks
Structures and textures of metamorphic rocksStructures and textures of metamorphic rocks
Structures and textures of metamorphic rocks
 
THE IMPORTANCE OF MARTIAN ATMOSPHERE SAMPLE RETURN.
THE IMPORTANCE OF MARTIAN ATMOSPHERE SAMPLE RETURN.THE IMPORTANCE OF MARTIAN ATMOSPHERE SAMPLE RETURN.
THE IMPORTANCE OF MARTIAN ATMOSPHERE SAMPLE RETURN.
 
Triploidy ...............................pptx
Triploidy ...............................pptxTriploidy ...............................pptx
Triploidy ...............................pptx
 
Ostiguy & Panizza & Moffitt (eds.) - Populism in Global Perspective. A Perfor...
Ostiguy & Panizza & Moffitt (eds.) - Populism in Global Perspective. A Perfor...Ostiguy & Panizza & Moffitt (eds.) - Populism in Global Perspective. A Perfor...
Ostiguy & Panizza & Moffitt (eds.) - Populism in Global Perspective. A Perfor...
 
Aerodynamics. flippatterncn5tm5ttnj6nmnynyppt
Aerodynamics. flippatterncn5tm5ttnj6nmnynypptAerodynamics. flippatterncn5tm5ttnj6nmnynyppt
Aerodynamics. flippatterncn5tm5ttnj6nmnynyppt
 
Multi-source connectivity as the driver of solar wind variability in the heli...
Multi-source connectivity as the driver of solar wind variability in the heli...Multi-source connectivity as the driver of solar wind variability in the heli...
Multi-source connectivity as the driver of solar wind variability in the heli...
 
Hemoglobin metabolism: C Kalyan & E. Muralinath
Hemoglobin metabolism: C Kalyan & E. MuralinathHemoglobin metabolism: C Kalyan & E. Muralinath
Hemoglobin metabolism: C Kalyan & E. Muralinath
 
Unveiling The Crucial Role Of Cobalt In Plant
Unveiling The Crucial Role Of Cobalt In PlantUnveiling The Crucial Role Of Cobalt In Plant
Unveiling The Crucial Role Of Cobalt In Plant
 
electrochemical gas sensors and their uses.pptx
electrochemical gas sensors and their uses.pptxelectrochemical gas sensors and their uses.pptx
electrochemical gas sensors and their uses.pptx
 
Climate extremes likely to drive land mammal extinction during next supercont...
Climate extremes likely to drive land mammal extinction during next supercont...Climate extremes likely to drive land mammal extinction during next supercont...
Climate extremes likely to drive land mammal extinction during next supercont...
 
Plasmapheresis - Dr. E. Muralinath - Kalyan . C.pptx
Plasmapheresis - Dr. E. Muralinath - Kalyan . C.pptxPlasmapheresis - Dr. E. Muralinath - Kalyan . C.pptx
Plasmapheresis - Dr. E. Muralinath - Kalyan . C.pptx
 
METHODS OF TRANSCRIPTOME ANALYSIS....pptx
METHODS OF TRANSCRIPTOME ANALYSIS....pptxMETHODS OF TRANSCRIPTOME ANALYSIS....pptx
METHODS OF TRANSCRIPTOME ANALYSIS....pptx
 
Mitosis...............................pptx
Mitosis...............................pptxMitosis...............................pptx
Mitosis...............................pptx
 
Extensive Pollution of Uranus and Neptune’s Atmospheres by Upsweep of Icy Mat...
Extensive Pollution of Uranus and Neptune’s Atmospheres by Upsweep of Icy Mat...Extensive Pollution of Uranus and Neptune’s Atmospheres by Upsweep of Icy Mat...
Extensive Pollution of Uranus and Neptune’s Atmospheres by Upsweep of Icy Mat...
 
Plasma proteins_ Dr.Muralinath_Dr.c. kalyan
Plasma proteins_ Dr.Muralinath_Dr.c. kalyanPlasma proteins_ Dr.Muralinath_Dr.c. kalyan
Plasma proteins_ Dr.Muralinath_Dr.c. kalyan
 
Phytogeography........................pptx
Phytogeography........................pptxPhytogeography........................pptx
Phytogeography........................pptx
 

Towards knowledge maintenance in scientific digital libraries with the keystone framework -jcdl2020--2020-08-02

  • 1. Towards Knowledge Maintenance in Scientific Digital Libraries with the Keystone Framework Yuanxi Fu & Jodi Schneider School of Information Sciences University of Illinois at Urbana-Champaign Presentation for JCDL 2020, Virtual, 2020-08-02
  • 2. Motivation: Need for KNOWLEDGE MAINTENANCE • When you’re not an expert, how do you judge papers? – Recency – Citation count – Other heuristics • Literature becomes obsolete but usually that’s not explicit. • These errors can be passed on & can lead to misinterpretations. – Most research needs information from cognate fields.
  • 3. How big is retraction & citation to retraction • Over 600,000 articles directly cite a retracted paper. • The Retraction Watch Database lists over 19,000 retracted publications. • In biomedicine 94% of retracted papers have received at least one citation, with an average citation count of 35 (Dinh, …, Schneider 2019).
  • 4. Motivating Questions • Are papers citing a retracted paper necessarily wrong? • Does it matter when citing authors make use of a paper whose findings are no longer considered valid? • When DOES the citation matter? • Could we selectively alert authors who cite a retracted or abandoned paper?
  • 6. Under our framework: 1) A scientific research paper puts forward at least one main finding, along with a logical argument, giving reasons and evidence to support the main finding. 2) The main finding is accepted (or not) on the basis of the logical argument. 3) Evidence from earlier literature may be incorporated into the argument by citing a paper and presenting it as support, using a citation context.
  • 7. Main Finding Arguments support support Data Methods Citations … Applying the Framework Step 1, model the argument
  • 8. Citing ArticleCited Article Citation Context “Many papers with known validity problems are highly cited [3].” Our paper [3] = Bar-Ilan, J. and Halevi, G. 2018. Temporal characteristics of retracted articles Applying the Framework: Step 2, find the citation contexts that contribute to the argument
  • 9. Applying the Framework: Step 3, analyze the citation context: How many items are cited? Citation Context “Many papers with known validity problems are highly cited [3].” Singleton Cluster “Digital library applications of argumentation theory include argument-based retrieval [21, 30]”
  • 10. Cited Article Citation Context “Many papers with known validity problems are highly cited [3].” [3] = Bar-Ilan, J. and Halevi, G. 2018. Temporal characteristics of retracted articles Applying the Framework: Step 4, analyze the cited article: What kind of support does it give? Main Findings Support
  • 11. Applying the Keystone Framework to Knowledge Maintenance Tasks
  • 12. Case Study 1: Citing Non-reproducible Code Assess the real impact of citing an unreliable paper
  • 13. Case Study 2: Citations Supporting One Paper’s Argument Curate a high impact paper (citation count > 1000) to find its keystone citations (de Calignon et al., 2012)
  • 14. Experts & Non-experts Step 4 Flag those articles that are potentially impacted. Workflow for Assessing the Citations of an Unreliable Paper Step 2 The domain expert develops a list of screening questions. Step 3 Experts/non- experts/text mining tools screen target articles using the check list. Step 1 The domain expert develops a generalized argument model. Experts
  • 15. Workflow for Targeted Curation of Important Papers Step 1 Identify main findings of the article from abstract, conclusion section, and section titles. Step 2 Construct an argument diagram for each main finding. Experts Step 3 Align citations and citation contexts to components in the argument diagrams. Step 4 Identify and categorize keystone citation contexts and keystone citations.
  • 16. Results from Case Study 1 Our assessment # of papers Why? What to do? Unaffected by the code glitch 6 Didn’t directly use the protocol. Cited it to either support decisions they made in their calculations or use it as background information. Nothing! Affected by the code glitch 4 Followed the protocol. Calculations supported claims that went into abstracts or the conclusion section. Authors of the 4 potentially affected papers should double-check their results and either amend their claims or document how the claims are sustained despite the code glitch.
  • 17. Results from Case Study 2 • 51 citations in the whole paper • 5 singleton keystone citations, including • 3 main finding singleton keystone citations support the choice of experimental materials • 1 pass-through keystone citation support the choice of experimental materials • 1 singleton main finding keystone citation support the interpretation of data • One keystone citation cluster with 3 reference items supports the choice of experimental method.
  • 18. An ad-hoc literature review from pass-through keystone citation 1 was used to support the choice of PSD-95 as a synaptic marker.
  • 19. Research Agenda • Scale up: Large-scale identification of keystone citations – Test argument-based curation approaches that have already been automated (e.g. rhetorical-based approaches, Teufel & Kan, 2009) – Develop text mining tools to aid manual curation and screening • Understand citation behaviors, esp. pass-through citations • Develop a taxonomy of validity for indicating the confidence of a reader can have in relying on or reusing the methods and findings of a paper
  • 20. References Bar-Ilan, J. and Halevi, G. 2018. Temporal characteristics of retracted articles. Scientometrics. 116, 3 (Jun. 2018), 1771–1783. https://doi.org/10.1007/s11192-018-2802-y Clark, T., Ciccarese, P., & Goble, C.A. (2014). Micropublications: a semantic model for claims, evidence, arguments and annotations in biomedical communications. Journal of Biomedical Semantics, 5, 28. https://doi.org/10.1186/2041-1480-5-28 Dinh, L., Sarol, J., Cheng, Y., Hsiao, T., Parulian, N.N., & Schneider, J. (2019). Systematic Examination of Pre- and Post-Retraction Citations. Proceedings of ASIST, 56(1), 390-394 https://doi.org/10.1002/pra2.35 Green, N.L. (2017). Argumentation Mining in Scientific Discourse. CMNA@ICAIL. http://ceur-ws.org/Vol-2048/paper02.pdf Greenberg, S.A. (2009). How citation distortions create unfounded authority: analysis of a citation network. The BMJ, 339, b2680. https://doi.org/10.1136/bmj.b2680 Teufel, S., & Kan, M. (2009). Robust Argumentative Zoning for Sensemaking in Scholarly Documents. NLP4DL/AT4DL, 154-170. https://doi.org/10.1007/978-3-642-23160-5_10 de Calignon, A., Polydoro, M., Suárez-Calvet, M., William, C., Adamowicz, D. H., Kopeikina, K. J., Pitstick, R., Sahara, N., Ashe, K. H., Carlson, G. A., Spires-Jones, T. L., & Hyman, B. T. (2012). Propagation of tau pathology in a model of early Alzheimer's disease. Neuron, 73(4), 685–697. https://doi.org/10.1016/j.neuron.2011.11.033
  • 22. Appendix A Keystone Citations Found in Case Study 2
  • 23. Three Types of Keystone Citation Contexts Observed Properties of the keystone citation context Removing the citation context would weaken the argument supporting a main finding Only one paper is cited. The main findings of the cited paper(s) provide evidence to support the argument. Corresponding cited article Singleton, main- findings support + + + Main-finding keystone citation Cluster, main-findings support + - + Main-finding keystone citation cluster Singleton, pass- through support + + - Pass-through keystone citation
  • 24. Three main-finding keystone citations supported the choice of experimental materials. Main-finding keystone citation to support choice of material
  • 25. The fourth main-finding keystone citation supports the interpretation of experimental data. Main-finding keystone citation to support data interpretation
  • 26. An ad-hoc literature review from pass-through keystone citation 1 was used to support the choice of PSD-95 as a synaptic marker. Main-finding keystone citation cluster to support material
  • 27. A main-finding keystone citation cluster supports the choice of experimental method. Main-finding keystone citation cluster to support method
  • 29. Concepts in the Framework • Keystone statement: any statement whose unreliability threatens the argument for a main finding of a paper. • Keystone citation context: citation contexts supporting keystone statements. – singleton vs. cluster citation context. • A singleton citation context cites one item, e.g. ‘[2]’ • A cluster citation context cites multiple items, e.g., ‘[2, 16]’ or ‘(DeKosky and Scheff, 1990; Scheff and Price, 2006; Terry et al., 1991)’.
  • 30. Concepts in the Framework • Keystone statement: any statement whose unreliability threatens the argument for a main finding of a paper. • Keystone citation context: citation contexts supporting keystone statements. – Number of supporting items • A singleton citation context cites one item, e.g. ‘[2]’ • A cluster citation context cites multiple items, e.g., ‘[2, 16]’ or ‘(DeKosky and Scheff, 1990; Scheff and Price, 2006; Terry et al., 1991)’.
  • 31. Concepts in the Framework • Keystone statement: any statement whose unreliability threatens the argument for a main finding of a paper. • Keystone citation context: citation contexts supporting keystone statements. – Strength of support • Main-findings support, if the citation context closely relates to a main finding of the cited item. • Pass-through support, if support can be found within the cited item but only in an unsupported statement or a statement referencing one or more other work(s). • No clear support, if the citation context does not clearly relate to the cited item, either its main findings, or other statements it makes.
  • 32. Concepts in the Framework • Keystone Statement (KS): any statement whose unreliability threatens the argument for a main finding of a paper. Main Finding Tau pathology results in neurodegeneration Data support Method Synaptic marker support Keystone Statement XXX is a good synaptic biomarker for detecting neurodegeneration
  • 33. Concepts in the Framework • Keystone Citation Context (KCC): citation contexts that supports keystone statements. Keystone Statement XXX is a good synaptic biomarker for detecting neurodegeneration Keystone Citation Context support
  • 34. Keystone Citation Contexts Continued How many items are cited? Singleton Cluster Whether the cited item’s main findings support the citation context? Main-finding support Pass-through support No support
  • 35. Distinction between KS and KCC • KSs are summation of KCCs. Domain experts can distill the same KS from several KCCs found in different papers. • One study found that several citation contexts can be distilled to a statement about β amyloid accumulation, but the cited papers mentioned the statement only as “hypothesis” or not at all. Greenberg, 2009 Statement: The accumulation of β amyloid occurs early and precedes other abnormalities. Citing Article Cited Article The appearance of Aβ- positive, noncongophilic deposits precedes vacuolization in IBM muscle fibers.8 (as fact) Some muscle fibers had Aβ-positive accumulations,… Those muscle fibers,…, may represent early changes of IBM. (as hypothesis) support
  • 37. (Teufel & Kan, 2011) Rhetoric-based Approaches • Extract information based on rhetorical feature • Limited need for domain knowledge • Provide a coarser argumentative structure • Well-automated
  • 38. Argument-scheme based Approaches Premises: • A group of individuals G have atypical phenotype P • All of the individuals in G have atypical genotype M • Another group of individuals (controls) do not have P • None of controls have M Conclusion: M may be the cause of P (in G) (Green, 2017) • Require deep domain analysis • Show the logic of how a discipline justifies its findings • Identify potential weakness through critical questions • Currently depend on manual curation, but potentially automatable through text mining
  • 39. Provenance-based Method (Clark et al., 2014) • Model scientists’ work process • Most suitable for modeling empirical research articles • Require manual curation