This document analyzes how highly cited references are used over time in the journal PLOS ONE. It finds that: 1) Over half of highly cited references are cited in the Method section. References with longer citation intervals (older at time of citing) are more likely to be cited in the Method section. 2) The proportion of sections changes over time, with newer references more likely to be cited in the Introduction and older references in the Method section. 3) The average number of co-citations in a sentence declines as the citation interval increases, meaning older references tend to be cited alone rather than with other references in a sentence.

1. Highly cited references in PLOS
ONE and their in-text usage over
time
Wolfgang Otto, Behnam Ghavimi, Philipp Mayr,
Rajesh Piryani, Vivek Kumar Singh
GESIS - Leibniz Institute for the Social Sciences
17th ISSI Conference in Rome, Italy
3rd September 2019

2. Introduction
• Motivation and background
– Reception of highly cited papers over time
– Analyse citation contexts concerning their position in
the text and their age at the time of citing
• Classical Bibliometrics
– Paper X cites Paper Y;
Paper Y is co-cited with Z
• Bibliometrics with full text information
– Paper X cites Paper Y n times in Introduction,
n times in Methods; citations are mostly neural
2
cites
cites

3. Background
• What is a citation context?
– We define a citation context as the sentence in which a concrete
citation occurs.
• Example: Citation context in a citing paper
– “Western immunoblotting was performed according to
Towbin method [60]”.
The matching reference string in the citing paper:
[60] Towbin, H., Staehelin, T., & Gordon, J. (1979). Electrophoretic
transfer of proteins from polyacrylamide gels to nitrocellulose sheets:
procedure and some applications. Proceedings of the National Academy of
Sciences, 76(9), 4350-4354.
3
The cited paper

4. Related Work
• NLP (n-grams) for identifying the function of
citations (Bertin et al. 2016)
• Characteristics of in-text citations in different
disciplines (Boyack et al. 2018)
• Method papers comprise 90 percent of 100
most cited biomedical papers; specific word
use in method (utility words) and non-method
(hedging words) papers (Small 2018)
4

5. Dataset: PLOS one corpus
• Highly cited PubMed papers in
a corpus of the journal PLOS
ONE during the period 2006 -
2017
• Highly cited papers: all PubMed
papers (n=666) which are cited
in more than 100 PLOS ONE
papers -> Top-666
• Only references with PubMed
ids in the reference part of the
citing publications have been
considered
5
Description value
No. of citing PLOS ONE papers 176,856
No. of extracted citation contexts 31,746,769
Publication period of citing papers 2006-2017
Publication period of cited papers 1951-2015
No. of relevant citing PLOS ONE
publications
62,127
No. of relevant references (highly
cited references)
666
No. of relevant extracted citation
contexts
173,630
No. of referenced objects per citation
context (mean)
1.78
Overview of the corpus by selected parameters.
The underlying dataset of our study contains all relevant contexts citing the top-666
referenced publications and is available under https://github.com/Scientotext/PLOS-ONE-Dataset

6. Dataset: citation context details
Citation context and
collected metadata
• Co-citation count
on the citation
context level
• Citation interval:
time distance of
the citing paper to
the time of
publication of the
reference
6
DOI of citing
paper
10.1371/journal.pone.0013678
Citation context “The molecular mechanisms of nuclear
reprogramming are still unsolved
although recent reports have shown that
reprogramming of human somatic cells
can be achieved in vitro by retroviral
expression of four transcription factors
creating induced pluripotent stem (iPS)
cells, which are comparable to ES cells
[1], [2], [3], [4].”
Pubmed-ID of
reference
18035408
IMRaD Section Introduction
Pub. year of the
citing paper
2010
Pub. year of the
reference
2007
Citation interval 3 years (2007-2010)Example of extracted data for one citation context.

7. Research questions
• RQ 1: In which IMRaD section are the top-
666 references cited?
• RQ 2: How is the proportion of concrete
IMRaD sections evolving over time?
• RQ 3: Is the number of co-citations on
citation context level declining when the
citation interval is becoming longer?
7

8. Results: IMRaD
8
Share of section where all top-666 references are cited.
• For more than half of our references the most used section for citing is
the method section
• A larger group of references which are mainly used in introduction and
discussion
• No correlation of the section usage with the mean citation interval of the
top-666.

9. Results: Citation interval
9
The proportion of IMRaD sections over time.
• We see a change for the most frequently used section based on the
citation interval at the beginning of a reference lifetime (0-3 years)
• A reference with a high citation interval, i.e. an older publication at the
time of citing, is likely to be used in a method part.

10. Results: Sentence-level co-citation
10
• The mean value is between 1 and 2.
• Mean value is declining for longer
citation intervals.
• Mean co-citation number in the
context given a citation interval
for IMRaD sections

11. Limitations
• Dataset limited to PubMed (biomedical)
papers
• Limited to references with PubMed Id in
the reference section
• Few data for larger citation intervals
11

12. Next steps & conclusions
• Replicate results concerning a larger data corpus
and different disciplines
• Consider the potential effects of various citation
intervals more carefully
• Deeper analysis of author name mentions in
citations contexts
• Analysis of word usage (e.g., adjective, verb, and
noun) in citation contexts of highly cited papers
and changes over time (Bertin et al., 2016; Small
2018)
12

13. Thank you
Dr Philipp Mayr
GESIS - Leibniz Institute for the Social Sciences, Germany
Team Leader Information & Data Retrieval
Email: philipp.mayr@gesis.org
Twitter: @philipp_mayr
13
Paper available:
https://arxiv.org/abs/1903.11693 Indo-German Joint Research Project titled
‘Design of a Sciento-text Computational
Framework for Retrieval and Contextual
Recommendations of High-Quality Scholarly
Articles’ (Grant No.
DST/INT/FRG/DAAD/P-28/2017)
The underlying dataset is available under
https://github.com/Scientotext/PLOS-
ONE-Dataset