This panel addresses the application of bibliometric and scientometric methods applied in the assessment of journals and the research they publish. The measurement of impact of scientific journals through citations has its origin in the documentalists activities at the late nineteenth century to organize the publications of specific areas. The unfolding of these efforts soon undertook quantitative approaches aiming at understanding trends which allowed us to establish, for example, the nucleus of journals and authors in the various areas, making it an important input for science historians and sociologists. Regarding the treatment of scientific information, the essays of the first half of the twentieth century were materialized into a system that would offer a new form of information retrieval – in the diachronic sense – allowing to identify the relation that literature establishes from the publishing of an article. This relationship, which expresses the repercussion of a new knowledge in the literature, did not take much time to attribute the idea of scientific impact, whose expression occurs through citations. The citation index then revolutionizes the way of accessing literature in the second half of that century, at the same time as it becomes a unique source for impact indicators, which from there would represent the world science in evaluative processes around of the world. At the turn of the twenty first century, many factors – such as subscription costs, the underrepresentation of the scientific literature of non-English-speaking countries, as well as the different practices of scientific communication among areas of knowledge – have given rise to initiatives aimed at broader sources of information, while at the same time facilitating free access to scientific information. However, besides the access issue, the already established need for impact measurement could not be ignored in order to provide the consolidated processes of evaluation of scientific output with more adequate indicators. In this sense, it is necessary that the new information sources, taking advantage of the new methodologies proposed by the community specialized in quantitative methods of science evaluation, may contribute with indicators that make the assessment of national (Brazilian) scientific output more adequate to the national scenario. In doing so, it is hoped that the group’s discussions will contribute not only to evidence the best that has been produced locally, but also to allow scientific journals edited nationally, particularly those of the SciELO Network, to have their impact recognized, allowing circulation of global and inclusive scientific knowledge. Syllabus Information sources for generating impact indicators; specificity of the culture of scientific communication in the different areas, especially the Human and Social Sciences; the limitations of the Impact Factor and mainstream journal-based indicators; methodologies for…

1. Journal Impact Factors:
history, limitations and
adverse effects
Vincent Larivière
Cassidy R. Sugimoto
vincent.lariviere@umontreal.ca
@lariviev
crc.ebsi.umontreal.ca

2. What will this guy talk about?
•Brief history of journal-level indicators
•ISI and the Journal Citation Report (JCR)
•Critiques of the JCR
•Adverse effects
•Why do we need journal level indicators?

3. Gross & Gross (1927)
College Libraries and Chemical Education
• Chemistry professors (US college)
o “What files of scientific periodicals are needed in a college library to
successfully prepare the student for advance work, taking into
consideration also those materials necessary for the stimulation and
intellectual development of the faculty?”
• Citations provide an “objective” indicator
o “One way to answer this question would be merely to sit down and
compile a list of those journals which one considers indispensable.
Such a procedure might prove eminently successful in certain cases,
but it seems reasonably certain that often the result would be
seasoned too much by the needs, likes and dislikes of the compiler.”

4. Gross & Gross (1927)
• Compiled references made by 1926 papers published in
the Journal of the American Chemical Society.
o Excluded journal self-citations
o Excluded cited papers from 1926 (to account for diffusion lag)
• This resulted in 3,633 references to 247 journals.

5. Follow-up analyses
• Several papers (≈20) scaled up the method devised by
Gross & Gross
• Growing datasets (i.e. manual work)
• Mostly mono-disciplinary
• All from the U.S.
Archambault, É., & Larivière, V. (2009). History of the journal impact factor: Contingencies and
consequences. Scientometrics, 79(3), 635-649.

6. Institute for Scientific Information
• First large-scale citation index
• Inspired on Shepard’s Citations—created in
1873 for legal research
• Science Citation Index (1961)
o NSF and NIH funded; cost of 400,000$
o Genetics Citation Index (subset)
o Officially launched in 1964
• Sample of most cited journals (Bradford’s
Law)

7. • Garfield and Sher (1963): exploration of the
1961 citation data (1.4 million references)
• First mention of a Journal Impact Factor:
o “One of the most interesting correlations is the
“journal impact factor.” In the usual citation count
methods, as, e.g., Gross and Gross (4), the
importance of n journal is determined by the absolute
number of citations to it.”
o “The first step in obtaining a more meaningful measure
of importance is to divide the number of times a journal
is cited by the number of articles that journal has
published.”
Institute for Scientific Information (2)

8. Garfield gets scooped!
• Martyn, J., Gilchrist, A. (1968), An Evaluation of
British Scientific Journals. Aslib.
• Based on the Science Citation Index!
o “All citations to 1,842 British journals in the broad areas of science and
technology which appeared in the 1965 Science Citation Index for the
years 1964 and 1963.”
• Data by discipline
• Asymmetry between numerator and denominator
o All citations / citable items
• 2 year citation window

9. 1975 Journal Citation Report
• First Journal Citation Reports (JCR) and reporting of the
Journal Impact Factor
• Based on more than 4.2 million references made in 1974
by 400,000 papers published in about 2,400 journals
• Justifies citations as indicators of usage:
o “The more frequently a journal’s articles are cited, the more the world’s
scientific community implies that it finds the journal to be a carrier of
useful information”
• For librarians: decisions about subscriptions
• For researchers: identifying publication venues, especially
multidisciplinarity ones
• Garfield did not advocate for using the JCR to identify
elite journals.

10. Impact Factor
• Average number of citations received by articles
published in a journal two years after their
publication
• Impact Factor of a given journal in 2017 would be
calculated as follows:
Number of citations received in 2017 by documents published
in the journal in 2015-2016
______________________________________
Number of articles and reviews
published in the journal in 2016-2017

11. Asymmetry numerator / denominator
Number and proportion of citations received by articles,
reviews, non-citable items, and unmatched citations, for four
journals from the field of biochemistry and molecular biology, as
well as Nature and Science, 2014-2015 papers and 2016
citations
Article Review
Non-Citable
Items
Unmatched
Citations
N % N % N % N %
Cell 20,885 78.6% 3,068 11.5% 601 2.3% 2,016 7.6% 869 27.564 30.410 10.3%
Nat. Chem. Biol. 3,263 77.4% 378 9.0% 217 5.1% 356 8.4% 268 13.586 15.066 10.9%
PLOS Biol. 3,088 85.3% 6 0.2% 237 6.5% 290 8.0% 384 8.057 9.797 21.6%
FASEB J. 3,650 74.6% 235 4.8% 203 4.2% 802 16.4% 881 4.410 5.498 24.7%
Nature 55,380 78.6% 3,925 5.6% 5,067 7.2% 6,047 8.6% 1,784 33.243 40.140 20.7%
Science 45,708 73.0% 4,886 7.8% 5,657 9.0% 6,340 10.1% 1,721 29.398 37.210 26.6%
Journal
N.
Citable
Items
JCR
Impact
Factor
Symmetric
Impact
Factor
%
Increase

12. Inclusion of journal self-citations
Percentage of journal self-citations, by discipline, for
citations received in 2016 by papers published in
2014-2015
0% 5% 10% 15% 20% 25%
All disciplines
Biomedical Research
Clinical Medicine
Chemistry
Mathematics
Psychology
Biology
Health
Social Sciences
Engineering and Technology
Physics
Earth and Space
Professional Fields
Humanities
Arts
Percentage ofjournal self-citations

13. Skewness of distributions
Distribution of
citations received
by articles and
reviews, for four
journals from the
field of
biochemistry and
molecular biology,
2014-2015
papers and 2016
citations
0
5
10
15
20
25
30
35
40
0 10 20 30 40 50 60 70 80 90 100+
Numberofpapers
Number of citations
Cell (JIF = 30.410)
0
5
10
15
20
25
0 10 20 30 40 50 60 70 80 90 100+
Numberofpapers
Number of citations
Nat. Chem. Biol. (JIF = 15.066)
0
5
10
15
20
25
30
35
40
45
0 10 20 30 40 50 60 70 80 90 100+
Numberofpapers
Number of citations
PLOS Biol. (JIF = 9.797)
0
20
40
60
80
100
120
140
160
0 10 20 30 40 50 60 70 80 90 100+
Numberofpapers
Number of citations
FASEB J. (JIF = 5.498)

14. Skewness of distributions (2)
Distribution of the number of journals, by proportion of papers that obtained
the JIF value, 2014-2015 papers and 2016 citations
0
100
200
300
400
500
600
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Numberofjournals
Proportion ofpapers thatobtained the JIF value

15. Length of citation window
Number of citations (left panel) and cumulative number of
citations (right panel), by year following publication for papers
published in 1985.
0
0.5
1
1.5
2
2.5
3
3.5
0 5 10 15 20 25 30
Citations
Year following publication
Number of citations
0
5
10
15
20
25
30
35
40
45
0 5 10 15 20 25 30
Citations
Year following publication
Cumulative citations
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0 5
Citations
Yea
C
0
5
10
15
20
25
30
35
40
45
0 5 10 15 20 25 30
Citations
Year following publication
Cumulative citations
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0 5 10 15 20 25 30
Citations
Year following publication
Cumulative % of citations
Biomedical
Research
Psychology
Physics
Social
Sciences

16. Disciplinary comparison
Mean and maximum JIF of journals, by discipline, 2014-2015 papers and
2016 citations
Discipline
Mean
JCR JIF
Maximum
JCR JIF
Mean
N. Ref.
M
Biology 1.683 22.81 48.99
Biomedical Research 3.526 46.6 48.94
Chemistry 2.768 47.93 46.37
Clinical Medicine 2.976 187.04 41.94
Earth and Space 2.173 30.73 53.71
Engineering and Technology 1.989 39.74 36.35
Health 1.647 17.69 39.08
Mathematics 1.017 9.44 26.56
Physics 2.699 37.85 36.57
Professional Fields 1.565 11.12 53.51
Psychology 2.050 19.95 54.56
Social Sciences 1.199 6.66 49.09

17. Disciplinary comparison (2)
Mean JIF values as a function of A) Number of references and B) Age of
cited references
R² = 0.523
0
1
2
3
4
5
6
0 10 20 30 40 50 60
MeanImpactFactor
Number of references
JIF and
cited references
R² = 0.4721
0
1
2
3
4
5
6
0 10 20 30 40
MeanImpactFactor
Age of references
JIF and
age of references
A B

18. Declining predictive power
Correlation between the JIF and the citations received by papers, 2 year
following publication
y = 0.0011x - 1.9418
r2 = 0.4666
y = 0.003x - 5.7759
r2 = 0.579
y = -0.0049x + 9.9863
r2 = 0.4788
0,00
0,05
0,10
0,15
0,20
0,25
0,30
0,35
1900 1920 1940 1960 1980 2000 2020
Annualcoefficientofdetermination(r2)

19. Declining predictive power
Declining share of top 1% most cited papers for elite journals

20. Impact Factor inflation

21. Impact Factor inflation

22. Impact Factor inflation

23. Impact Factor inflation
Impact Factor by journal as a function of rank, for years 1997, 2007, and
2016
0.001
0.01
0.1
1
10
100
1000
1 10 100 1000 10000 100000
ImpactFactor(logscale)
Rank of journal (log scale)
2016
2007
1997

24. Post JIF indicators
• Five-year Impact Factor (Clarivate)
• Eigenfactor (Clarivate) and SCImago Journal
Rank (Scopus)
• Source Normalized Impact per Paper—SNIP
(Scopus)
• CiteScore
• None of those indicators solve the main misuse
of the JIF: the application of a journal-level
indicator to papers and researchers

25. Systemic effects
• Orients researchers’ publication strategies
o and research topics
o Weakens non-indexed journals
• Impact Factor engineering
o Optimizing citation window
o Numerator/denominator and front matter
o Coercitive citations
• Cash per publication policies
o Science/Nature article  20 times mean academic salary (China)
o Disciplinary inequalities…
• Multiplication of citation indicators
o Anyone can find an indicator where one looks better
o Knock off indicators

26. Knock off Impact Factors

27. Knock off Impact Factors

28. Do we need the JIF?
What problem does the JIF solve?
Information filtering and ranking. Are there other mechanisms for this?
What are the effects of the JIF?
Journal indicators reify the role of journals—especially anglo-american.
Is such hierarchy necessary?
The community must ask:
Is the use of the Journal Impact Factor good for science?
… AND EVEN MORE IMPORTANTLY:
Is research evaluation good for science?

29. Vincent Larivière
Cassidy R. Sugimoto
vincent.lariviere@umontreal.ca
@lariviev
crc.ebsi.umontreal.ca
Obrigado!

30. Garfield’s 1972 paper
• Citations received during the last
quarter of 1969 (Multiplied by four
to obtain a year estimate)
o First calculation of an Impact Factor (except
M & G)
• Top 152 journals account for 50%
of all references
• “It is apparent, even from the
makeup of this partial listing, that
a good multidisciplinary journal
collection need contain no more
than a few hundred titles.”
• “This analysis gives good reason
for concern about any increase in
the number of scientific and
technical journals.”
• Citation Analysis as a Tool in Journal Evaluation. Journals can be ranked by
frequency and impact of citations for science policy studies.