SlideShare a Scribd company logo
1 of 13
Download to read offline
Use of the journal impact factor for
assessing individual articles need not
be wrong
CWTS, Leiden University
6 September 2017
L. Waltman & V.A. Traag
Impact Factor Debate
• DORA (2013)
• Seglen (1997):
“the most cited half of the articles (in a journal) are cited, on
average, 10 times as often as the least cited half. Assigning the
same score (the JIF) to all articles masks this tremendous
difference–which is the exact opposite of what an evaluation is
meant to achieve”
• Garfield (2006):
“Typically, when the author’s work is examined, the impact
factors of the journals involved are substituted for the actual
citation count. Thus, the JIF is used to estimate the expected
count of individual papers, which is rather dubious considering
the known skewness observed for most journals”
2/13
Impact Factor Debate
• Abramo et al. (2010):
“Citations observed at a moment too close to the date of
publication will not necessarily offer a proxy of quality that is
preferable to impact factor”
• Levitt & Thelwall (2011):
“Particularly for very recently published articles, an indicator
based on the average of the standard indicator of citation and
the IF of the journal . . . could form the basis of a useful indicator
for peer review panels”
3/13
Skewness
• Skewness in citations main statistical argument against IF.
• Assumption that skewness in citations represent ‘real’ differences
in value.
• However, skewness may also result from citations being a ‘noisy
signal’ for value.
4/13
Scenario 1 (Citations accurate)
Citations in journals are clearly skewed. But why?
Scenario 1 (citations accurate)
1 Citations reasonably accurate proxy of the value of an article.
2 Value of articles in a journal is skewed distributed.
3 Skewness of journal citation distributions results mainly from 2.
Scenario 2 (citations noisy)
1 Citations are a weak proxy of the value of an article.
2 Value of articles in a journal is distributed homogeneous.
3 Skewness of journal citation distributions results mainly from 1.
5/13
Scenario 1 (Citations accurate) example
Journal A
Citations
Low High Total
Value
Low 18 2 20
High 8 72 80
Total 26 74 100
Journal B
Citations
Low High Total
Value
Low 72 8 80
High 8 18 20
Total 74 26 100
Select papers: how many have a high value?
1 Select highly cited publications: 72+18
74+26 = 90
100 = 0.9.
2 Select publications from high impact journal: 8+72
100 = 0.8.
Citations more accurate.
6/13
Scenario 2 (Citations noisy)
Citations in journal are clearly skewed. But why?
Scenario 1 (citations accurate)
1 Citations reasonably accurate proxy of the value of an article.
2 Value of articles in a journal is skewed distributed.
3 Skewness of journal citation distributions results mainly from 2.
Scenario 2 (citations noisy)
1 Citations are a weak proxy of the value of an article.
2 Value of articles in a journal is distributed homogeneous.
3 Skewness of journal citation distributions results mainly from 1.
7/13
Scenario 2 (Citations noisy) example
Journal A
Citations
Low High Total
Value
Low 14 6 20
High 24 56 80
Total 38 62 100
Journal B
Citations
Low High Total
Value
Low 56 24 80
High 6 14 20
Total 62 38 100
Select papers: how many have a high value?
1 Select highly cited publications: 56+14
62+38 = 70
100 = 0.7.
2 Select publications from high impact journal: 24+56
100 = 0.8.
Journal more accurate.
8/13
Simulation
Procedure
1 Create articles with some value vi ∼ LogN(σ2
v ).
2 Articles are peer reviewed with some error.
Estimated value eik = vi ik .
Error is ik ∼ LogN(σ2
r ).
3 Articles are selected for a journal based on peer review.
All articles submitted to highest journal.
Only top x articles accepted, others rejected.
If rejected, submit to next highest journal, and so on . . . .
4 Articles attract citations according to their value.
Citations ci = vi i where i ∼ LogN(σ2
c ).
Citations distributed as ci ∼ LogN(σ2
v + σ2
c ).
Consistent with empirical observation that ci ∼ LogN(1.3).
IF is average of all citations to articles published in journal.
9/13
Article selection
Two possible ways to try to select high value (top 10%) articles.
1 Select top 10% most highly cited articles.
2 Select top 10% highest impact factor articles.
Accuracy
Proportion of articles that fall in the top 10% highest value.
Parameters: 20 journals, 100 articles/journal, 1000 simulations.
10/13
Results
11/13
Conclusions
• Skewness alone is no argument for rejecting the IF.
• If citations are reasonably accurate while peer review is noisy,
then IF may be less accurate than citations.
• If citations are noisy while peer review is reasonably accurate,
then IF may be more accurate than citations.
• In practice, parameters used in this model unknown, so requires
further (empirical) analysis.
• There may be good non-statistical arguments against the IF
(e.g. goal displacement).
Statistics is hard. Make assumptions and arguments explicit.
12/13
Thank you!
Questions?
arχiv:1703.02334
v.a.traag@cwts.leidenuniv.nl @vtraag
13/13

More Related Content

Similar to Use of the journal impact factor for assessing individual articles need not be wrong

Wenner gren citation-non_sense_reedijk
Wenner gren citation-non_sense_reedijkWenner gren citation-non_sense_reedijk
Wenner gren citation-non_sense_reedijkJan Reedijk
 
Publication impact factors and your scientific career
Publication impact factors and your scientific careerPublication impact factors and your scientific career
Publication impact factors and your scientific careerBarbara Janssens
 
Compton-Week12-Annotated Bibliographies.pptx
Compton-Week12-Annotated Bibliographies.pptxCompton-Week12-Annotated Bibliographies.pptx
Compton-Week12-Annotated Bibliographies.pptxAmberPierdinock
 
Scholarly impact metrics traditions
Scholarly impact metrics traditionsScholarly impact metrics traditions
Scholarly impact metrics traditionsntunmg
 
Wang - Bias againt novelty in science
Wang - Bias againt novelty in scienceWang - Bias againt novelty in science
Wang - Bias againt novelty in scienceinnovationoecd
 
Lawrence-Week11-Annotated Bibliography.pptx
Lawrence-Week11-Annotated Bibliography.pptxLawrence-Week11-Annotated Bibliography.pptx
Lawrence-Week11-Annotated Bibliography.pptxAmberPierdinock
 
Using Impact Factors in Research & Publication
Using Impact Factors in Research & PublicationUsing Impact Factors in Research & Publication
Using Impact Factors in Research & PublicationRebecca Kate Miller
 
Writing a Research Paper
Writing a Research PaperWriting a Research Paper
Writing a Research PaperSusha Das
 
Scopus: compare journals using journal metrics from UCT Libraries
Scopus: compare journals using journal metrics from UCT LibrariesScopus: compare journals using journal metrics from UCT Libraries
Scopus: compare journals using journal metrics from UCT LibrariesUCT
 

Similar to Use of the journal impact factor for assessing individual articles need not be wrong (9)

Wenner gren citation-non_sense_reedijk
Wenner gren citation-non_sense_reedijkWenner gren citation-non_sense_reedijk
Wenner gren citation-non_sense_reedijk
 
Publication impact factors and your scientific career
Publication impact factors and your scientific careerPublication impact factors and your scientific career
Publication impact factors and your scientific career
 
Compton-Week12-Annotated Bibliographies.pptx
Compton-Week12-Annotated Bibliographies.pptxCompton-Week12-Annotated Bibliographies.pptx
Compton-Week12-Annotated Bibliographies.pptx
 
Scholarly impact metrics traditions
Scholarly impact metrics traditionsScholarly impact metrics traditions
Scholarly impact metrics traditions
 
Wang - Bias againt novelty in science
Wang - Bias againt novelty in scienceWang - Bias againt novelty in science
Wang - Bias againt novelty in science
 
Lawrence-Week11-Annotated Bibliography.pptx
Lawrence-Week11-Annotated Bibliography.pptxLawrence-Week11-Annotated Bibliography.pptx
Lawrence-Week11-Annotated Bibliography.pptx
 
Using Impact Factors in Research & Publication
Using Impact Factors in Research & PublicationUsing Impact Factors in Research & Publication
Using Impact Factors in Research & Publication
 
Writing a Research Paper
Writing a Research PaperWriting a Research Paper
Writing a Research Paper
 
Scopus: compare journals using journal metrics from UCT Libraries
Scopus: compare journals using journal metrics from UCT LibrariesScopus: compare journals using journal metrics from UCT Libraries
Scopus: compare journals using journal metrics from UCT Libraries
 

More from Vincent Traag

Peer review uncertainty at the institutional level
Peer review uncertainty at the institutional levelPeer review uncertainty at the institutional level
Peer review uncertainty at the institutional levelVincent Traag
 
Replacing peer review by metrics in the UK REF?
Replacing peer review by metrics in the UK REF?Replacing peer review by metrics in the UK REF?
Replacing peer review by metrics in the UK REF?Vincent Traag
 
Uncovering important intermediate publications
Uncovering important intermediate publicationsUncovering important intermediate publications
Uncovering important intermediate publicationsVincent Traag
 
Complex contagion of campaign donations
Complex contagion of campaign donationsComplex contagion of campaign donations
Complex contagion of campaign donationsVincent Traag
 
Polarization and consensus in citation networks
Polarization and consensus in citation networksPolarization and consensus in citation networks
Polarization and consensus in citation networksVincent Traag
 
Community structure in complex networks
Community structure in complex networksCommunity structure in complex networks
Community structure in complex networksVincent Traag
 
Introduction to complex networks
Introduction to complex networksIntroduction to complex networks
Introduction to complex networksVincent Traag
 
Public thesis defence: groups and reputation in social networks
Public thesis defence: groups and reputation in social networksPublic thesis defence: groups and reputation in social networks
Public thesis defence: groups and reputation in social networksVincent Traag
 
Structure of media attention
Structure of media attentionStructure of media attention
Structure of media attentionVincent Traag
 
Dynamics of Media Attention
Dynamics of Media AttentionDynamics of Media Attention
Dynamics of Media AttentionVincent Traag
 
Dynamical Models Explaining Social Balance
Dynamical Models Explaining Social BalanceDynamical Models Explaining Social Balance
Dynamical Models Explaining Social BalanceVincent Traag
 
Significant scales in community structure
Significant scales in community structureSignificant scales in community structure
Significant scales in community structureVincent Traag
 
Reconstructing Third World Elite Rotation Events from Newspapers
Reconstructing Third World Elite Rotation Events from NewspapersReconstructing Third World Elite Rotation Events from Newspapers
Reconstructing Third World Elite Rotation Events from NewspapersVincent Traag
 
Reputation Dynamics Through Gossiping
Reputation Dynamics Through GossipingReputation Dynamics Through Gossiping
Reputation Dynamics Through GossipingVincent Traag
 
Limits of community detection
Limits of community detectionLimits of community detection
Limits of community detectionVincent Traag
 
Cooperation, Reputation & Gossiping
Cooperation, Reputation & GossipingCooperation, Reputation & Gossiping
Cooperation, Reputation & GossipingVincent Traag
 
Resolution-free community detection
Resolution-free community detectionResolution-free community detection
Resolution-free community detectionVincent Traag
 
Cooperation, Reputation & Gossiping
Cooperation, Reputation & GossipingCooperation, Reputation & Gossiping
Cooperation, Reputation & GossipingVincent Traag
 
Exponential Ranking: Taking into account negative links.
Exponential Ranking: Taking into account negative links.Exponential Ranking: Taking into account negative links.
Exponential Ranking: Taking into account negative links.Vincent Traag
 
Social Event Detection
Social Event DetectionSocial Event Detection
Social Event DetectionVincent Traag
 

More from Vincent Traag (20)

Peer review uncertainty at the institutional level
Peer review uncertainty at the institutional levelPeer review uncertainty at the institutional level
Peer review uncertainty at the institutional level
 
Replacing peer review by metrics in the UK REF?
Replacing peer review by metrics in the UK REF?Replacing peer review by metrics in the UK REF?
Replacing peer review by metrics in the UK REF?
 
Uncovering important intermediate publications
Uncovering important intermediate publicationsUncovering important intermediate publications
Uncovering important intermediate publications
 
Complex contagion of campaign donations
Complex contagion of campaign donationsComplex contagion of campaign donations
Complex contagion of campaign donations
 
Polarization and consensus in citation networks
Polarization and consensus in citation networksPolarization and consensus in citation networks
Polarization and consensus in citation networks
 
Community structure in complex networks
Community structure in complex networksCommunity structure in complex networks
Community structure in complex networks
 
Introduction to complex networks
Introduction to complex networksIntroduction to complex networks
Introduction to complex networks
 
Public thesis defence: groups and reputation in social networks
Public thesis defence: groups and reputation in social networksPublic thesis defence: groups and reputation in social networks
Public thesis defence: groups and reputation in social networks
 
Structure of media attention
Structure of media attentionStructure of media attention
Structure of media attention
 
Dynamics of Media Attention
Dynamics of Media AttentionDynamics of Media Attention
Dynamics of Media Attention
 
Dynamical Models Explaining Social Balance
Dynamical Models Explaining Social BalanceDynamical Models Explaining Social Balance
Dynamical Models Explaining Social Balance
 
Significant scales in community structure
Significant scales in community structureSignificant scales in community structure
Significant scales in community structure
 
Reconstructing Third World Elite Rotation Events from Newspapers
Reconstructing Third World Elite Rotation Events from NewspapersReconstructing Third World Elite Rotation Events from Newspapers
Reconstructing Third World Elite Rotation Events from Newspapers
 
Reputation Dynamics Through Gossiping
Reputation Dynamics Through GossipingReputation Dynamics Through Gossiping
Reputation Dynamics Through Gossiping
 
Limits of community detection
Limits of community detectionLimits of community detection
Limits of community detection
 
Cooperation, Reputation & Gossiping
Cooperation, Reputation & GossipingCooperation, Reputation & Gossiping
Cooperation, Reputation & Gossiping
 
Resolution-free community detection
Resolution-free community detectionResolution-free community detection
Resolution-free community detection
 
Cooperation, Reputation & Gossiping
Cooperation, Reputation & GossipingCooperation, Reputation & Gossiping
Cooperation, Reputation & Gossiping
 
Exponential Ranking: Taking into account negative links.
Exponential Ranking: Taking into account negative links.Exponential Ranking: Taking into account negative links.
Exponential Ranking: Taking into account negative links.
 
Social Event Detection
Social Event DetectionSocial Event Detection
Social Event Detection
 

Recently uploaded

Efficient spin-up of Earth System Models usingsequence acceleration
Efficient spin-up of Earth System Models usingsequence accelerationEfficient spin-up of Earth System Models usingsequence acceleration
Efficient spin-up of Earth System Models usingsequence accelerationSérgio Sacani
 
SaffronCrocusGenomicsThessalonikiOnlineMay2024TalkOnline.pptx
SaffronCrocusGenomicsThessalonikiOnlineMay2024TalkOnline.pptxSaffronCrocusGenomicsThessalonikiOnlineMay2024TalkOnline.pptx
SaffronCrocusGenomicsThessalonikiOnlineMay2024TalkOnline.pptxPat (JS) Heslop-Harrison
 
A Scientific PowerPoint on Albert Einstein
A Scientific PowerPoint on Albert EinsteinA Scientific PowerPoint on Albert Einstein
A Scientific PowerPoint on Albert Einsteinxgamestudios8
 
Classification of Kerogen, Perspective on palynofacies in depositional envi...
Classification of Kerogen,  Perspective on palynofacies in depositional  envi...Classification of Kerogen,  Perspective on palynofacies in depositional  envi...
Classification of Kerogen, Perspective on palynofacies in depositional envi...Sangram Sahoo
 
Technical english Technical english.pptx
Technical english Technical english.pptxTechnical english Technical english.pptx
Technical english Technical english.pptxyoussefboujtat3
 
Soil and Water Conservation Engineering (SWCE) is a specialized field of stud...
Soil and Water Conservation Engineering (SWCE) is a specialized field of stud...Soil and Water Conservation Engineering (SWCE) is a specialized field of stud...
Soil and Water Conservation Engineering (SWCE) is a specialized field of stud...yogeshlabana357357
 
GBSN - Biochemistry (Unit 3) Metabolism
GBSN - Biochemistry (Unit 3) MetabolismGBSN - Biochemistry (Unit 3) Metabolism
GBSN - Biochemistry (Unit 3) MetabolismAreesha Ahmad
 
Film Coated Tablet and Film Coating raw materials.pdf
Film Coated Tablet and Film Coating raw materials.pdfFilm Coated Tablet and Film Coating raw materials.pdf
Film Coated Tablet and Film Coating raw materials.pdfPharmatech-rx
 
Heat Units in plant physiology and the importance of Growing Degree days
Heat Units in plant physiology and the importance of Growing Degree daysHeat Units in plant physiology and the importance of Growing Degree days
Heat Units in plant physiology and the importance of Growing Degree daysBrahmesh Reddy B R
 
Taphonomy and Quality of the Fossil Record
Taphonomy and Quality of the  Fossil RecordTaphonomy and Quality of the  Fossil Record
Taphonomy and Quality of the Fossil RecordSangram Sahoo
 
PARENTAL CARE IN FISHES.pptx for 5th sem
PARENTAL CARE IN FISHES.pptx for 5th semPARENTAL CARE IN FISHES.pptx for 5th sem
PARENTAL CARE IN FISHES.pptx for 5th semborkhotudu123
 
TEST BANK for Organic Chemistry 6th Edition.pdf
TEST BANK for Organic Chemistry 6th Edition.pdfTEST BANK for Organic Chemistry 6th Edition.pdf
TEST BANK for Organic Chemistry 6th Edition.pdfmarcuskenyatta275
 
ANITINUTRITION FACTOR GYLCOSIDES SAPONINS CYANODENS
ANITINUTRITION FACTOR GYLCOSIDES SAPONINS CYANODENSANITINUTRITION FACTOR GYLCOSIDES SAPONINS CYANODENS
ANITINUTRITION FACTOR GYLCOSIDES SAPONINS CYANODENSDr. TATHAGAT KHOBRAGADE
 
NuGOweek 2024 programme final FLYER short.pdf
NuGOweek 2024 programme final FLYER short.pdfNuGOweek 2024 programme final FLYER short.pdf
NuGOweek 2024 programme final FLYER short.pdfpablovgd
 
FORENSIC CHEMISTRY ARSON INVESTIGATION.pdf
FORENSIC CHEMISTRY ARSON INVESTIGATION.pdfFORENSIC CHEMISTRY ARSON INVESTIGATION.pdf
FORENSIC CHEMISTRY ARSON INVESTIGATION.pdfSuchita Rawat
 
Electricity and Circuits for Grade 9 students
Electricity and Circuits for Grade 9 studentsElectricity and Circuits for Grade 9 students
Electricity and Circuits for Grade 9 studentslevieagacer
 
GBSN - Biochemistry (Unit 8) Enzymology
GBSN - Biochemistry (Unit 8) EnzymologyGBSN - Biochemistry (Unit 8) Enzymology
GBSN - Biochemistry (Unit 8) EnzymologyAreesha Ahmad
 
PHOTOSYNTHETIC BACTERIA (OXYGENIC AND ANOXYGENIC)
PHOTOSYNTHETIC BACTERIA  (OXYGENIC AND ANOXYGENIC)PHOTOSYNTHETIC BACTERIA  (OXYGENIC AND ANOXYGENIC)
PHOTOSYNTHETIC BACTERIA (OXYGENIC AND ANOXYGENIC)kushbuR
 
MSCII_ FCT UNIT 5 TOXICOLOGY.pdf
MSCII_              FCT UNIT 5 TOXICOLOGY.pdfMSCII_              FCT UNIT 5 TOXICOLOGY.pdf
MSCII_ FCT UNIT 5 TOXICOLOGY.pdfSuchita Rawat
 
MSC IV_Forensic medicine - Mechanical injuries.pdf
MSC IV_Forensic medicine - Mechanical injuries.pdfMSC IV_Forensic medicine - Mechanical injuries.pdf
MSC IV_Forensic medicine - Mechanical injuries.pdfSuchita Rawat
 

Recently uploaded (20)

Efficient spin-up of Earth System Models usingsequence acceleration
Efficient spin-up of Earth System Models usingsequence accelerationEfficient spin-up of Earth System Models usingsequence acceleration
Efficient spin-up of Earth System Models usingsequence acceleration
 
SaffronCrocusGenomicsThessalonikiOnlineMay2024TalkOnline.pptx
SaffronCrocusGenomicsThessalonikiOnlineMay2024TalkOnline.pptxSaffronCrocusGenomicsThessalonikiOnlineMay2024TalkOnline.pptx
SaffronCrocusGenomicsThessalonikiOnlineMay2024TalkOnline.pptx
 
A Scientific PowerPoint on Albert Einstein
A Scientific PowerPoint on Albert EinsteinA Scientific PowerPoint on Albert Einstein
A Scientific PowerPoint on Albert Einstein
 
Classification of Kerogen, Perspective on palynofacies in depositional envi...
Classification of Kerogen,  Perspective on palynofacies in depositional  envi...Classification of Kerogen,  Perspective on palynofacies in depositional  envi...
Classification of Kerogen, Perspective on palynofacies in depositional envi...
 
Technical english Technical english.pptx
Technical english Technical english.pptxTechnical english Technical english.pptx
Technical english Technical english.pptx
 
Soil and Water Conservation Engineering (SWCE) is a specialized field of stud...
Soil and Water Conservation Engineering (SWCE) is a specialized field of stud...Soil and Water Conservation Engineering (SWCE) is a specialized field of stud...
Soil and Water Conservation Engineering (SWCE) is a specialized field of stud...
 
GBSN - Biochemistry (Unit 3) Metabolism
GBSN - Biochemistry (Unit 3) MetabolismGBSN - Biochemistry (Unit 3) Metabolism
GBSN - Biochemistry (Unit 3) Metabolism
 
Film Coated Tablet and Film Coating raw materials.pdf
Film Coated Tablet and Film Coating raw materials.pdfFilm Coated Tablet and Film Coating raw materials.pdf
Film Coated Tablet and Film Coating raw materials.pdf
 
Heat Units in plant physiology and the importance of Growing Degree days
Heat Units in plant physiology and the importance of Growing Degree daysHeat Units in plant physiology and the importance of Growing Degree days
Heat Units in plant physiology and the importance of Growing Degree days
 
Taphonomy and Quality of the Fossil Record
Taphonomy and Quality of the  Fossil RecordTaphonomy and Quality of the  Fossil Record
Taphonomy and Quality of the Fossil Record
 
PARENTAL CARE IN FISHES.pptx for 5th sem
PARENTAL CARE IN FISHES.pptx for 5th semPARENTAL CARE IN FISHES.pptx for 5th sem
PARENTAL CARE IN FISHES.pptx for 5th sem
 
TEST BANK for Organic Chemistry 6th Edition.pdf
TEST BANK for Organic Chemistry 6th Edition.pdfTEST BANK for Organic Chemistry 6th Edition.pdf
TEST BANK for Organic Chemistry 6th Edition.pdf
 
ANITINUTRITION FACTOR GYLCOSIDES SAPONINS CYANODENS
ANITINUTRITION FACTOR GYLCOSIDES SAPONINS CYANODENSANITINUTRITION FACTOR GYLCOSIDES SAPONINS CYANODENS
ANITINUTRITION FACTOR GYLCOSIDES SAPONINS CYANODENS
 
NuGOweek 2024 programme final FLYER short.pdf
NuGOweek 2024 programme final FLYER short.pdfNuGOweek 2024 programme final FLYER short.pdf
NuGOweek 2024 programme final FLYER short.pdf
 
FORENSIC CHEMISTRY ARSON INVESTIGATION.pdf
FORENSIC CHEMISTRY ARSON INVESTIGATION.pdfFORENSIC CHEMISTRY ARSON INVESTIGATION.pdf
FORENSIC CHEMISTRY ARSON INVESTIGATION.pdf
 
Electricity and Circuits for Grade 9 students
Electricity and Circuits for Grade 9 studentsElectricity and Circuits for Grade 9 students
Electricity and Circuits for Grade 9 students
 
GBSN - Biochemistry (Unit 8) Enzymology
GBSN - Biochemistry (Unit 8) EnzymologyGBSN - Biochemistry (Unit 8) Enzymology
GBSN - Biochemistry (Unit 8) Enzymology
 
PHOTOSYNTHETIC BACTERIA (OXYGENIC AND ANOXYGENIC)
PHOTOSYNTHETIC BACTERIA  (OXYGENIC AND ANOXYGENIC)PHOTOSYNTHETIC BACTERIA  (OXYGENIC AND ANOXYGENIC)
PHOTOSYNTHETIC BACTERIA (OXYGENIC AND ANOXYGENIC)
 
MSCII_ FCT UNIT 5 TOXICOLOGY.pdf
MSCII_              FCT UNIT 5 TOXICOLOGY.pdfMSCII_              FCT UNIT 5 TOXICOLOGY.pdf
MSCII_ FCT UNIT 5 TOXICOLOGY.pdf
 
MSC IV_Forensic medicine - Mechanical injuries.pdf
MSC IV_Forensic medicine - Mechanical injuries.pdfMSC IV_Forensic medicine - Mechanical injuries.pdf
MSC IV_Forensic medicine - Mechanical injuries.pdf
 

Use of the journal impact factor for assessing individual articles need not be wrong

  • 1. Use of the journal impact factor for assessing individual articles need not be wrong CWTS, Leiden University 6 September 2017 L. Waltman & V.A. Traag
  • 2. Impact Factor Debate • DORA (2013) • Seglen (1997): “the most cited half of the articles (in a journal) are cited, on average, 10 times as often as the least cited half. Assigning the same score (the JIF) to all articles masks this tremendous difference–which is the exact opposite of what an evaluation is meant to achieve” • Garfield (2006): “Typically, when the author’s work is examined, the impact factors of the journals involved are substituted for the actual citation count. Thus, the JIF is used to estimate the expected count of individual papers, which is rather dubious considering the known skewness observed for most journals” 2/13
  • 3. Impact Factor Debate • Abramo et al. (2010): “Citations observed at a moment too close to the date of publication will not necessarily offer a proxy of quality that is preferable to impact factor” • Levitt & Thelwall (2011): “Particularly for very recently published articles, an indicator based on the average of the standard indicator of citation and the IF of the journal . . . could form the basis of a useful indicator for peer review panels” 3/13
  • 4. Skewness • Skewness in citations main statistical argument against IF. • Assumption that skewness in citations represent ‘real’ differences in value. • However, skewness may also result from citations being a ‘noisy signal’ for value. 4/13
  • 5. Scenario 1 (Citations accurate) Citations in journals are clearly skewed. But why? Scenario 1 (citations accurate) 1 Citations reasonably accurate proxy of the value of an article. 2 Value of articles in a journal is skewed distributed. 3 Skewness of journal citation distributions results mainly from 2. Scenario 2 (citations noisy) 1 Citations are a weak proxy of the value of an article. 2 Value of articles in a journal is distributed homogeneous. 3 Skewness of journal citation distributions results mainly from 1. 5/13
  • 6. Scenario 1 (Citations accurate) example Journal A Citations Low High Total Value Low 18 2 20 High 8 72 80 Total 26 74 100 Journal B Citations Low High Total Value Low 72 8 80 High 8 18 20 Total 74 26 100 Select papers: how many have a high value? 1 Select highly cited publications: 72+18 74+26 = 90 100 = 0.9. 2 Select publications from high impact journal: 8+72 100 = 0.8. Citations more accurate. 6/13
  • 7. Scenario 2 (Citations noisy) Citations in journal are clearly skewed. But why? Scenario 1 (citations accurate) 1 Citations reasonably accurate proxy of the value of an article. 2 Value of articles in a journal is skewed distributed. 3 Skewness of journal citation distributions results mainly from 2. Scenario 2 (citations noisy) 1 Citations are a weak proxy of the value of an article. 2 Value of articles in a journal is distributed homogeneous. 3 Skewness of journal citation distributions results mainly from 1. 7/13
  • 8. Scenario 2 (Citations noisy) example Journal A Citations Low High Total Value Low 14 6 20 High 24 56 80 Total 38 62 100 Journal B Citations Low High Total Value Low 56 24 80 High 6 14 20 Total 62 38 100 Select papers: how many have a high value? 1 Select highly cited publications: 56+14 62+38 = 70 100 = 0.7. 2 Select publications from high impact journal: 24+56 100 = 0.8. Journal more accurate. 8/13
  • 9. Simulation Procedure 1 Create articles with some value vi ∼ LogN(σ2 v ). 2 Articles are peer reviewed with some error. Estimated value eik = vi ik . Error is ik ∼ LogN(σ2 r ). 3 Articles are selected for a journal based on peer review. All articles submitted to highest journal. Only top x articles accepted, others rejected. If rejected, submit to next highest journal, and so on . . . . 4 Articles attract citations according to their value. Citations ci = vi i where i ∼ LogN(σ2 c ). Citations distributed as ci ∼ LogN(σ2 v + σ2 c ). Consistent with empirical observation that ci ∼ LogN(1.3). IF is average of all citations to articles published in journal. 9/13
  • 10. Article selection Two possible ways to try to select high value (top 10%) articles. 1 Select top 10% most highly cited articles. 2 Select top 10% highest impact factor articles. Accuracy Proportion of articles that fall in the top 10% highest value. Parameters: 20 journals, 100 articles/journal, 1000 simulations. 10/13
  • 12. Conclusions • Skewness alone is no argument for rejecting the IF. • If citations are reasonably accurate while peer review is noisy, then IF may be less accurate than citations. • If citations are noisy while peer review is reasonably accurate, then IF may be more accurate than citations. • In practice, parameters used in this model unknown, so requires further (empirical) analysis. • There may be good non-statistical arguments against the IF (e.g. goal displacement). Statistics is hard. Make assumptions and arguments explicit. 12/13