• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Berlin10 beyond the impact factor
 

Berlin10 beyond the impact factor

on

  • 662 views

Berlin10SA presentation nov 7, 2012

Berlin10SA presentation nov 7, 2012

Statistics

Views

Total Views
662
Views on SlideShare
655
Embed Views
7

Actions

Likes
4
Downloads
6
Comments
0

3 Embeds 7

https://twitter.com 4
http://www.linkedin.com 2
https://www.linkedin.com 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

Berlin10 beyond the impact factor Berlin10 beyond the impact factor Presentation Transcript

  • Beyond the Impact Factor: Why the Thomson-Reuters impact factor has to be replaced Tom Olijhoek SURF NL Acknowledgements Paul Wouters Leiden University Jelle Wicherts Tilburg University Björn Brembs RegensburgThis work is licensed under a CreativeCommons Attribution 3.0 Unported License
  • You are free to: Copy, share, adapt, or re-mix; Blog, live-blog, or post video of; This presentation. Provided that: You attribute the work to its author and respect the rights and licenses associated with its components. Slide Concept by Cameron Neylon, who has waived all copyright and related or neighbouring rights. This slide only ccZero. Social Media Icons adapted with permission from originals by Christopher Ross. Original images are available under GPL at;Beyond the impact factorTom Olijhoek SURF-NL
  • The Thomson Reuters impact factor is used to assess the quality of a journal The TR impact factor CORRELATES VERY WELL with the perceived quality of a journal SO WHAT IS WRONG WITH IT?Beyond the impact factorTom Olijhoek SURF-NL
  • MANY THINGS ARE WRONG WITH THE IMPACT FACTOR  Fortunately Open Access enables other methods for Quality Assessment  But…..the Impact Factor is an obstacle for Open Access  To get Open Access we need to get rid of the Impact Factor  For that we need an attitude change  For that we need commitment of scientist communities  In all parts of the worldBeyond the impact factorTom Olijhoek SURF-NL
  •  Especially in the developing world  Participation of scientists in the developing word will make the difference  To participate on an equal basis in the making of science  To profit on an equal basis of the fruits of science  Science is the motor of economic development  OPEN ACCESS IS THE KEY TO EDUCATION, INNOVATION, ECONOMIC DEVELOPMENT AND PROSPERITY EVERYWHERE ALSO IN THE DEVELOPING WORLDBeyond the impact factorTom Olijhoek SURF-NL
  • Lars Bjørnshauge Quote from talk at the PKP conference September 2011, Berlin: The push for researchers from [developing countries] and continents to publish in high impact factor journals has decisive influence on the subject of their research and much more so is a big obstacle for open access publishing.Beyond the impact factorTom Olijhoek SURF-NL
  • The Impact Factor Introduced in 1960‟s by Eugene Garfield: ISI citations articles 2008 2006 and 2007 IF=5 Articles published in 06/07 were cited an average of 5 times in 08.Beyond the impact factorTom Olijhoek SURF-NL
  • The impact factor Nothing wrong with using citations quality criteria for Using average citations for the average article as quality indicator for a That is where things go wrong You can NOT draw conclusions on INDIVIDUAL article qualities based on the AVERAGE quality of ALL ARTICLES in a journalBeyond the impact factorTom Olijhoek SURF-NL
  • Our management discourages us from supporting new open access journals due to their low, or unassigned, impact factor MalariaWorld survey 2012 The USE of the impact factor for ASSESSING SCIENTISTS is obstructing the move towards Open Access Weak correlation of individual article citation rate with journal IF But scientists are judged on NUMBER of publications in HIGH IMPACT JOURNALS Most scientists do not publish in OA journals for one reason: because it could hamper their careersBeyond the impact factorTom Olijhoek SURF-NL
  • Weak correlation of individual article citation rate with journal IF Seglen PO (1997): Why the impact factor of journals should not be used for evaluating research. BMJ 1997;314(7079):497 (15 February)Beyond the impact factorTom Olijhoek SURF-NL
  • PLoS Medicine, IF 2-11 (8.4) (The PLoS Medicine Editors (2006) The Negotiable Impact Factor Game. PLoS Med 3(6): e291. Irreproducible Rossner M, van Epps H, Hill E (2007): Show me the data. The Journal of Cell Biology, Vol. 179, No. 6, 1091-1092 Can be gamed Journals exert pressure to get cited themselves often Scientists „ask“ to be cited Fake authors Björn Brembs http://www.slideshare.net/brembs/limited- access-is-a-symptom-not-the-diseaseBeyond the impact factorTom Olijhoek SURF-NL
  • NEGOTIABLE: number of articles which are cited can be adjusted PLoS Medicine, IF 2-11 (8.4) (The PLoS Medicine Editors (2006) The Impact Factor Game. PLoS Med 3(6): e291. Current Biology IF from 7 to 11 in 2003 Bought by Cell Press (Elsevier) in 2001…Beyond the impact factorTom Olijhoek SURF-NL
  • NEGOTIABLE What is obvious from this equation is that the impact factor depends crucially on which article types Thomson Scientific deems as “citable”—the fewer, the better (i.e., the lower the denominator, the higher the impact factor). PLoS Medicine, IF 2-11 (8.4) (The PLoS Medicine Editors (2006) The Impact Factor Game. PLoS Med 3(6): e291.Beyond the impact factorTom Olijhoek SURF-NL
  • Beyond the impact factorTom Olijhoek SURF-NL
  • Methodological problems with the Impact Factor Correlation of IF with: •Retraction rates: high •Subjective journal rank: very high •Quality of individual articles: low •Citations: low •Expert opinion: low •Methodological standards: low •Replicability: none Björn Brembs and Marcus Munafò http://bit.ly/WNzA1ZBeyond the impact factorTom Olijhoek SURF-NL
  • Two More Problems The Impact Factor is commercially produced The TR-Impact Factor underscores research topics from the South by design 50,000 employees US$600million profit/quarter Thomson family owns 53% €30,000-130,000/year subscription rates €30,000-130,000/year subscription rates Covers ~11,500 journals (Scopus covers ~16,500)Beyond the impact factorTom Olijhoek SURF-NL
  • OPEN ACCESS ENABLES THE DEVELOPMENT OF NEW WAYS OF QUALITY ASSESSMENT While Open Access is hindered by the TR Impact Factor new methods made possible by Open Access can replace it! Measure Impact beyond mere citation analysis Measure Impact beyond scientific impactBeyond the impact factorTom Olijhoek SURF-NL
  • „New‟ Definition of Scientific ImpactBeyond the impact factorTom Olijhoek SURF-NL
  • “When I light my candle from yours, I gain from you without subtracting from you. That’s what sharing knowledge is like”. Peter Suber Open Access Toll Access getting new ideas by fear of losing ideas sharing Collaboration Competition Publish for impact Publish or perish Focus on quality Focus on quantity We shouldn‟t be counting the beans but instead taste them The proof is in the puddingBeyond the impact factorTom Olijhoek SURF-NL
  • We need commitment Open Access is crucial for scientists in the global south conference The Hague 25 oct 2012 as long as scientific output remains behind walls of paid content, no possibility for a dialogue will exist Science is the Open Access opens motor for Science for All economic development Participation of scientists from Research is the Africa, Asia and key to fighting Latin America disease is necessary for successBeyond the impact factorTom Olijhoek SURF-NL
  • Specs for a global system Open Access can contribute to that, provided that the existing dominant features of the existing system: Citation counts and the JIF measures of impact (inadequate, insufficient and subject to gaming) will be replaced by measures that much better reflects the impact of research not only on research itself, but on innovation, health, wealth and societies (altmetrics). Luckily these are as well requirements of a successful breakthrough of OA in the North. Lars Bjørnshauge SPARC Europe – www.sparceurope.org The Hague Oct 25th 2012Beyond the impact factorTom Olijhoek SURF-NL
  • ultimately scientists need to realize that they hold the power in their own hands TAKE ACTIONBeyond the impact factorTom Olijhoek SURF-NL
  • When did scientists start agreeing with this slave-type of agreement with publishing houses? How could this nonsense have started? We inherited this sick system, but that does not mean we should allow it to continue MalariaWorld Survey on Open Access 2012Beyond the impact factorTom Olijhoek SURF-NL
  • A new Journal Impact Factor? Do we really need it? In an Open Access world We only need: • article level metrics to assess the impact of articles AND scientists • a quality indicator (seal?) but NOT impact factor to assess the quality of journals In an Open Access world It does not matter much where something is published, more important is the quality of the individual articles HOW GOOD VERSUS HOW MUCH AND WHEREBeyond the impact factorTom Olijhoek SURF-NL
  • The Relevance Index Use of new metrics to assess the impact of scientific works in all areas not only science but including innovation, health, wealth and societiesBeyond the impact factorTom Olijhoek SURF-NL
  • The Reputation Index Use of the Relevance Index to assess the reputation of authors / scientistsBeyond the impact factorTom Olijhoek SURF-NL
  • We need a Quality Indicator for (0pen access) journals WHY?Beyond the impact factorTom Olijhoek SURF-NL
  • New quality assessment methods in science Articles Journals Authors • Citations • Impact Factor • Citation index • H-Index • NEW • Citation based • NEW • Relevance Index • Reputation • Multi-level tools • NEW Index • Total Impact • A-Vector • Subjective • Altmetric • Based on quality • Based on more explorer of peer review & than citations quality of editorial boardBeyond the impact factorTom Olijhoek SURF-NL
  • A-VECTOR: INTRODUCING A NEW FORM OF JOURNAL LEVEL METRICS Results of the Rotterdam colloquium held on 22-23 October 2012 Quality of editorial board Quality of peer-review • Citation index • “Transparency” indicators • Reputation • Criteria used by reviewers • Collaboration • Duration of review process • Reference density • Post-publication comments • Openness about • More indicators • submission and Rejection rates • potential conflicts of interest • Aims, scopes and expected readership • Reviewer‟s comments and editorial correspondence ( published alongside papers • More indicatorsBeyond the impact factorTom Olijhoek SURF-NL
  • PREDICTION OF JOURNAL QUALITY BY A-VECTOR USING EDITORIAL BOARD MEMBER QUALITIES AS INDICATORS  CONCLUSION • Prediction of Impact Factor based on TCS (total citation  Quality of editorial score), MCS (mean citation score), H-index of Editorial board members can be Board Members used to judge the • Performance test: prediction of IF for established journals (potential) quality of with error of 1.5 points (95% confidence) Open Access journals • Requires some refinement, but can be applied to young OA journalsBeyond the impact factorTom Olijhoek SURF-NL
  • PREDICTION OF JOURNAL QUALITY BY A-VECTOR USING PEER-REVIEW TRANSPARENCY FACTORS AS INDICATORS 100 journals / 221 authors 17 established journals: score = 45.4, SD =11.9 CONCLUSION: 12 OA established journals: score = 54.7, SD = 7.7 13 OA Predatory journals: score = 34.1, SD = 6.7 authors‟ assessments of the quality of the peer- review of accepted papers can be predicted by using a set of 15 indicators of transparency for the peer- review process of journals EXAMPLES: Recent Scientific Research score 24 PLoSONE score 67.7 Malaria Journal score 50 MalariaWorld Journal score 47Beyond the impact factorTom Olijhoek SURF-NL
  • Item/criterion [authors] Converg Item- ence (η2) rest correl. 1. Aims, scope, and expected readership of the .866 .457 journal are clearly specified on the journal’s website 2. Types of submissions that are deemed appropriate for the journal are explicated on the .899 .587 website 3. Criteria used by reviewers to rate submissions are .692 .699 specified on the website 4. The website indicates whether all submissions are sent out for review and who will make final .821 .660 decisions about them (e.g., editor, associate/action editor) 5. The website provides timely updates of the status of submissions during the peer-review process (e.g., .749 .723 under review)Beyond the impact factorTom Olijhoek SURF-NL
  • Reliability Across raters: Convergence Across items: Reliability Journal q1 q2 q3 q4 q5 q6 1 5 5 5 5 5 5 1 5 5 1 4 3 2 1 5 4 3 1 5 5 1 4 2 1 4 4 2 2 4 4 5 3 5 3 2 5 5 5 5 5 3 2 5 5 4 3 4 3 2 5 5 3 3 5 3 3 5 5 4 4 4 2 Convergence across raters (13 journals): .838 Reliability across items .900Beyond the impact factorTom Olijhoek SURF-NL
  • YOU CAN NOW USE A-VECTOR YOURSELVES TO PREDICT JOURNAL QUALITY USING THE NIEW TRANSPARENCY FACTORS Go to the link : http://tinyurl.com/8br9m8w And complete the surveyBeyond the impact factorTom Olijhoek SURF-NL
  • YOU CAN NOW USE A-VECTOR YOURSELVES TO PREDICT JOURNAL QUALITY USING THE NIEW TRANSPARENCY FACTORS Applying the indicator The journals we would like you to assess are as follows: Journal no. 40 Journal title: Regional Studies Journal URL http://www.tandfonline.com/toc/cres20/current Journal no. 46 Journal title: The Internet Journal of Psychiatry Journal URL http://www.ispub.com/journal/the-internet-journal-of-psychiatry/ Journal no. 8 Journal title: BMC Neuroscience Journal URL http://www.biomedcentral.com/bmcneurosci/ The indicator can be found here: tinyurl.com/8br9m8wBeyond the impact factorTom Olijhoek SURF-NL