Advantages and drawbacks of bibliometrics

2,743 views

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,743
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
61
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • Ten Highest ranking (on IF-values) review journals
  • Advantages and drawbacks of bibliometrics

    1. 1. Application of bibliometric analysis Advantages & pitfalls Thed van Leeuwen Workshop on Research Evaluation in Statistical Sciences , Bologna, 25 th March 2010
    2. 2. Introduction of bibliometrics <ul><li>Bibliometrics can be defined as the quantitative analysis of science and technology performance and the cognitive and organizational structure of science and technology. </li></ul><ul><li>Basic for these analyses is the scientific communication between scientists through (mainly) journal publications . </li></ul><ul><li>Key concepts in bibliometrics are output and impact , as measured through publications and citations . </li></ul><ul><li>Important starting point in bibliometrics: scientists express, through citations in their scientific publications, a certain degree of influence of others on their own work. </li></ul><ul><li>By large scale quantification, citations indicate influence or (inter)national visibility of scientific activity, but should not be interpreted as synonym for ‘quality’. </li></ul>
    3. 3. CWTS data system <ul><li>CWTS has a full bibliometric license from Thomson Reuters Scientific to conduct evaluation studies using the Web of Science. </li></ul><ul><li>Our database covers the period 1981-2009. </li></ul><ul><li>Some characteristics: </li></ul><ul><ul><li>Over 31.000.000 publications. </li></ul></ul><ul><ul><li>Over 350.000.000 citation relations between source papers. </li></ul></ul><ul><ul><li>100.000.000 authors (incl. variations), 15.000.000 ‘unique’ names. </li></ul></ul><ul><ul><li>Over 60.000.000 addresses, some 90% cleaned up over the last 10 years. </li></ul></ul><ul><ul><li>Contains reference sets for journal and field citation data . </li></ul></ul>
    4. 4. Bibliometric indicators produced by CWTS
    5. 5. Some basic indicators are … <ul><li>P : number of publications in journals processed for the </li></ul><ul><li>Web of Science. </li></ul><ul><li>C : number of received citations, excl. self-citations. </li></ul><ul><li>CPP: mean number of citations per publication, excl. self- </li></ul><ul><li>citations </li></ul><ul><li>Pnc: percentage of the publications not cited (within a </li></ul><ul><li>certain time-frame !!!) </li></ul><ul><li>% SC : percentage self-citations related to an output set. </li></ul>
    6. 6. Important indicators are… <ul><li>CPP/JCSm : ratio between real, actual impact, and mean journal impact. </li></ul><ul><li>CPP/FCSm : ratio between real, actual impact, and mean field impact. </li></ul><ul><li>JCSm/FCSm: ratio between journal impact, and field impact, indicative for the ‘quality’ of the journal package in the field </li></ul>
    7. 7. Various types of analysis focus on … <ul><li>Research profiles: a break down of the output over various fields of science. </li></ul><ul><li>Scientific cooperation analysis: a break down of the output over various types of scientific collaboration. </li></ul><ul><li>Knowledge user analysis: a break down of the ‘responding’ output into citing fields, countries or institutions. </li></ul><ul><li>Highly cited paper analysis: which publications are among the most highly cited output (top 10%, 5%, 1%) of the global literature in that same field(s). </li></ul><ul><li>Social network analysis: how is the network of partners composed, based on scientific cooperation. </li></ul>
    8. 8. Journal & Field Normalization
    9. 9. Calculating the JCSm & FCSm <ul><li>---------------------------------------------------------------------------------------------- </li></ul><ul><li>Type publ. Journal Journal # citations </li></ul><ul><li> year category until 1999 </li></ul><ul><li>----------------------------------------------------------------------------------------------  </li></ul><ul><li>I review 1996 CANCER RES Oncology 17 </li></ul><ul><li>  </li></ul><ul><li>II note 1997 J CLIN END Endocrinology 4 </li></ul><ul><li>  </li></ul><ul><li>III article 1999 J CLIN END Endocrinology 6 </li></ul><ul><li>IV article 1999 J CLIN END Endocrinology 8 </li></ul><ul><li>----------------------------------------------------------------------------------------------   </li></ul>
    10. 10. Calculating the JCSm & FCSm 2 <ul><li>----------------------------------------------------------------- </li></ul><ul><li> CPP JCS FCS </li></ul><ul><li>----------------------------------------------------------------- </li></ul><ul><li>I 17 16.9 23.7 </li></ul><ul><li>  </li></ul><ul><li>II 4 3.1 3.0 </li></ul><ul><li>  </li></ul><ul><li>III 6 4.8 4.1 </li></ul><ul><li>  </li></ul><ul><li>IV 8 4.8 4.1 </li></ul><ul><li>----------------------------------------------------------------- </li></ul>
    11. 11. Calculating the JCSm & FCSm 3 <ul><li>The mean citation score is determined as: </li></ul><ul><li> 17 + 4 + 6 + 8 </li></ul><ul><li>CPP = ------------------ = 8.8 </li></ul><ul><li> 1 + 1 + 1 + 1 </li></ul>The mean journal citation score as: (1 x 16.9) + (1 x 3.1) + (2 x 4.8) JCSm = -------------------------------------- = 7.4 1 + 1 + 2 The mean field citation score as : (1 x 23.7) + (1 x 3.0) + (2 x 4.1) FCSm = -------------------------------------- = 8.7 1 + 1 + 2 CPP / JCSm (8.8 / 7.4) = 1.19 CPP / FCSm (8.8 / 8.7) = 1.01
    12. 12. Citation Windows & Impact Measurement
    13. 13. Citation measurement and ‘windows’ <ul><li>Publication years, fixed citation ‘ window ’ . </li></ul><ul><li>P ublications of 2002, with three citation years (namely 2002, 2003, and 2004), followed by 2003, with three years, etc. </li></ul><ul><li>Blocks of publication years with a window decreasing in length . </li></ul><ul><li>P ublications of 2002-2005, with citation window of 4 years (2002-2005), 3 years (2003-2005) , 2 years (2004-2005), and 1 year (2005). </li></ul>
    14. 14. Citation measurement with ‘fixed window’ <ul><li>Citation years </li></ul><ul><li> 2002 2003 2004 2005 2006 2007 2008 2009 </li></ul>2002 2003 2004 2005 2006 2007 2008 2009 2002 2003 2004 2003 2004 2005 2004 2005 2006 2005 2006 2007 2006 2007 2008 2007 2008 2009 2008 2009 2009
    15. 15. Citation measurement with ‘year blocks’ Citation years 2002 2003 2004 2005 2006 2007 2008 2009 2002 2003 2004 2005 2006 2007 2008 2009 2002 2003 2004 2005 2003 2004 2005 2004 2005 2005 2003 2004 2005 2006 2004 2005 2006 2005 2006 2006 2004 2005 2006 2007 2005 2006 2007 2006 2007 2007 2005 2006 2007 2008 2006 2007 2008 2007 2008 2008 2006 2007 2008 2009 2007 2008 2009 2008 2009 2009
    16. 16. Methodological issues
    17. 17. Adequacy of citation indexes : implications for bibliometric studies
    18. 18. How to tackle this issue ? <ul><li>We conduct analyses on the adequacy of the citation indexes across disciplines based on reference behavior of researchers themselves. </li></ul><ul><li>The degree of referring towards other indexed literature indicates the importance of journal literature in the scientific communication process. </li></ul>
    19. 19. WoS Non-WoS Non-WoS WoS Citing/Source Cited/Target ?% ?% Assessment of WoS Coverage Non-Wos Journals Books Conference proceedings Reports Etc.
    20. 20. WoS Non-WoS Non-WoS WoS Citing/Source Cited/Target 75% 25% Total ISI/WoS Database (2002)
    21. 21. The medical & Life sciences
    22. 22. The natural sciences
    23. 23. Statistical sciences
    24. 24. The engineering sciences
    25. 25. The social– and behavioral sciences
    26. 26. The humanities
    27. 27. Overall WoS coverage by main field EXCELLENT (> 80%) VERY GOOD (60-80%) GOOD(40-60%) Biochem & Mol Biol Appl Phys & Chem Mathematics & Statistical sciences Biol Sci – Humans Biol Sci – Anim & Plants Economics Chemistry Psychol & Psychiat Engineering Clin Medicine Geosciences MODERATE (<40 %) Phys & Astron Soc Sci ~ Medicine Other Soc Sci Humanities & Arts
    28. 28. Conclusions on adequacy issue <ul><li>We can clearly conclude that the application of bibliometric techniques, solely based on WoS (but very likely also Scopus) will not be valid for some of the ‘soft’ fields in the social sciences and the humanities. </li></ul><ul><li>That is why the tool box has to be extended ! </li></ul>
    29. 29. The H-Index and its limitations
    30. 30. The H-Index, defined as … <ul><li>The H-Index is the score that indicates the position at which a publication in a set, the number of received citations is equal to the ranking position of that publication. </li></ul><ul><li>Idea of an American physicist, J. Hirsch, who published about this index in the Proc. NAS USA. </li></ul>
    31. 31. Examples of Hirsch-index values <ul><li>Environmental biologist, output of 188 papers, cited 4,788 times in the period 80-04. </li></ul><ul><li>Hirsch-index value of 31 </li></ul><ul><li>Clinical psychologist, output of 72 papers, cited 760 time sin the period 80-04. </li></ul><ul><li>Hirsch-index value of 14 </li></ul>
    32. 32. Problems with the H-Index <ul><li>For serious evaluation of scientific performance, the H-Index is as indicator not suitable, as the index: </li></ul><ul><ul><li>Is insensitive to field specific characteristics (e.g., difference in citation cultures between medicine and other disciplines). </li></ul></ul><ul><ul><li>Does not take into account age and career length of scientists, a small oeuvre leads necessarily to a low H-Index value. </li></ul></ul><ul><ul><li>Is inconsistent in its ‘behaviour’. </li></ul></ul>
    33. 33. <ul><li>Actual versus field normalized impact (CPP/FCSm) displayed against the output. </li></ul><ul><li>Large output can be combined with a relatively low impact </li></ul>
    34. 34. <ul><li>H-Index displayed against the output. </li></ul><ul><li>Larger output is strongly correlated with a high H-Index value. </li></ul>
    35. 35. Consistency: Definition <ul><li>Definition. A scientific performance measure is said to be consistent if and only if for any two actors A and B and for any number n ≥ 0 the ranking of A and B given by the performance measure does not change when A and B both have a new publication with n citations. </li></ul>
    36. 36. Consistency: Motivation <ul><li>Consistency ensures that if the publishing behavior of two actors does not change over time, their ranking relative to each other also does not change </li></ul><ul><li>Consistency ensures that if the individual researchers in one research group X outperform the individual researchers in another research group Y , the former research group X as a whole outperforms the latter research group Y. </li></ul>
    37. 37. Inconsistency of the h-index Actor A Actor B h = 4 h = 6 h = 6 h = 8
    38. 38. ISI Impact Factors: calculation and validity
    39. 39. Methodology: ISI’s classical IF <ul><li>The ISI Impact Factor (IF) is defined as the number of citations received by a journal in year t, divided by the number of citeable documents in that same journal in the years t-1 and t-2, </li></ul><ul><li>Or, as a Formula: </li></ul>Citations in year t Number of ‘citeable documents’ in t -1 & t -2
    40. 40. Share ‘citations-for-free’ for The Lancet <ul><li>Publications Citations </li></ul><ul><li> 90+91 1992 </li></ul><ul><li>Article 784 2986 </li></ul><ul><li>Note 144 593 </li></ul><ul><li>Review 29 232 </li></ul><ul><li>Sub-total 957 (a) 7959 (b) </li></ul><ul><li>Letter 4181 (d) 4264 (e) </li></ul><ul><li>Editorial 1313 905 </li></ul><ul><li>Other 1421 909 </li></ul><ul><li>Total 7872 14037 (c) </li></ul><ul><li>ISI Method : </li></ul><ul><li>Citations in 2000 . </li></ul><ul><li>Citeable documents in ‘98 and ‘99 </li></ul><ul><li>14037 (c) </li></ul><ul><li>957 (a) </li></ul>IF=14.7 <ul><li>CWTS Method : </li></ul><ul><li>Citations to Art/Not/Rev in 2000 . </li></ul><ul><li>Art/Not/Rev in ‘98 and ‘99 </li></ul><ul><li>7959 (b) </li></ul><ul><li>957 (a) </li></ul><ul><li>Citations to Art/Let/Not/Rev in 2000 . </li></ul><ul><li>Art/Let/Not/Rev in ‘98 and ‘99 </li></ul><ul><li>7959+4264 (b+e) </li></ul><ul><li>957+4181 (a+d) </li></ul>IF=8.3 IF=2.4
    41. 41. ISI Impact Factors <ul><li>From 1995 onwards CWTS has analyzed the uses and validity ISI Journal Impact Factor (IF). </li></ul><ul><li>Most important points of criticism were: </li></ul><ul><ul><li>Calculated erroneously . </li></ul></ul><ul><ul><li>Not sensitive for the composition of the journal in terms of the document types. </li></ul></ul><ul><ul><li>Not sensitive for the science fields a journal is attached to … </li></ul></ul><ul><ul><li>Based on too short ‘citation windows’ . </li></ul></ul>
    42. 42. Distribution of citations used for the calculationof the IF value of The Lancet <ul><li>The IF-score of The Lancet is seriously ‘overrated’ by the scientific ‘audience’ of the journal. </li></ul><ul><li>The red area indicates citations ‘for free’, while the blue area indicates ‘correct citations ’ </li></ul>
    43. 43. Impact Factors for Br. J. Clin. Pharm. and Clin. Pharm. & Ther. <ul><li>The graph shows the correct and erroneous impact factors of BJCP and CPT </li></ul><ul><li>In the case of CPT , citations to published meeting abstracts are included, while BJCP has stopped publishing of meeting abstracts ! </li></ul>
    44. 44. Document types and fields Field Journal IF JFIS The IF is for ‘02, JFIS covers ‘98-‘02 IMMUNOLOGY ANN REV IMMUNOL 50.49 1 5.18 1 BIOCHEM & MOLECULAR BIOL ANN REV BIOCHEM 34.61 1 4.10 3 PHARMACOL & PHARMACY PHARMACOLOGICAL REV 27.74 1 4.75 1 CELL BIOL ANN REV CELL & DEVELOPM BIOL 27.53 1 1.72 13 DEVELOPMENTAL BIOL ANN REV CELL & DEVELOPM BIOL 27.53 1 1.72 3 PHYSIOLOGY PHYSIOLOGICAL REV 24.82 1 3.18 1 CELL BIOLOGY NATURE REV MOL CELL BIOL 22.21 4 2.76 8 ENDOCRINOL & METABOLISM ENDOCRINE REV 21.98 1 2.87 1 NEUROSCIENCES ANN REV NEUROSCIENCE 21.89 1 3.12 4 PHYSICS REV MODERN PHYSICS 20.14 1 5.02 1 CHEMISTRY CHEMICAL REV 19.67 1 2.89 2
    45. 45. Fields and Citation windows
    46. 46. Citation measurement of IF <ul><li>2002 2003 2004 2005 2006 2007 2008 2009 </li></ul>2002 2003 2004 2005 2006 2007 2008 2009 2002 2003 2004 2003 2004 2005 2004 2005 2006 2005 2006 2007 2006 2007 2008 2007 2008 2009 2008 2009 2009
    47. 47. CWTS answer to the problems of the IF <ul><li>This indicator is the JFIS , the J ournal-to- F ield I mpact S core . </li></ul><ul><li>The JFIS solves the main objections against the Impact Factor, as </li></ul><ul><ul><li>the calculation of JFIS is based on equally large entities, </li></ul></ul><ul><ul><li>document types are taken into account , </li></ul></ul><ul><ul><li>JFIS is field-normalized, and finally, </li></ul></ul><ul><ul><li>based on longer citation windows (1-4 years) </li></ul></ul>
    48. 48. Citation measurement of JFIS Citation years 2002 2003 2004 2005 2006 2007 2008 2009 2002 2003 2004 2005 2006 2007 2008 2009 2002 2003 2004 2005 2003 2004 2005 2004 2005 2005 2003 2004 2005 2006 2004 2005 2006 2005 2006 2006 2004 2005 2006 2007 2005 2006 2007 2006 2007 2007 2005 2006 2007 2008 2006 2007 2008 2007 2008 2008 2006 2007 2008 2009 2007 2008 2009 2008 2009 2009
    49. 49. <ul><li>End of the presentation </li></ul><ul><li>For questions regarding the contents of the presentation, mail to: [email_address] </li></ul>

    ×