Citation analysis for research evaluation

11,459 views
11,192 views

Published on

The powerpoint presentation for the course on citation analysis we had this morning

Published in: Economy & Finance, Technology
0 Comments
23 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
11,459
On SlideShare
0
From Embeds
0
Number of Embeds
91
Actions
Shares
0
Downloads
0
Comments
0
Likes
23
Embeds 0
No embeds

No notes for slide

Citation analysis for research evaluation

  1. 1. Citation analysis for research evaluation Wouter Gerritsma, Marc Loman, Marianne Renkema Information Specialists, Wageningen UR Library
  2. 2. Programme <ul><li>Introduction </li></ul><ul><li>Citation data </li></ul><ul><li>Benchmarking </li></ul><ul><li>Journal quality </li></ul><ul><li>Ranking University </li></ul><ul><li>On publishing </li></ul>
  3. 3. Why research evaluation/citation analysis? <ul><li>Research assement of graduate schools (5 year cycle, stipulated by the VSNU) </li></ul><ul><ul><li>Other research groups are reviewed in a similar way </li></ul></ul><ul><ul><li>Benchmarking becomes more important </li></ul></ul><ul><li>Grant applications </li></ul><ul><li>Job positions </li></ul><ul><li>In the press </li></ul>Sooner of later, you will be subjected to such an analysis
  4. 4. Objective of this course <ul><li>How are citation analyses performed, and what are their shortcomings or weaknesses? </li></ul><ul><li>What can you do to enhance your results </li></ul>
  5. 5. Citation data
  6. 6. Citation data <ul><li>Paid subscription (wide coverage) </li></ul><ul><ul><li>Web of Science </li></ul></ul><ul><ul><li>Scopus </li></ul></ul><ul><li>Free systems broad coverage </li></ul><ul><ul><li>Google Scholar </li></ul></ul><ul><li>Paid subscription (specialized coverage) </li></ul><ul><ul><li>SciFinder </li></ul></ul><ul><ul><li>Psychindex </li></ul></ul><ul><li>Free systems (specialized coverage) </li></ul><ul><ul><li>Citeseer </li></ul></ul><ul><ul><li>Citebase </li></ul></ul>
  7. 7. Web of Science <ul><li>Thomson Scientific </li></ul><ul><ul><li>Founder Eugene Garfield </li></ul></ul><ul><ul><li>Formerly: Institute of Scientific Information (ISI) </li></ul></ul><ul><li>Indexes articles of about 8700 journals, standard bibliographic information + reference lists </li></ul><ul><li>Sell the same data in various databases to libraries and others (CC, WoS, JCR, ESI) </li></ul><ul><li>They (still) have monopoly position for citation data; they are the Golden Standard </li></ul>
  8. 8. CC, WoS, JCR, ESI <ul><li>CC: Current Contents, weekly updated bibliographic database covering the top of Sciences (no citation data) </li></ul><ul><li>WoS: Web of Science, weekly updated database, that includes references and citation data as well (slower than CC) </li></ul><ul><li>JCR: Journal Citation Report, yearly analysis of WoS data, presenting perfomance measures for journals. </li></ul><ul><li>ESI, Essential Science Indicators, bimonthly updated analytical database giving performance measures for Countries, Institutes, Scientists and Journals next to research fronts, hot papers and baselines. </li></ul>
  9. 9. Database Coverage WoS/JCR/CC Source: Moed (2005) Mathematics / Engineering / Economics Other social sciences related to medicine & health Physics & Astronomy Psychology & psychiatry Clinical medicine Humanities Biological Sciences related to plants & animals Biological Sciences related to humans Other Social Sciences Applied physics & Chemistry Molecular biology & Biochemistry Moderate Good Excellent
  10. 10. Scopus <ul><li>(New) product of Elsevier Science </li></ul><ul><li>Covers 15,000 serials </li></ul><ul><ul><li>Not all of them peer reviewed </li></ul></ul><ul><li>Coverage of citation data 1996 -> </li></ul>
  11. 11. Google Scholar <ul><li>Free database </li></ul><ul><li>Some concerns about data quality </li></ul><ul><li>Unknown source base </li></ul><ul><li>Very labour intensive to extract data </li></ul>
  12. 12. Why Scopus doesn't add substantially to the number of citations found in WoS Source: http://www.wowter.nl/blog/2007/01/why-scopus-doesnt-add-substantially-to.html
  13. 13. Searching WoS <ul><li>General search: Searches bibliographic data: Title, Abstract, keywords, Authors, Addresses </li></ul><ul><ul><li>Not all citations are found due to errors in citing, indexing etc. </li></ul></ul><ul><li>Cited reference search: searches in the references of source data </li></ul><ul><ul><li>Not all articles are found. Non cited articles are missing </li></ul></ul>
  14. 15. Exercises !
  15. 16. Benchmarking
  16. 17. Benchmarking <ul><li>Essential Science Indicators (ESI) </li></ul><ul><li>h-index </li></ul>
  17. 18. How do we compare numbers? <ul><li>Scientist Z. Math has a publication from 1996 with 17 citations </li></ul><ul><li>Scientist M. Biology has a publication from 2003 with 24 citations </li></ul>
  18. 19. Baseline mathematics
  19. 20. Baseline Molecular Biology
  20. 21. Essential Science Indicators (ESI) <ul><li>Analytical database of SCI, covering 10 years + current year building </li></ul><ul><li>Comparison between Countries, Institutes, Scientists and Journals </li></ul><ul><li>Hot papers / Highly cited papers </li></ul><ul><li>Research fronts </li></ul><ul><li>Baselines </li></ul>
  21. 22. Publication age : world averages citations (1)
  22. 23. In practice
  23. 24. Relative impact of a research institute 1,84 2,26 2,08 2,06 All subfields 1,73 0.57 1,70 Microbiology 1,11 1,81 1,73 Clinical medicine 1,76 1,76 Chemistry 1,09 0,44 1,55 0,91 Biology & biochemistry 3,60 3,87 3,86 3,82 Agricultural Sciences Group 3 Group 2 Group 1 All groups Subfield
  24. 25. Evaluation of candidates 0 3 0,83 0,58 0,75 346 57 e 3 16 1,94 1,69 1,86 1886 88 d 0 8 0,9 1,39 1,15 972 93 c 1 17 1,95 1,84 1,93 498 65 b 2 4 1,52 1,76 1,64 1565 80 a #papers top 1% #papers top 10% RI 1999-2003 RI 1994-1998 Relatieve Impact #Citations # Papers 1994-2003 Author
  25. 26. h-index <ul><li>A scientist has index h if h of his/her Np papers have at least h citations each, and the other (Np − h) paper have no more than h citations each. </li></ul>
  26. 27. h-index <ul><li>h-index tries to find a balance between productivity and citation impact </li></ul><ul><li>Only published in 2005, has made a substantial impact in the world of bibliometrics. </li></ul><ul><li>Implemented in Spires, WoS (and soon) in Scopus </li></ul><ul><li>Applicable to authors, journals, research groups, compounds, subjects etc … </li></ul>
  27. 28. Exercises !
  28. 29. Journal quality
  29. 30. Journal quality <ul><li>JCR (Impact factors) </li></ul><ul><li>ESI </li></ul><ul><li>h-index </li></ul><ul><ul><ul><li>Or pagerank ? </li></ul></ul></ul>
  30. 31. Warning ! <ul><li>It has often been attempted to use Journal Impact factors in some or other way to judge the performance of a group or individual scientists. </li></ul>This is proven very bad practice!
  31. 32. Impact factor <ul><li>Performance measure for journals </li></ul><ul><ul><li>“… it is also used for assessment of the quality of individual papers, scientists and departments. For the latter a scientific basis is lacking, as we will demonstrate in this contribution” (Opthof, 1997) </li></ul></ul><ul><ul><li>Opthof, T. (1997). Sense and nonsense about the impact factor. Cardiovascular Research 33 (1): 1-7. http:// dx.doi.org /10.1016/S0008-6363(96)00215-5 </li></ul></ul>
  32. 33. Journal Citation Reports (JCR) <ul><li>Reports 3 measures </li></ul><ul><li>Impact factor </li></ul><ul><li>Immediacy Index </li></ul><ul><li>Cited half life </li></ul>Cited half-life 50% citations 50% citations 3 1 Immediacy index Window Impact Factor Window
  33. 34. Definition of Impact factor <ul><li>The impact factor of a journal J in year T is defined as follows: </li></ul><ul><li>“ The number of citations received in year T by all documents published in J in the years T-1 and T-2” </li></ul><ul><li>÷ </li></ul><ul><li>“ The number of citable documents published in J in the years T-1 and T-2” </li></ul>
  34. 35. Citable items <ul><li>Cited items (nominator) include: </li></ul><ul><ul><li>Research articles, review arrrticles, notes, letters, editorials, news items, corrections and meeting abstracts </li></ul></ul><ul><li>Citable items (by definition) </li></ul><ul><ul><li>Articles, notes and reviews </li></ul></ul>
  35. 36. Lancet (2002) JCR Like IF = (b)/(a) = 10.2 2.4 15,670(b) 6,443 Total 0.5 2,564 4,899 Other types 8.5 13,106 1,544 (a) Articles/reviews (citable items) Cites/Doc Cites No. Docs Type of document
  36. 37. 50 % of articles generate 90% of all cites Seglen, P. O. (1997). Why the impact factor of journals should not be used for evaluating research. BMJ 314 (7079): 497-502. http://bmj.bmjjournals.com/cgi/content/full/314/7079/497
  37. 38. Some other points <ul><li>IF measures citation impact of articles in 2 nd & 3 rd year after publication. Therefore biased towards journals showing rapid maturing. </li></ul><ul><li>Reference practices, particularly the number of references per article and their age distribution, vary considerably among subfields. IF between subfields can not be compared . </li></ul>
  38. 39. When do you make use of IF <ul><li>Answering the questions where do you want to publish your manuscript. </li></ul><ul><li>Other questions, to be asked </li></ul><ul><ul><li>Reach colleagues ? </li></ul></ul><ul><ul><li>In which article indexes included? </li></ul></ul><ul><ul><li>Open access? </li></ul></ul>
  39. 40. Exercises !
  40. 41. Ranking universities <ul><li>Popular lists </li></ul><ul><ul><li>Shanghai Jiao Tong University </li></ul></ul><ul><ul><li>THES </li></ul></ul><ul><ul><li>Newsweek </li></ul></ul><ul><li>ISIhighly cited </li></ul><ul><li>ESI </li></ul><ul><li>Other reports </li></ul><ul><ul><li>EC, NOWT, OECD, USA </li></ul></ul>
  41. 42. Winding down
  42. 49. What’s in a name? <ul><li>A.G.J. Voragen or A. Voragen or F. Voragen or F.A.G.J. Voragen or F.G.J. Voragen </li></ul>

×