1. A new role for libraries in researchassessmentsTicer International Summer School 2012Wouter Gerritsma
2. Contents Introduction Research assessments CRIS and repository Advanced bibliometrics Alternatives Why in the library How can you start?
3. Introducing Wageningen UR Library Wageningen University & Research Centre is the results of the merger between Wageningen University and the former DoA research institutes Life Sciences is our domain Some 6500 researchers and 10000 students Strong international focus Ranked 75th worldwide, 2nd in the Netherlands, 17th in Life Sciences ranking of THE WUR 2001-12 ranking
4. Bibliometrics at Wageningen UR Library Since the 1990s few citation analysis with SciSearch on Dialog and DIMDI 2001: Web of Science ● Collection analysis ● First exercises with bibliometrics 2004: Essential Science Indicators ● Bibliometric analysis for graduate school WIAS 2008: Start of implementation of bibliometric services on our (metadata-)repository Wageningen Yield
5. National research assessments elsewhere Australia: The Excellence in Research for Australia (ERA) http://arc.gov.au/era UK: Research Excellence Framework http://www.ref.ac.uk ● The pilot exercise concluded that citation information is not sufficiently robust to be used formulaically or as a primary indicator of quality in the REF; but there is scope for such data to inform and enhance the process of expert review.
6. Research assessment in the Netherlands Supervised by VSNU/QANU ● 6 year cycle for external peer reviews ● After 3 year midterm review ● Unit of analysis (in Wageningen): Graduate schools Citation analyses are not stipulated in the current Standard Evaluation Protocol (SEP). ● This has become mandatory at Wageningen UR, als at the social sciences department and for the research institutes
7. SEP Criteria Quality (including international academic reputation and PhD training) Productivity (the relationship between input and output) Societal relevance (including valorisation) Vitality and feasibility (the ability to react adequately to important changes in the environment).
8. Metis, our CRIS Metis is a Current Research Information System (CRIS) Data entry at chair group level Quality control by the library ● Locating full text (uploading to e-depot) ● Maintenance journal lists ● Document type assignation Compulsory output registration ● Research assessments only on metis registered pubications Information on all labour relations of faculty and staff Information on all projects
9. Wageningen Yield Wageningen Yield (WaY) is the bibliographic database of all output of Wageningen UR and OA repository Comprehensive for the university since 1976, for the research institutes since 1996 Metadata records as well as full text Ca. 180,000 metadata records Ca. 37,000 full text OA publications Since 2003 fed by metis, ca. 90,000 records Synchronization each night with updates from metis
10. Collecting citation data from Web ofScience WoS record Identifier (UT) Thomson Reuters Article Match Retrieval service http://wokinfo.com/directlinks/amrfaq/ API ● Input DOI or ISSN/Journal Title/Volume/Start Page ● Return: UT ● Input: UT ● Return: Times Cited
11. Full screen image with title
12. How are we able to compare numbers? Scientist Z. Math has a publication from 2001 with 17 citations Scientist M. Biology has a publication from 2007 with 32 citations
13. Baselines for Mathematics
14. Baselines for Molecular Biology
15. A quantitative example Bouma, J, Bulte, EH, & DP van Soest (2008) Trust, Trustworthiness and Cooperation: Social Capital and Community Resource Management. J. Env. Ec. & Mngmt 56(2)155-166. ● Cited 12 times Journal of Environmental Economics and Management from journals menu in ESI: ● Economics & Business Baseline data for Economics & Business (from ESI) ● Article from 2008: Average: 3.62 Citations; Top 10%: 9 citations; Top 1% 27 citations RI = 12 / 3.62 = 3.31
16. Advanced bibliometric indicators Follow Moed (1995) as closely as possible; but..... Web of Science is used for citation data ● We can’t make corrections for self citations Essential Science Indicators for baseline data (World average, Top 10% and Top 1%) ● Limited number of research fields (22)BUT: We can determine the representativeness of the citation analysis!
17. Representativeness Publication type #Pubs Refereed articles 324 Non-refereed articles 7 Books 1 Refereed book chapters 36 Non-refereed book chapters 13 PhD Theses 45 Conference papers 137 Total Academic Publications 563
18. RepresentativenessPublication type #Pubs WoS Repr.Refereed articles 324 288 89%Non-refereed articles 7Google Scholar 1Refereed book chapters 36Non-refereed book chapters 13PhD Theses 45Conference papers 137Total Academic Publications 563
19. Slice and dice any way you want
20. Standard citation reports are created forany set of publications
21. Accountability is of utmost importance!
22. Register once use many
23. Register once use many
24. Register once use many
25. Register once use many
26. Matching Way and WoS1161 peer reviewedarticles not in ISI Missing from WoS:journals 1159 articlesWaY: 10933 articles WoS: 9577 articlesMissing in Way:807 articles
27. Sources of citation data Web of Science Scopus Google Scholar Microsoft Academic SciFinder; Psychinfo Citeseer and other open access repositories Various Altmetrics initiatives
28. Differences in citations reflected in h-index
29. Differences in citations reflected in h-index
30. Differences in citations reflected in h-index
31. Web of Science & Incites Citation data (includes also citations from other databases on Wok) API to download citation data Baselines from ESI "New" product InCites
32. Scopus & Strata Citation data obtainable through an API Benchmarking with SciVal Strata, no API yet Not yet fully developed
33. Google Scholar Give them a few more years Coverage? Ghost citations Content duplication Benchmarking?
34. Altmetrics http://altmetrics.org/manifesto/ Quickly developing ● PLoS article level metrics ● ScienceCard Wouters, P. & R. Costas (2012). Users, narcissism and control. Utrecht, NL: SURFfoundation. http://www.surffoundation.nl/ ● Total-Impact en/publicaties/Pages/Users_narcissism_control.aspx. ● Readermeter ● Microsoft Academic Search ● etc.
35. Why in the library? Library is the functional manager of CRIS / repository because of wide experience with bibliographic metadata Library manages contracts with publisher(s) of external databases that are being used Library has experience in developing and maintaining large databases Library has ample experience in searching complicated databases such as Web of Science
36. Advantage of CRIS/repository combination Improvements in publication lists, etc. recorded Knowledge of, and experience with bibliometric analyses is better institutionalized Clarity / transparency for researchers Analysis of a single unit of the institute offers advantages for whole institute Better understanding of our own researchers ● We know where they publish ● We know what they cite ● We know something about their impact
37. Raising library awareness Improvement of the (metadata) quality in the repository Quality has lead to compulsory registration for research assessments Presentations for research groups during the preparation for peer reviews Presentations based on detailed studies of single groups Library gives advice on elements for publication strategies for groups and individuals ● there is a huge demand for these workshops
38. Closing the circle With the coupling of publications with WoS We have gained insight in the relation ● Research group – Researchers – Publications – Reference list ● It is feasible to assign journal usage at faculty level, or more detailed (chair groups)Journal title Total AFSG ASG ESG Imares PSG SSGNATURE 2584 511 341 753 93 989 59PNAS 2467 787 325 166 20 1225 29SCIENCE 2303 529 239 594 52 970 99APPLIED AND ENVIRONMENTALMICROBIOLOGY 2257 1320 257 139 12 696 27PLANT PHYSIOLOGY 1597 379 4 58 0 1296 2JOURNAL OF BIOLOGICAL CHEMISTRY 1543 931 223 13 6 379 8
39. But where should you start? Courses on bibliometrics ● CWTS http://www.socialsciences.leiden.edu/cwts/ ● ESSS http://www.scientometrics-school.eu/ Conferences ● ISSI http://www.issi-society.info/aboutconf.html ● STI http://2012.sticonference.org/ Journals ● Scientometrics ● Journal of Informetrics Books ● Moed, H.F. (2005). Citation analysis in research evaluation. Dordrecht, Springer.
40. Thank you!http://tinyurl.com/7r67fmmOn the Web:@wowterwowter.netwww.slideshare.net/wowter