Your SlideShare is downloading. ×
Citation impact: an introduction to bibliometrics for researchers
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Citation impact: an introduction to bibliometrics for researchers

398
views

Published on

Published in: Education, Technology

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
398
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
10
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Citation Impact: An Introduction to Bibliometrics for Researchers Breeda Herlihy (Institutional Repository Manager) b.herlihy@ucc.ie Ronan Madden (Arts & Humanities Librarian) r.madden@ucc.ie
  • 2. Aims and Outcomes Aim: • To look at methods of identifying and interpreting research performance data as a measure of research impact Outcomes: • Use citation analysis tools (as provided by Web of Science, Scopus, Google Scholar) to evaluate research impact • Understand the limitations of bibliometrics • Utilise publishing strategies to improve citation performance 28/11/2013 UCC Library Introduction to Bibliometrics
  • 3. Format 1. Introduction Research evaluation, peer-review, citation analysis & bibliometrics 2. Citation Impact Citations per paper, individual, institution, country 3. Journal Impact Impact factor, SJR, etc. 4. 5. 28/11/2013 Caveats Publishing Strategies incl. Open Access UCC Library Introduction to Bibliometrics
  • 4. ‗Scholarly‘ Communication • Traditional Approach: From informal to formal, communication of research at every stage; evaluation and feedback from peers at every stage 1. Initial/Informal: meetings, discussions, seminars, emails, funding applications, blogs, social networking, twitter 2. Report on-going research at a conference 3. Publication in an academic journal, scholarly monograph 4. Indexed in research/bibliographic databases, catalogues, repositories 28/11/2013 UCC Library Introduction to Bibliometrics
  • 5. Evaluating Research • Qualitative approach: ‗Peer-reviewed‘ Possible at every stage of research process from funding to publication. Types: Double-blind, Single-blind, Open etc. Limitations: See ‗Peer-review: a guide for Researchers‘ • Quantitative approach: Bibliometrics/Citation analysis – Measures ‗research impact‘ (not ‗quality‘) 28/11/2013 UCC Library Introduction to Bibliometrics
  • 6. Research Impact • Not just bibliometrics (which focuses on journals) Other methods of dissemination: monographs, book chapters, conferences, reports, digital objects Also includes: - research income generated - funding & awards - number of PhD students/graduates - patents and licences obtained - impact on society, ‗cultural capital‘ (social, economic, cultural) 28/11/2013 UCC Library Introduction to Bibliometrics
  • 7. Citation Impact/bibliometrics ‘ The branch of library science concerned with the application of mathematical and statistical analysis to bibliography; the statistical analysis of books, articles, or other publications’ Oxford English Dictionary • Quantitative measures… • Impact of a paper, a journal, an individual researcher, institution/university, country etc. • Controversial? – each should be judged on their own merits within context. Fair? Objective? Cost effective? • Caveats – more later… 28/11/2013 UCC Library Introduction to Bibliometrics
  • 8. Who‘s interested in bibliometrics? • Governments and funding agencies - return on investment, grant applications [See video] • Institutions/ Universities - benchmarking, strengths/weaknesses, recruiting, return on investment • Units within universities e.g. schools, colleges - benchmarking, strengths/weaknesses, collaboration, recruiting [video] • Individual researchers - promotion/recruitment, tenure, grants [video] • Librarians! – subscriptions, value for money 28/11/2013 UCC Library Introduction to Bibliometrics
  • 9. Citation Analysis (Bibliometrics) • The number of times a paper, researcher, journal is cited by others; assumes that influential authors and important works are cited more often than others • 2 elements: - 1. Citation Impact: measures the influence of particular papers (times cited) and authors (e.g. H-index). Data can be used to help evaluate departments, universities, countries - 2. Journal Impact : measures the influence that particular journals have within a field of study 28/11/2013 UCC Library Introduction to Bibliometrics
  • 10. Where to find the data: 1. Web of Knowledge (Thomson Reuters) incl. Web of Science’s 12,300+ peer-reviewed journals, includes ‘Journal Citation Reports’ and ‘Essential Science Indicators’. Data is used by ‘InCites’. See title listing. 2. Scopus (Elsevier) 19,800+ peer-reviewed journals; 1996 on. Scimago Country & Journal Rank Database details journals and country-specific scientific indicators developed from the information contained in Scopus 3. Google Scholar provides links to ‘cited by’ information, and some metrics. Data is harvested by ‗Publish or Perish‘ software to produce more detailed metrics 4. ‗Altmetrics‘ & Web metrics is an emerging area 28/11/2013 UCC Library Introduction to Bibliometrics
  • 11. SCOPUS & WOS coverage comparison 934 11,377 8,432 Source: JISC http://adat.crl.edu Scopus: from 1996 only i.e. width rather than depth 28/11/2013 UCC Library Introduction to Bibliometrics
  • 12. Source: http://www.ndlr.ie/myri/ 28/11/2013 UCC Library Introduction to Bibliometrics
  • 13. Citation Impact The impact of a paper or group of papers i.e. number of times an article/paper is cited by others. Can be used to measure impact of a researcher, institution, country etc. - Web of Science (part of Web of Knowledge) ‘Times cited‘, also includes ‘Cited Reference Searching’ to find articles that cite a person’s work. - Scopus see ‗Cited by‘ - Google Scholar: provides ‗Cited by‘ information 28/11/2013 UCC Library Introduction to Bibliometrics
  • 14. Example: Number of times cited Article Title: Author(s): Source: Published: 1. 2. 3. 4. 5. 6. 7. 8. 9. 10 28/11/2013 Identification of pancreatic cancer stem cells Li CW, Heidt DG, Dalerba P, et al. CANCER RESEARCH 67 (3): 1030-1037 FEB 1 2007 Go to Web of Knowledge Pick ‘Web of Science’ Enter the Title to locate article Note Number of ‘Times Cited’ Go to Google Scholar Type article title in box Note ‘Cited by -’ Go to Scopus Enter title to locate article Note number ‘Citations’ UCC Library Introduction to Bibliometrics
  • 15. Example: Number of times cited Article Title: Author(s): Source: Published: 1. 2. 3. 4. 5. 6. 7. 8. 9. 10 28/11/2013 Identity, politics and conflict in dockland development in Cork, Ireland: European Capital of Culture 2005 O’Callaghan, C. and Linehan, D. Cities 24(4): 311-323 2007 Go to Web of Knowledge Pick ‘Web of Science’ Enter the Title to locate article Note Number of ‘Times Cited’ Go to Google Scholar Type article title in box Note ‘Cited by -’ Go to Scopus Enter title to locate article Note number ‘Citations’ UCC Library Introduction to Bibliometrics
  • 16. Exercise: Mullally B.J. et al. (2008) ‗Prevalence of smoking among bar workers prior to the Republic of Ireland smokefree workplace legislation‘, Irish Journal of Medical Science, 177(4): 309-16 • How many times has this article been cited in Web of Science, Scopus and Google Scholar? 28/11/2013 UCC Library Introduction to Bibliometrics
  • 17. Citation Impact expanded • Individual Researchers (promotions, funding etc.): - Cited reference search: Web of Science - H-index: -highest number of papers a researcher has that have each received at least that number of citations e.g. someone with h-index 10 has written 10 papers that have each had at least 10 citations. Where to find it (3 different versions): - Web of Science: ‘Author Finder’ and ‘Create Citation Report’. - Scopus: ‘Author Search’ and ‘Citation overview’ N.B. 1996 on - Publish or Perish and Google Scholar (Humanities?) [In each case: difficult to identify all of a researcher’s work] 28/11/2013 UCC Library Introduction to Bibliometrics
  • 18. Example: • Find a H-index for a researcher Heidt, D.G. working in the field of Biomedicine Use: • • • • Web of Science Scopus Google Scholar ‗My citations‘ Publish or Perish http://www.harzing.com/pop.htm N.B. Heidt D v Heidt DG (ResearcherID, ORCID can help) 28/11/2013 UCC Library Introduction to Bibliometrics
  • 19. Citation Impact expanded • An Department/School, Institution, Region, Country: o Web of Science- ‗Organization Enhanced‘ search: ‘University College Cork’ Select from the index o Essential Science Indicators (WOK): NATL UNIV IRELAND UNIV COLL CORK o Scopus – Affiliation search o Scimago Journal & Country Rank o InCites: see here, institutional performance, subject areas, comparison, collaboration N.B. conference papers and monographs are under-represented 28/11/2013 UCC Library Introduction to Bibliometrics
  • 20. University rankings • Shanghai Rankings: uses Web of Science • Times Higher Education uses Web of Science (normalised citation impact 32.5%) • QS Rankings uses Scopus (switched in 2007) 20% • HEEACT: uses Web of Science • Leiden Ranking WoS data only • U-Multirank: a new European ranking initiative • Other metrics: WoS docs per academic, relative citation impact etc. 28/11/2013 UCC Library Introduction to Bibliometrics
  • 21. Journal Impact The average number of citations to papers within a particular journal (journal performance metrics) Why useful? - Deciding where to publish to maximise impact - Librarians in managing acquisitions/subscriptions Where are the metrics? 2 main sources: - Web of Knowledge: ‘Journal Citation Reports’ - Scopus and Scimago see also ‗Publish or Perish‘ and Google Scholar 28/11/2013 UCC Library Introduction to Bibliometrics
  • 22. Journal Impact Factor ‘Journal Citation Reports’ (Journal Performance Metrics) Part Thomson Reuter’s Web of Knowledge (the original) • N.B. Journal Impact Factor: average number of times articles from the journal published in past two years have been cited in JCR year • Journal Immediacy Index: average number of times article is cited in the year it is published • Five year Impact Factor: average number of times articles from the journal published in past five years have been cited in JCR year 28/11/2013 UCC Library Introduction to Bibliometrics
  • 23. Journal Impact Scopus and Scimago Scimago Country & Journal Rank Database: (free) journals and country scientific indicators developed from information in ‘Scopus’ • N.B. SJR: ‗Scimago Journal Rank’ reflects prestige of source where such citations come from. Covers a 3 year citation window. • SNIP: ‗Source normalized impact per paper’: corrects for differences in the frequency of citation across research fieldscitations given a weighting based on the total number of citations in that field. N.B. Scopus only 28/11/2013 UCC Library Introduction to Bibliometrics
  • 24. Journal Impact Other metrics • Google Scholar Journal Metrics • Eigenfactor : (free on the web) citations from highly-ranked journals weighted to make a larger contribution to the eigenfactor than those from poorly-ranked journals (similar to SJR) • Publish or Perish: journal h-index - the number of a journal’s papers that have been cited at least ‘h’ times e.g. h-factor 10 means that 10 papers have been cited at least 10 times 28/11/2013 UCC Library Introduction to Bibliometrics
  • 25. Example: Impact Factor 1. What is the impact factor of the Journal ‘The Lancet’? 1. Go to ISI Web of Knowledge 2. Click on ‘Select a database’ 3. Pick ‘Journal Citation Reports’ 4. Choose ‘JCR Science Edition’ and ‘Search for Specific Journal’ 5. Click on ‘Submit’ 6. Type ‘Lancet’ in box and search for ‘Full Journal Title’ 2. What are the highest impact journals for Oncology? 1. Go to ISI Web of Knowledge 2. Click on ‘Select a database’ 3. Pick ‘Journal Citation Reports’ 4. Choose ‘JCR Science Edition’ and ‘View a Group of Journals by Subject Category’ 6. Click on ‘Submit’ 7. Scroll down to ‘Oncology’ 8. Click on ‘Submit’ 9. Sort by Impact factor 28/11/2013 UCC Library Introduction to Bibliometrics
  • 26. Exercise • What is the 2011 impact factor for the journal ‗Applied Linguistics‘? • What is the 2011 impact factor for the journal ‗Cell‘? • What Sociology journal had the highest impact factor for 2011? • What Psychology journal had the highest impact factor for 2011? 28/11/2013 UCC Library Introduction to Bibliometrics
  • 27. Journal Impact Other Sources • Journal Quality List (Harzing) (mainly business-related subjects) • ERA: Excellence in Research for Australia produce a list of ‗eligible publications‘ • Other national systems: e.g. New Zealand, Norway, Poland, Denmark etc. • ERIH: European Reference Index for the Humanities European humanities research • Also : various subject specific lists available 28/11/2013 UCC Library Introduction to Bibliometrics
  • 28. Other Metrics • Microsoft Academic Search • Altmetrics: attempt to track the impact of scholarly works in the social web e.g. iopscience and Plos • Books? Library holdings analysis: OCLC Worldcat ‘Libraries Worldwide’, Book sales, E-book downloads, Book citation index. Also : who published it, peer-review process • Data citation index • Webometrics: quantifying the impact of web sites e.g. http://repositories.webometrics.info/ http://www.webometrics.info/ 28/11/2013 UCC Library Introduction to Bibliometrics
  • 29. How many times cited? • Wakefield, A. J.; Murch, S. H.; Anthony, A. (1998) ‗Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children‘. Lancet, 351(9103): pp 637-41 Note: Citation data is then used to create h-index for authors, impact factor of the journal, metrics for the university etc. etc. 28/11/2013 UCC Library Introduction to Bibliometrics
  • 30. How many times cited? Stapel, DA. And Koomen, W (2001). ―I, we, and the effects of others on me: How self-construal level moderates social comparison effects.‖ Journal of Personality and Social Psychology 80(5): 766-781. Note: Citation data is then used to create h-index for authors, impact factor of the journal, metrics for the university etc. etc. 28/11/2013 UCC Library Introduction to Bibliometrics
  • 31. Bibliometrics- limitations Useful tools, but not a measure of true research quality e.g.: • current popularity of topic & availability of article/journal: increases citations / higher IF • IF (and H-Index) based on average over all articles: underestimates citations of top cited articles, IF exaggerates number of citations of the average article • comparison of impact factors and H-Index across different fields is invalid e.g. not as relevant for literature (books citing other books) . 80% of humanities citations are not included in WoS. • What about books, book chapters, conference papers, digital humanities etc.? 28/11/2013 UCC Library Introduction to Bibliometrics
  • 32. Bibliometrics- limitations (contd.) • Coverage differs in each resource (Web of Science, Scopus, Google Scholar); none is complete and reliable e.g. for IF only WoS database journals used; undercounts number of citations from other journals e.g. ‘less-developed’ countries, other languages, national & niche journals etc. • Why have the papers been cited? e.g. Review articles, Wakefield • Over 50% of papers are never cited, does this mean they are all if inferior quality? • Takes time for papers to collect citations, especially in some fields • Culture of systematic review and comprehensive citing not established in every field of studies • Question of self-cites, cronyism (reciprocal citing), IF manipulation 28/11/2013 UCC Library Introduction to Bibliometrics
  • 33. Bibliometrics- limitations (contd.) • H-index is a poor metric for young researchers • H-index underestimates the importance of seminal articles • Pressure on academics to publish may lead to a drop in quality, ‘over publication’, implications for peer-review system? • Google metrics are not yet reliable enough in isolation (what dataset is used? lack of quality control: citations from unpublished sources, web sites, early versions of articles etc.) • Pressure on Library budgets – subscribe to all the ‘top’ journals • Citation patterns are becoming increasingly dispersed, lessening the importance of the impact factors 28/11/2013 UCC Library Introduction to Bibliometrics
  • 34. Impact Factor- limitation • ‘Like nuclear energy, the impact factor is a mixed blessing.’ • ‘The use of journal impacts in evaluating individuals has its inherent dangers. In an ideal world, evaluators would read each article and make personal judgments.’ Garfield, E. (2006) ‘The History and Meaning of the Impact Factor’. JAMA, 295(1): pp 90-93 [Garfield invented scholarly citation in the 1960’s] 28/11/2013 UCC Library Introduction to Bibliometrics
  • 35. H-index - limitations ‘Obviously a single number can never give more than a rough approximation to an individual’s multifaceted profile, and many other factors should be considered in combination in evaluating an individual.’ Hirsch, J.E. (2005) ‘An index to quantify an individual's scientific research output’. arXiv:physics/0508025 v5 29 Sep 2006 (Hirsch proposed the H-index in 2005) 28/11/2013 UCC Library Introduction to Bibliometrics
  • 36. Remember: • No substitute for human judgment (be it a paper, researcher or unit that is being evaluated) assess each on its own merits • Metrics must be used appropriately • Don‘t apply in all cases • Must be considered in combination with other indicators But: when used appropriately metrics can assist and supplement peer review, while awareness of metrics can assist researchers seeking to increase research impact 28/11/2013 UCC Library Introduction to Bibliometrics
  • 37. Publishing strategies to improve citation performance • Good quality research …but it‘s not the only driver of citations • Citation patterns – Field dependent – Time dependent – Article dependent – Author/reader dependent – Availability • Citation motivations UCC Library Introduction to Bibliometrics – Why do researchers cite papers? 28/11/2013
  • 38. Journal Selection • Journals with high impact factors • Journals indexed by Web of Science and/or Scopus • Less specialized journals • English language journals • Review / special issue publishing
  • 39. Article content • Titles, abstracts, keywords - 10 times more likely to be read than article itself Indexing databases Page ranking algorithms Long titles • References – Number of references – Highly cited papers • Co-authors – Number of co-authors – Highly cited co-authors • Quality – Avoid minimum publishable unit
  • 40. Your name • Use a middle initial if possible. Be consistent with the use of variants of your name – e.g. Smith, John A. / Smith J. A. and not Smith, John /Smith J. • Do you have a publishing doppelgänger? – Check your name against authors in Google Scholar, Pub Med, Web of Science, Scopus • Be pedantic about your name and affiliation on publications – Published information sent to databases…GIGO
  • 41. Your name…a solution on the horizon • Open Researcher & Contributor ID – International collaboration between Funders, Research – – – – Institutions and Publishers to solve author ambiguity issues Unique author identity registry Data exchange possible between ResearcherID, LinkedIn and Scopus Author Profiles No affiliation necessary and no cost to author Takes 30 seconds to register at http://orcid.org 28/11/2013 UCC Library Introduction to Bibliometrics
  • 42. 28/11/2013 UCC Library Introduction to Bibliometrics
  • 43. Your public research identity • Give full and proper references to your work – Make it easy to find – Make it accessible i.e. link to full text in CORA • Claim your publications > accurate author identification – – – – – IRIS Web of Science through Researcher ID Scopus Google Scholar through My Citations ResearchGate / Academia.edu / Mendeley / LinkedIn
  • 44. 28/11/2013 UCC Library Introduction to Bibliometrics
  • 45. Availability of your publications • Is your work or a sample of your work available online? ‗Make it frictionless‘ ―The diffusion of research — publication of working papers, as well as conference and workshop presentations — has a first-order positive impact on the citation rate.‖ • If work is not accessible, will it be cited? • Is your work behind a pay wall? • Archive your publications in an open access repository e.g. CORA, UCC institutional repository
  • 46. Archive your work in CORA • Open Access Citation advantage – Archive early and often leads to increased citations to your work Engineering Clinical Medicine Social Sciences Gargouri & Harnad, 2010: http://eprints.ecs.soton.ac.uk/18493/
  • 47. Open Access Citation Advantage Size of OA citation advantage when found (and where explicitly stated by discipline) % increase in citations with Open Access Physics/astronomy Mathematics Electrical engineering Computer science Political science 170 to 580 35 to 91 51 157 86 Philosophy Medicine Communications studies (IT) Agricultural sciences 45 300 to 450 200 200 to 600 Swan, Alma (2010) The Open Access citation advantage: Studies and results to date. http://eprints.soton.ac.uk/268516/ A summary of studies 27 found an Open Access citation advantage, while 4 reported no OA citation advantage
  • 48. Promote your publications • Use social media such as blogs, twitter and Facebook to highlight your research – Develop your online network – A distributed online identity • Liaise with UCC Research Support Services & Office of Media and Public Relations to arrange the media promotion of your work • Monitor impact – Monitor impact > Altmetrics Real Time > Citation alerts longer term – Can tweets prediction citations?
  • 49. Monitor impact of social media 28/11/2013 UCC Library Introduction to Bibliometrics
  • 50. Altmetrics • Alternative metrics – Download statistics, social media mentions, mainstream media mentions – Timeliness – Evidence of reach beyond academia – Measures for a diversity of research outputs • Tools including – Altmetric , ImpactStory, Plum Analytics
  • 51. 28/11/2013 UCC Library Introduction to Bibliometrics
  • 52. 28/11/2013 UCC Library Introduction to Bibliometrics
  • 53. Altmetrics Altmetric bookmarklet for Chrome, Firefox and Safari http://www.altmetric.com/bookmarklet.php 28/11/2013 UCC Library Introduction to Bibliometrics
  • 54. http://impactstory.org/CarlBoettiger 28/11/2013 UCC Library Introduction to Bibliometrics
  • 55. Figure 2. Article-level metrics used by PLOS in August 2013 and their categories. Fenner M (2013) What Can Article-Level Metrics Do for You?. PLoS Biol 11(10): e1001687. doi:10.1371/journal.pbio.1001687 http://www.plosbiology.org/article/info:doi/10.1371/journal.pbio.1001687
  • 56. Earlier Web usage statistics as predictors of later citation impact Brody, T., Harnad, S. and Carr, L. (2006), Earlier Web usage statistics as predictors of later citation impact. J. Am. Soc. Inf. Sci., 57: 1060–1072. doi: 10.1002/asi.20373
  • 57. Affiliation • Ensuring that the ‘University College Cork’ identifier is used in all research publications – ensures credit for your publications and citations is accumulated by UCC • National Citation Report project – UCC was found to have more than 1,600+ name variants in Web of Science
  • 58. Aggregation of Citations per Publication across all 22 Fields Web of Science 2002 – 2012 (February 2013) 10 9 8 7 6 5 4 3 2 1 0 2002-2006 2003-2007 2004-2008 2005-2009 2006-2010 2007-2011 2008-2012 TCD 6.17 6.37 6.94 7.44 7.84 8.34 8.85 UCD 4.39 5.06 5.83 6.22 6.59 6.54 6.72 UCC 4.49 4.29 4.65 5.12 5.43 6.06 6.54 NUIG 3.66 3.98 4.12 4.42 5.16 5.31 6.21 DCU 3.05 3.2 3.59 4.04 4.78 5.2 5.22 DIT 3.79 3.63 3.6 3.84 4.36 4.97 5.03 UL 2.03 2.36 2.5 3.07 3.06 3.34 3.75 Prepared by Deirdre Kearney, Research Support Services, UCC
  • 59. • Slides: • Further reading • Database Tutorials: – Web of Science – Journal Citation Reports – Scopus Tutorials • MyRi: Measuring your Research Impact http://www.ndlr.ie/myri/ 28/11/2013 UCC Library Introduction to Bibliometrics
  • 60. Appendix 1: Criteria for Journal Inclusion Scopus: Web Of Science: see http://ip-science.thomsonreuters.com/mjl/
  • 61. Appendix 2: Additional tools for Computer Science 28/11/2013 UCC Library Introduction to Bibliometrics
  • 62. Appendix 3: Further metrics for individuals • • • • • • • Egghe's g-index Zhang's e-index Contemporary h-index Age-weighted citation rate (AWCR) and AW-index Individual h-index (original) Individual h-index (PoP variation) Multi-authored h-index • See: http://www.harzing.com/pop.htm#metrics 28/11/2013 UCC Library Introduction to Bibliometrics
  • 63. Appendix 4: • REF: Assessment framework and guidance on submissions See in particular REF Section 131 ‘Citation data’ 28/11/2013 UCC Library Introduction to Bibliometrics