Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Similar to What you need to know about research assessment
Similar to What you need to know about research assessment (20)
Recently uploaded
Recently uploaded (20)
What you need to know about research assessment
- 1. Research Assessment: What you need to know. A Presentation by Ian Gibson, Collections Librarian James A. Gibson Library
- 2. It’s all his fault Image from http://vw.indiana.edu/places- images/authors/eugene-garfield.jpg
- 3. The Journal Impact Factor •𝐼𝐹 = 𝐶𝑖𝑡𝑎𝑡𝑖𝑜𝑛𝑠 𝐶𝑖𝑡𝑎𝑏𝑙𝑒 𝐴𝑟𝑡𝑖𝑐𝑙𝑒𝑠 • Where – Citations = citations in the current year to articles published in the 2 previous years – Citable Articles = the number of articles published in the 2 previous years (in theory)
- 4. if you use impact factors you are statistically illiterate1 1Stephen Curry, Sick of Impact Factors http://occamstypewriter.org/scurry/2012/08/13/sick-of-impact- factors Photo from http://www.theproteinwrangler.com/wp-content/uploads/2011/10/scurry.jpg
- 5. Alternatives to Impact Factors? • 5 Year Impact Factor • Eigenfactor • Source-Normalized Impact per Paper (SNIP) • SCImago Journal Rank (SJR) • Google Journal Rank
- 6. What Metrics Exist to Judge Individuals? Let me hook you up with my h-index Jorge Hirsch http://www-physics.ucsd.edu/fac_staff/fac_profile/faculty_description.php?person_id=99
- 7. h-Index • h = n papers that have been cited at least n times.
- 8. We have a problem.
- 9. “Citation‐based statistics can play a role in the assessment of research, provided they are used properly, interpreted with caution, and make up only part of the process. Citations provide information about journals, papers, and people. We don't want to hide that information; we want to illuminate it.” - Joint Committee on Quantitative Assessment of Research, International Mathematics Union in cooperation with the International Council of Industrial and Applied Mathematics (ICIAM) and the Institute of Mathematical Statistics (IMS) http://www.mathunion.org/fileadmin/IMU/Report/CitationStatistics.pdf
- 10. Declaration on Research Assessment General Recommendation 1. Do not use journal-based metrics, such as Journal Impact Factors, as a surrogate measure of the quality of individual research articles, to assess an individual scientist's contributions, or in hiring, promotion, or funding decisions. http://am.ascb.org/dora/
- 11. Altmetrics • http://altmetrics.org/manifesto • Putting the focus at the article level • Capturing broader impact: – “Hallway chats” (e.g. twitter, blog postings) – Those papers that you don’t cite but keep in that drawer in your office (e.g. saved in a reference manager like Zotero or Mendeley) – The papers that you read just for general interest (e.g. download statistics)
- 12. PLOS Article Level Metrics • http://www.plosone.org/article/metrics/info% 3Adoi%2F10.1371%2Fjournal.pone.0030366 • http://www.plosone.org/article/metrics/info% 3Adoi%2F10.1371%2Fjournal.pone.0081461
- 13. Impact Story • http://impactstory.org/heatherpiwowar • http://impactstory.org/IanGibson • This works best if you have a ORCID or ResearcherID
- 15. What do Altmetrics tell us? http://mikuru.net/wp-content/uploads/2011/12/under-construction.gif
- 16. Conclusion
- 17. Questions?
- 18. THANK YOU Ian Gibson Collections Librarian igibson@brocku.ca
- 19. Bibliography • Brembs, B., Button, K., & Munafò, M. (2013). Deep impact: unintended consequences of journal rank. Frontiers in Human Neuroscience, 7, 291. doi:10.3389/fnhum.2013.00291 • Cantrill, S. (2012, June 30). Inside our Impact Factor. The Sceptical Chymist. Retrieved fromhttp://blogs.nature.com/thescepticalchymist/2012/06/inside-our-impact-factor.html • Colquhoun, D., & Plested, A. (2014, January 16). Why you should ignore altmetrics and other bibliometric nightmares. DC’s Improbable Science. Retrieved April 2, 2014, from http://www.dcscience.net/?p=6369 • Costas, R., Zahedi, Z., & Wouters, P. F. (2014, January 17). Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective. CWTS-WP-2014-001. External research report. Retrieved April 2, 2014, fromhttps://openaccess.leidenuniv.nl/handle/1887/23041 • Curry, S. (2012, August 13). Sick of Impact Factors. Reciprocal Space. Retrieved April 7, 2014, from http://occamstypewriter.org/scurry/2012/08/13/sick-of-impact-factors/ • Fenner, M. (2013). What Can Article-Level Metrics Do for You? PLoS Biol,11(10), e1001687. doi:10.1371/journal.pbio.1001687 • Garfield, E. (1955). Citation Indexes for Science. Science, 122(3159), 108–111. Retrieved from http://www.jstor.org/stable/1749965 • Garfield, E. (1964). “Science Citation Index”-A New Dimension in Indexing.Science, 144(3619), 649–654. Retrieved fromhttp://www.jstor.org/stable/1712875 • Garfield, E. (2005). The Agony and the Ecstasy— The History and Meaning of the Journal Impact Factor. Presented at the International Congress on Peer Review And Biomedical Publication, Chicago. Retrieved fromhttp://garfield.library.upenn.edu/papers/jifchicago2005.pdf • Rossner, M., Epps, H. V., & Hill, E. (2007). Show me the data. The Journal of Cell Biology, 179(6), 1091–1092. doi:10.1083/jcb.200711140 • Seglen, P. O. (1992). The skewness of science. Journal of the American Society for Information Science, 43(9), 628–638. doi:10.1002/(SICI)1097- 4571(199210)43:9<628::AID-ASI5>3.0.CO;2-0 • Sud, P., & Thelwall, M. (2014). Evaluating altmetrics. Scientometrics, 98(2), 1131–1143. doi:10.1007/s11192-013-1117-2 • The PLoS Medicine Editors. (2006). The Impact Factor Game. PLoS Medicine, 3(6), e291. doi:10.1371/journal.pmed.0030291 • Thelwall, M., Haustein, S., Lariviere, V., & Sugimoto, C. R. (2013). Do Altmetrics Work? Twitter and Ten Other Social Web Services. PLoS ONE, 8(5), e64841. doi:10.1371/journal.pone.0064841 • Vanclay, J. K. (2012). Impact factor: outdated artefact or stepping-stone to journal certification? Scientometrics, 92(2), 211–238. doi:10.1007/s11192- 011-0561-0
Editor's Notes
- This presentation is about awareness – research assessment is not a topic that most of you will have reason to encounter in your day to day life.
- This is Eugene and it’s all his fault In 1955 my boy here decided that “science” was too difficult to follow given the limitations of subject indexing. What was needed was a means of easily following ideas. Law provided the example of a functioning citation based indexing system in Sheppard’s Citations. In 1961 the Genetics Citation Index was launched. This eventually grew into Science Citation Index and into the Web of Science Most importantly it made counting citations easy and desirable; small step in the quantification of quality Introduced a few issues Transcription errors actually make counting citations much more difficult Which citations actually count? There is no comprehensive citation index – not an issue if we don’t care about new journals; patents/grey literature; book coverage is spotty at best (though TR and Google are trying to rectify this) Context of citation – citation does not imply quality; citation does not imply agreement; citation does not imply importance It became obvious rather quickly that a number of metrics could be created using these citation counts to judge quickly the importance of articles, journals, and people. Garfield initially mused in 1955 that one could create an “impact factor” to judge the impact of individual articles. Both the initial Genetics Citation Index concept and the larger Science Citation Index were based around author… however if you switch the focus from author to journal title…
- Initially Garfield and Sher developed the JIF to determine what journals to include in the Science Citation Index – no idea that it would become such a controversial figure (see http://garfield.library.upenn.edu/papers/jifchicago2005.pdf) When the numbers were released publicly in the 70s the idea was that it was a tool for libraries to help decide how to allocate our journal subscriptions (i.e. which titles formed a “core” collection) The formula looks straight forward enough but when you start digging into it one sees it as a deeply flawed metric For instance what citations are we counting? Well all the citations obviously except that we’re not really – it’s all the citations coming from WoS indexed content – that list is getting more comprehensive with the inclusion of books, conferences and data recently but we’re still not getting patents, gov’t reports, grey literature, preprints, etc. But there isn’t any distinction between citations to news, opinion, and review pieces and cites of research articles – doesn’t seem like a big deal but… What counts as a citable article? PLOS Medicine published an article detailing the negotiations around what would count as citable when they go their first impact factor (http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0030291). Journals have an obvious interest in keeping the denominator low and the numerator high For an interesting look at what happens when you manipulate see Stuart Cantrill’s analysis of how publishing review aritcles inflated the impact factor of Nature Chemistry from 16 up to 20. There are also questions about reproducibility – Rockefeller University Press tried to reproduce their impact factors and the IFs of their competitors first using just WoS then using data provided by TR – despite multiple attempts they could not find the magic numbers to make the calculations work (see http://jcb.rupress.org/content/179/6/1091.full)
- This is Stephen Curry a BioChemist @ Imperial College London he wrote a blog post talking about being sick of the use of impact factors as some kind of signifier of quality for individuals. This post mentioned most of the major critiques of IF Citation distribution is skewed Per Seglens showed in 1992 that 15% of the articles in a typical journal are responsible for ½ total citations – so journals are being rewarded for publishing a few good papers in a sea of mediocrity How it is being used in grant aps/demanded by granting bodies even though it says nothing about the individuals actual contributions to that particular journal Highlights a paper by Vanclay (http://arxiv.org/abs/1201.3076) that points out issues around a short time frame; doesn’t account for different citation practices; is selective, etc. One of the interesting consequences of this calculation is that IF should increase every year – for the simple reason that every year more and more stuff is being published and therefore more and more citations
- So we know that IF isn’t great what are the alternatives Eigenfactor measures the influence of journals using a method similar to Google’s PageRank algorithm – the inventors claim that by using a longer timeframe 5 years and their enhanced algorithm they can eliminate many of the problems of the JIF – Basis in Web of Science but with additional data put out by University of Washington Researchers SNIP based on Scopus data + 3 previous years – aims to correct for variations in citing behaviour between disciplines – claims better representation b/c Scopus indexes more journals (though there is some debate about how comprehensively Scopus actually covers those journals…) put out by CWTS @ U Liedan SJR also based on Scopus data – uses the same ideas as Eigenfactor/Page Rank – citations weighted by the quality of citing source – again weighting is used to correct for differing citation practices between fields Google Journal Rank – based on the H-Index – the h5 index calculates the h-index based on a 5 year window THESE ARE ALL JOURNAL BASED METRICS Brembs, Button & Munafo (http://journal.frontiersin.org/Journal/10.3389/fnhum.2013.00291/full) suggest that journal rank (and in fact journals themselves) need to be done away (1) journal rank is a weak to moderate predictor of utility and perceived importance; (2) journal rank is a moderate to strong predictor of both intentional and unintentional scientific unreliability; (3) journal rank is expensive, delays science and frustrates researchers; and, (4) journal rank as established by IF violates even the most basic scientific standards, but predicts subjective judgments of journal quality
- Anyone see a problem with this? It rewards a long career or a prolific career It doesn’t reward pathbreaking ideas and papers that get a lot of citations It doesn’t account for multiple authors Rewards prolific authors (e.g. if you’ve only published 1 paper the highest your h-index could be is 1)
- Regardless of whether you feel we actually can We have a proliferation of metrics all being built on the same creeky foundationg Citation based metrics have started to creep into disciplines well outside the scientific core (e.g. Business and education)
- - 2008 report on citation statistics found that citation statics though “easy” were not well understood “Numbers are not inherently superior to sound judgement”
- -the declaration on research assessment was formulated by the American Society for Cell Biology (in conjunction with others) to shed light on the wide spread practice of judging grant applications and T&P decisions on where the applicants had published instead of what they published.
- Altmetrics sprang from a parallel movement to offer some kind of article level metrics or at least article level data on citations & downloads. Altmetrics is trying to capture impact that research has in and out of academia Research is broadly defined (right now anything that you can give a permanent identifier to… blog posts, git hub repositories, etc. Trying to capture not just use but also some of the nuance behind the use. The important point is that a lot of interaction with an article has to happen before it is cited and previously we couldn’t capture much of that process
- PLOS was one of the first publishers to give metrics publically displayed for each article Useful categorization: Viewed, Cited, Saved, Discussed, and Recommended “All PLOS Biology articles are viewed and downloaded, and almost all of them (all research articles and nearly all front matter) will be cited sooner or later. Almost all of them will also be bookmarked in online reference managers, such as Mendeley, but the percentage of articles that are discussed online is much smaller. Some of these percentages are time dependent; the use of social media discussion platforms, such as Twitter and Facebook for example, has increased in recent years (93% of PLOS Biology research articles published since June 2012 have been discussed on Twitter, and 63% mentioned on Facebook). These are the locations where most of the online discussion around published articles currently seems to take place; the percentage of papers with comments on the PLOS website or that have science blog posts written about them is much smaller.” What Can Article-Level Metrics Do for You? http://dx.doi.org/10.1371/journal.pbio.1001687 PLOS has analyzed this data and come up with some interesting things: HTML views seem to be driven mostly by social sharing PDF views have no correlation to social sharing Most articles have 4:1 HTML to PDF views but as that ratio increases there is a very strong correlation to social sharing. Suggests that social sharing is a good indicator of public engagement They don’t see much correlation between ppl saving in a citation manager (like Mendeley) & citation counts They find that meaningful citation numbers don’t accumulate for 2-5 years even with the typically high citation culture of the life sciences
- Impact story is a not for profit open source project that seeks to contextualize the raw numbers that you might find at PLOS Right now only limited data is available. Can input DOIs, URLs, github username + repository name or use your ORCID Show final report in link then show input screen. Can use your ORCID 0000-0001-9154-8247
- Similar to Impact story but right now mostly focused on NPG products Can use their free bookmarklet which will automatically provide a report
- Altmetrics are quite new – so it’s still under development Working paper by Costas, Zahedi and Wouters from University of Leiden showed that many articles have very little altmetric activity H&SS along w/ biomed has the most. Much research still to be done Thelwell et al. (http://dx.doi.org/10.1371/journal.pone.0064841) in PLOSOne report that significant altmetric activity (for certain services) does correlate to higher citation but that use of some altmetric services are so seldome used it might not actually be worthwhile Sud and Thelwell (http://dx.doi.org/10.1007/s11192-013-1117-2) recommend methods for evaluation altmetrics David Colquhoun writes on his blog that it’s best to ignore altmetrics and other bibliometric nightmares “Altmetrics are numbers generated by people who don’t understand research, for people who don’t understand research. People who read papers and understand research just don’t need them and should shun them.”
- Despite what I’ve said today I’m not below keeping tabs on my paper Other key takeaways – Citations are difficult to deal with and any measure built on them must be taken with a grain of salt. Impact Factors are not the be all and end all and are in fact highly flawed; they have their place but you don’t have one. Many bibliometric scores are at the journal level