Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Measuring Research Impact on the Web


Published on

Beth Bernhardt (speaker), Iain Hrynaszkiewicz (speaker)

Published in: Technology
  • Be the first to comment

Measuring Research Impact on the Web

  1. 1. MEASURING RESEARCH IMPACT ON THE WEB Charleston Conference, November 2013 Iain Hrynaszkiewicz Outreach Director, Faculty of 1000 @iainh_z
  2. 2. SOME PROBLEMS WITH SCHOLARLY COMMUNICATION 1. 2. 3. 4. Individual papers and scientists judged on journal-based metrics Closed, pre-publication peer review Lack of access to original research – and data Lack of credit for many of scientists’ contributions The Seer of Science Publishing Science 4 October 2013: Vol. 342 no. 6154 pp. 66-67 DOI: 10.1126/science.342.6154.66
  3. 3. SOME PROBLEMS WITH SCHOLARLY COMMUNICATION #1 “I am sick of impact factors and so is science.” Prof Stephen Curry, Imperial College, Aug 2012 2/08/13/sick-of-impact-factors/ “[Journal Impact Factor is] a poor indicator of citations to specific papers or of the future performance of individual researchers” Nature Materials 12, 89 (2013) “Citations are heavily gamed and are painfully slow to accumulate, and overlook increasingly important societal and clinical impacts.” Priem et al., PLoS ONE 7(11): e48753 “The widely held notion that high-impact publications determine who gets academic jobs, grants and tenure is wrong.” Dr Michael Eisen, UC Berkeley, Feb 2012
  4. 4. Themes of DORA recommendations: 1.Eliminate the use of journal-based metrics in funding, appointment, and promotion considerations 2.Assess research on its own merits not the journal 3.Capitalize on the opportunities provided by online publication such as exploring new indicators of significance and impact DORA has been signed by >9000 individuals and nearly 400 organisations. Original signatories included American Association for the Advancement of Science (AAAS), American Society for Cell Biology, Howard Hughes Medical Institute, Faculty of 1000, Public Library of Science
  5. 5. IMPACT FACTORS AND CITATION METRICS Advantages Disadvantages Reproducible Slow – delay of up to 2 years Transparent calculation (kind of) Data not publicly available Curated – some human filtering Can be manipulated Predictor of journal quality Poor predictor of paper and researcher quality
  6. 6. I, Cawi 2001 [CC-BY-SA-2.5 (], via Wikimedia Commons
  7. 7. “ALTERNATIVE” METRICS Source: Wikipedia,
  8. 8. ALTMETRICS – WHY ARE THEY IMPORTANT? “For all new grant applications from 14 January, the US National Science Foundation (NSF) asks a principal investigator to list his or her research “products” rather than “publications” in the biographical sketch section. This means that, according to the NSF, a scientist's worth is not dependent solely on publications. Data sets, software and other non-traditional research products will count too.” Heather Piwowar, UBC, in Nature Jan 2013
  9. 9. ALTMETRICS – WHY ARE THEY IMPORTANT? “Metrics --- such as the number of article citations, your h-index or others (such as those available at --- can be useful in making the case that the publication or scholarly work was significant. The Faculty Promotions Committee discourages the use of journal-based metrics (such as journal impact factors), since it is the quality and importance of the research contribution itself that is the key.” Excerpt from UC Denver promotions handbook guide to dossier preparation
  12. 12. Source:
  13. 13. Source:
  14. 14. (ALT)METRICS – ADVANTAGES AND DISADVANTAGES Advantages Disadvantages Fast – data available immediately Heterogeneity across tools Transparent (mostly) Can lack context/meaning Lots of open data and tools available Can be manipulated All research products tracked Not always reproducible (ephemeral) Much broader picture of impact Emergence and disappearance of tools
  15. 15. WHAT’S OFTEN MISSING? By Taco Hoekwater; Wolfgang Schuster; Xan; svg from Lumu [CC-BY-SA-3.0 (], via Wikimedia Commons
  16. 16. HOW F1000PRIME WORKS • Faculty includes over 5000 peer-nominated scientists and clinical researchers and ~5000 Associates • Faculty Members select, rate and comment on the most interesting and important research articles (2-3% of the life science literature) from ~3,700 journals • Assigns one of three positive ratings: Exceptional (3 stars), Very Good (2 stars) or Good (1 star) • Text also serves as a short, expert, plain English recommendation written for a global readership • Adds relevant classifications (e.g. changes clinical practice) • Publishes about 1500 recommendations per month (>143,000 published to date)
  18. 18. F1000, ALTMETRICS AND RESEARCH ASSESSMENT/IMPACT The Wellcome Trust published in PLOS ONE an analysis of F1000Prime, finding that expert review highlights important papers that bibliometric indicators alone would miss ( A 2009 study by the MRC concluded that F1000Prime recommendations are an indicator of future citation impact ( Collection of studies of F1000Prime: Also, studies have shown that tweets and number of readers on Mendeley predict citations. How much does all this matter?
  20. 20. IS THIS IMPACT? Source:
  21. 21. THIS IS IMPACT Source:
  22. 22. SUMMING UP • Impact factors are not good for assessing individual papers and individuals • The internet has given us a wealth of new data and tools to give a broader picture of the impact of research • We are still trying to understand the meaning of many new metrics – we need data with context • Some “alternative” metrics are beginning to influence research assessment • All metrics – citation and non-citation metrics – have advantages and disadvantages