Kinga Hosszu
Outreach Director, Faculty of 1000
Kinga.hosszu@f1000.com
@KingaNYC
http://f1000.com
MEASURING RESEARCH IMPAC...
THE FACULTY OF 1000
The Faculty of 1000 is the publisher of unique scientific
literature services that support and inform ...
“[Journal Impact Factor is] a poor
indicator of citations to specific
papers or of the future performance
of individual re...
IMPACT FACTORS AND CITATION METRICS
Advantages Disadvantages
Reproducible Slow – delay of up to 2 years
Transparent calcul...
I, Cawi 2001 [CC-BY-SA-2.5 (http://creativecommons.org/licenses/by-sa/2.5)], via Wikimedia Commons
http://commons.wikimedi...
TOOLS FOR SHARING RESEARCH CONTRIBUTIONS...
Sum of total impact > Sum of research citations
Themes of DORA recommendations:
1.Eliminate the use of journal-based metrics in funding, appointment, and promotions
2.Ass...
ALTMETRICS AND ARTICLE-LEVEL METRICS – DISAMBIGUATION
Article level metrics (ALMs): established metrics that measure the
o...
ALTMETRICS: WHY ARE THEY IMPORTANT? - FUNDING
“For all new grant applications from 14 January, the US
National Science Fou...
ALTMETRICS: WHY ARE THEY IMPORTANT? - PROMOTIONS
“Metrics --- such as the number of article citations, your h-index or
oth...
Scholarly Public
Recommendations F1000 Popular Press
Citations Traditional citations Wikipedia
Discussions Scholarly blogs...
Scholarly Public
Recommendations F1000 Popular Press
Citations Traditional citations Wikipedia
Discussions Scholarly blogs...
SOME TOOLS TO INTERACT WITH RESEARCH CONTRIBUTIONS
Source: http://impactstory.org/collection/v7b25c
ALTMETRICS TOOLS
Source: http://www.altmetric.com/details.php?citation_id=938211
HOW F1000PRIME WORKS
• Faculty of over 5000 peer-nominated scientists and clinical researchers
• Faculty Members select, r...
F1000, ALTMETRICS AND RESEARCH IMPACT
Collection of studies of F1000Prime: http://bit.ly/F1000studies
• Published study by...
(ALT)METRICS – ADVANTAGES AND DISADVANTAGES
Advantages Disadvantages
Fast – data available immediately Heterogeneity acros...
SOME PERSPECTIVE. WHAT IS IMPACT?
http://blog.f1000.com/2013/05/20/what-is-impact/
Sutherland WJ, Goulson D, Potts SG, Dicks LV. PLoS ONE 2011; 6(11):e27537
IMPACT - DOWNLOADS
Source: http://www.jmedicalcasereports.com/mostviewed/alltime
IMPACT – LIVES SAVED
Research finds that antibiotic
drug almost halves AIDS-
related death in children
SUMMARY
• All metrics – citation and non-citation metrics – have advantages and
disadvantages
• Impact factors are not goo...
Kinga Hosszu
Outreach Director, Faculty of 1000
http://f1000.com
How to measure research impact on the web
How to measure research impact on the web
How to measure research impact on the web
How to measure research impact on the web
Upcoming SlideShare
Loading in …5
×

How to measure research impact on the web

342 views

Published on

This presentation explains how research impact measurement has changed with the advent of the internet, and provides examples of how impact can be measurement using several online tools.

Published in: Science
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
342
On SlideShare
0
From Embeds
0
Number of Embeds
24
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

How to measure research impact on the web

  1. 1. Kinga Hosszu Outreach Director, Faculty of 1000 Kinga.hosszu@f1000.com @KingaNYC http://f1000.com MEASURING RESEARCH IMPACT ON THE WEB
  2. 2. THE FACULTY OF 1000 The Faculty of 1000 is the publisher of unique scientific literature services that support and inform the work of life scientists and clinicians worldwide. http://f1000.com/prime http://f1000.com/posters http://f1000research.com
  3. 3. “[Journal Impact Factor is] a poor indicator of citations to specific papers or of the future performance of individual researchers” Nature Materials 12, 89 (2013) http://bit.ly/1jN7Y6a “I am sick of impact factors and so is science.” Prof. Stephen Curry, Imperial College, Aug 2012 http://bit.ly/1onlWjj “The widely held notion that high-impact publications determine who gets academic jobs, grants and tenure is wrong.” Dr. Michael Eisen, UC Berkeley, Feb 2012 http://bit.ly/1kACAwz “Citations are heavily gamed and are painfully slow to accumulate, and overlook increasingly important societal and clinical impacts.” Priem et al., PLoS One 2012 http://bit.ly/1hsIi4c
  4. 4. IMPACT FACTORS AND CITATION METRICS Advantages Disadvantages Reproducible Slow – delay of up to 2 years Transparent calculation (kind of) Data not publicly available Predictor of journal quality Poor predictor of paper and researcher quality
  5. 5. I, Cawi 2001 [CC-BY-SA-2.5 (http://creativecommons.org/licenses/by-sa/2.5)], via Wikimedia Commons http://commons.wikimedia.org/wiki/File:Internet-Sign.jpg
  6. 6. TOOLS FOR SHARING RESEARCH CONTRIBUTIONS... Sum of total impact > Sum of research citations
  7. 7. Themes of DORA recommendations: 1.Eliminate the use of journal-based metrics in funding, appointment, and promotions 2.Assess research on its own merits not the journal 3.Capitalize on opportunities provided by online publication such as exploring new indicators of significance and impact DORA has been signed by >9000 individuals and nearly 400 organizations, including: American Association for the Advancement of Science (AAAS) American Society for Cell Biology Howard Hughes Medical Institute Faculty of 1000 Public Library of Science http://am.ascb.org/dora/
  8. 8. ALTMETRICS AND ARTICLE-LEVEL METRICS – DISAMBIGUATION Article level metrics (ALMs): established metrics that measure the overall performance and reach of published research articles (bibliometrics + altmetrics) Altmetrics: A measure of web-based article usage and reader interaction. Example: how many data and knowledge bases refer to the work, article views, downloads, or mentions in social media and news
  9. 9. ALTMETRICS: WHY ARE THEY IMPORTANT? - FUNDING “For all new grant applications from 14 January, the US National Science Foundation (NSF) asks a principal investigator to list his or her research “products” rather than “publications” in the biographical sketch section. This means that, according to the NSF, a scientist's worth is not dependent solely on publications. Data sets, software and other non-traditional research products will count too.” Heather Piwowar, UBC, in Nature Jan 2013 http://www.nature.com/nature/journal/v493/n7431/full/49315 9a.html
  10. 10. ALTMETRICS: WHY ARE THEY IMPORTANT? - PROMOTIONS “Metrics --- such as the number of article citations, your h-index or others (such as those available at ImpactStory.org) --- can be useful in making the case that the publication or scholarly work was significant. The Faculty Promotions Committee discourages the use of journal-based metrics (such as journal impact factors), since it is the quality and importance of the research contribution itself that is the key.” Excerpt from UC Denver promotions handbook guide to dossier preparation
  11. 11. Scholarly Public Recommendations F1000 Popular Press Citations Traditional citations Wikipedia Discussions Scholarly blogs Blogs, Twitter Saved Mendeley, CiteULike Google Scholar Views/Downloads PDF views HTML views CITATIONS CAN MEASURE…
  12. 12. Scholarly Public Recommendations F1000 Popular Press Citations Traditional citations Wikipedia Discussions Scholarly blogs Blogs, Twitter Saved Mendeley, CiteULike Google Scholar Views/Downloads PDF views HTML views ALTMETRICS CAN MEASURE…
  13. 13. SOME TOOLS TO INTERACT WITH RESEARCH CONTRIBUTIONS
  14. 14. Source: http://impactstory.org/collection/v7b25c
  15. 15. ALTMETRICS TOOLS Source: http://www.altmetric.com/details.php?citation_id=938211
  16. 16. HOW F1000PRIME WORKS • Faculty of over 5000 peer-nominated scientists and clinical researchers • Faculty Members select, rate and comment on the top research • Assigns one of three positive ratings: Exceptional/ Very Good/ Good • Text also serves as a short, expert, recommendation written for a global readership • Publishes about 1500 recommendations per month (>150,000 published to date)
  17. 17. F1000, ALTMETRICS AND RESEARCH IMPACT Collection of studies of F1000Prime: http://bit.ly/F1000studies • Published study by the Welcome Trust: F1000Prime highlights important papers that bibliometric indicators alone would miss • A study by the Medical Research Council: F1000Prime recommendations indicate future citation impact Studies have shown that tweets and number of readers on Mendeley predict citation impact
  18. 18. (ALT)METRICS – ADVANTAGES AND DISADVANTAGES Advantages Disadvantages Fast – data available immediately Heterogeneity across tools All research products tracked Emergence and disappearance of tools Lots of open data and tools available Can be manipulated Much broader picture of impact Can lack context/meaning
  19. 19. SOME PERSPECTIVE. WHAT IS IMPACT? http://blog.f1000.com/2013/05/20/what-is-impact/
  20. 20. Sutherland WJ, Goulson D, Potts SG, Dicks LV. PLoS ONE 2011; 6(11):e27537
  21. 21. IMPACT - DOWNLOADS Source: http://www.jmedicalcasereports.com/mostviewed/alltime
  22. 22. IMPACT – LIVES SAVED Research finds that antibiotic drug almost halves AIDS- related death in children
  23. 23. SUMMARY • All metrics – citation and non-citation metrics – have advantages and disadvantages • Impact factors are not good for assessing individual papers and scientists • The internet has given us a wealth of new data and tools for a broader picture on the impact • Still trying to understand the meaning of new metrics – we need data with context • Some “alternative” metrics are beginning to influence research assessment
  24. 24. Kinga Hosszu Outreach Director, Faculty of 1000 http://f1000.com

×