Downloaded 14 times
Uploaded byTaylor Moore
Evaluating Research Impact
This document discusses evaluating the impact of research beyond just counting publications. It presents various outputs of research including peer-reviewed publications, reports, presentations, and patents. It also discusses measuring research impact through citations, follow-up research, and research priorities. Additionally, it proposes using bibliometric, dissemination, social media, and download metrics to more fully capture a research project's impact.
Evaluating Research Impact
- 1. Evaluating Research Impact: Beyond Counting Publications J.Taylor Moore, PhD CDC Evaluation Fellow National Institute for Occupational Safety and Health (NIOSH)
- 2. Outputs of Research: •Peer-reviewed publications • Final reports • Conference presentations • Patents • Training materials
- 3.
- 4. Bibliometric Approach toImpact
- 7.
- 8. Moore, J. T.,Cigularov, K. P., Sampson, J. M., Rosecrance, J. C., & Chen, P. Y. (2013). Construction workers’ reasons for not reporting work-related injuries: An exploratory study. International Journal of Occupational Safety and Ergonomics (JOSE), 19(1), 97-105.
- 12.
- 13.
- 15.
- 16. Peerreviewed Journal Article • Downloads • 16Downloads Industry Trade Journals/ Newsletters • Shares • 8 articles Social Media (Sharing) • Reach • 102,000 followers
- 18. • Publications • Citations • • • • • • E-newsletters Tweets FacebookPosts Blogs Shares Downloads
- 20.
Editor's Notes
- #3 Jumping right into it, We are all pretty familiar with the resulting products of scientific research that include final reports, presentations, patents, training materials to name a few. But the primary outputs are peer-reviewed publications, the focus of this presentation.
- #4 So what is the impact of research outputs? Well,a publication may lead to:Other researchers citing the research in their work Follow-up research projects from the same researcherFinally, the research findings may contribute to setting new research priorities.This is by no means an exhaustive list of outcomes.
- #5 The traditional approach to measuring research impact has relied on measures such as a Citation Count, an h-index, and other various indexes whose goal it is to describe the impact based on how many times a given article is cited in other published papers. This is called the bibliometric approach.
- #6 However, there are a number of limitations of these traditional measures of research impact. Mainly, they take months or years to bubble up. Surely there is a more timely way to measure the impact of research.
- #7 A second limitation of the current bibliometric approach to assessing research impact is that it many of the findings from research never leave the ivory towers of the universities and research institutes. While this is acceptable for many of the basic sciences, in applied research settings there should be an emphasis on research that translates to real world application. And a corresponding measure of whether this is occuring.
- #8 Given these limitations of the bibliometricapproach, I propose that we are only measuring the tip of the ice berg. There has to be more to research impact than just counting publications and citations. So what lies below the surface? I will start with an example.
- #9 Earlier this year I had a paper published in the International Journal of Occupational Safety and Ergonomics. It was a qualitative study that examined construction workers’ reasons for not reporting work-related injuries.
- #10 In July and August, a number of trade journals took notice of the published paper after the Center for Construction Research and Training featured it in their newsletter. I was interviewed by 4 different trade journals and there were 8 articles published in August that covered my work.
- #11 These trade journal articles were then shared via social media. The article in Paint Square was tweeted out to the 102,000 followers of the US Dept. of Labor.
- #12 So hereis a recap of the example I just shared, put into a flow process diagram. Notice that the traditional bibliometrics approach described earlier does not measure these types of outcomes.
- #13 We all know that in logic model world, outputs lead to outcomes… But in reality, there is a process by which outputs, specifically publications, get disseminated and then contribute to an outcome. So how do we bridge this chasm?
- #14 I propose that there is a process similar to that in the example that I described that fits between the outputs and outcomes, what is described as knowledge dissemination. It is all about the channels of communication that research findings may take from a publication, to trade/industry journals, to social media as awareness of the research findings spreads.
- #15 Now I want to tell you about an emerging set of alternativemetrics, or altmetrics, that adds a piece to the puzzle of how to measure Research Impact.Altmetrics are analytical tools designed to measure the social reaction to our research via traditional and digital social media. It’s a strategy to measure reach, engagement and influence.Unlike bibliometrics which take months or years to demonstrate impact, altmetrics can be collected fast, letting researchers see, almost in real time, how an article or data set or blog post is moving through all levels of the scholarly ecosystem.
- #16 Going back to my example, some of the altmetrics that can be measured include the number of times that journal article is downloaded, the number of times that the research is covered by trade journals or newsletters, and finally how many people are potentially reached through the sharing of these articles on social media.
- #17 For those of you who want to see the actual numbers, here they are for the example.
- #18 There are a couple of services currently available that collect these metrics for you. Altmetric is one of them. Each article is given a score that measures the quantity and quality of attention it has received to date. The score donutvisualization tells you what kind of attention at a glance. No it’s not free. ImpactStory is a Web-based application that makes it easy to track the impact of a wide range of research artifacts (such as papers, datasets, slides, research code). The system aggregates impact data from many sources, from Mendeley to GitHub to Twitter and more, and displays it in a single, permalinked report.
- #19 As I said earlier, publications, citations, and the bibliometirc approach to evaluation research impact is just the tip of the ice berg. Considering the process by which research knowledge is disseminated via electronic media and social media provides more of a complete picture of the true impact of research.
- #20 I hope my presentation has helped convince you that counting publications is not enough to understand the impact of research. And maybe I’ve convinced you that alternative metrics have the potential to add to the evaluation of research impact.
- #21 Thank you for listening to my ideas and if you are interested in learning more about evaluating research impact and altmetrics let Tom Chapel know!