Downloaded 17 times
Assessing and Reporting Research Impact – A Role for the Library - Kristi L. Holmes, Ph.D., Director, Galter Health Sciences Library, Northwestern University, Feinberg School of Medicine
The document discusses the role of libraries in assessing and reporting the impact of research, emphasizing the importance of metrics and frameworks for quantifying research impact and facilitating collaboration. It highlights alternative metrics, the need for strategic approaches to enhance research visibility, and the integration of tools like Plum Analytics for monitoring research outputs. The author calls for proactive engagement with stakeholders to improve research assessment practices at both local and institutional levels.
More Related Content
Similar to Assessing and Reporting Research Impact – A Role for the Library - Kristi L. Holmes, Ph.D., Director, Galter Health Sciences Library, Northwestern University, Feinberg School of Medicine
More from National Information Standards Organization (NISO)
Assessing and Reporting Research Impact – A Role for the Library - Kristi L. Holmes, Ph.D., Director, Galter Health Sciences Library, Northwestern University, Feinberg School of Medicine
- 1. Assessing and ReportingResearch Impact – A Role for the Library Kristi L. Holmes, PhD Director, Galter Health Sciences Library Associate Professor, Preventative Medicine-Health and Biomedical Informatics Northwestern University, Feinberg School of Medicine http://orcid.org/0000-0001-8420-5254 NISO Virtual Conference: Transforming Assessment: Alternative Metrics and Other Trends June 18, 2014
- 2. overview • Impact metrics- the ususal suspects and more • A view from the library • Identifying needs • Roles played by libraries • Developing services • Moving this space forward on the local and larger level
- 3. impact metrics –why do we care? • Quantify and document research impact • Justify future requests for funding • Quantify return on research investment • Discover how research findings are being used • Identify similar research projects • Identify possible collaborators • Determine if research findings are duplicated, confirmed, corrected, improved or repudiated • Determine if research findings were extended (different human populations, different animal models/species, etc.) • Confirm that research findings were properly attributed/credited • Demonstrate that research findings are resulting in meaningful health outcomes • Discover community benefit as a result of research findings • Progress reports • Tenure • Promotion dossiers https://becker.wustl.edu/impact-assessment/model
- 4. alternative metrics • Twitter •Facebook • Blogs • Research Highlights • Google+ • Main Stream Media • Reddit • Forums • Q&A – StackExchange • Pintrest • LinkedIn • FigShare • F1000 Reviews • GitHub Thelwall M, Haustein S, Larivière V, Sugimoto CR (2013) Do Altmetrics Work? Twitter and Ten Other Social Web Services. PLoS ONE 8(5): e64841. doi:10.1371/journal.pone.0064841 Konkiel S (2013) Altmetrics: A 21st-Century Solution to Determining Research Quality. Information Today. Available at http://www.infotoday.com/OnlineSearcher/Articles/Features/Altmetrics-A-stCentury- Solution-to-Determining-Research-Quality-90551.shtml Articles Datasets Slides Software Webpages
- 5. Understanding the impactof research is a massive task. How do we understand the impact of a published work? A person? A research center? A university? What IS impact?
- 6. Ioannidis JA, KhouryMJ. Assessing Value in Biomedical Research: The PQRST of Appraisal and Reward. JAMA. Published online June 09, 2014. doi:10.1001/jama.2014.6932. • We need to move assessment toward “desired outcomes: research that is productive, high-quality, reproducible, shareable, and translatable (PQRST)” • Productivity metrics should reward high- influence science rather than least publishable units and decrease publication bias against negative results. Research impact and assessment
- 7. go beyond counts! https://becker.wustl.edu/impact-assessment Investigateand adapt frameworks to help put things into context More and more organizations and efforts are considering the impact of their work through frameworks. The Becker Model Frameworks from the IOM, CDC, and NIEHS Snowball Metrics Discipline-specific apporaches and more…
- 8. go beyond counts! https://becker.wustl.edu/impact-assessment Investigateand adapt frameworks to help put things into context
- 9. The Becker Model •Provides a supplement to publication analysis to provide a more robust and comprehensive perspective of biomedical research impact. – reporting templates, glossary of resources and terms, examples of relevant indicators of impact across the research process, and readings • Straightforward framework for tracking diffusion of research outputs and activities to locate indicators that demonstrate evidence of biomedical research impact – individual, core, and institutional-level; modify for different disciplines • Guidance for quantifying and documenting research impact as well as resources for locating evidence of impact. • Strategies for enhancing the impact of research
- 10. strategies for enhancingthe impact of research https://becker.wustl.edu/impact-assessment/strategies Repetition, consistency, and an awareness of the intended audience form the basis of most the strategies. The strategies focus upon Preparing for Publication, Dissemination, and Keeping Track of Your Research. Optimizing discoverability and access of your research is the surest way to enhance its visibility and impact. Suggestions for researchers and recommendations to reach out to their library for assistance.
- 11. Understanding the impactof research is a massive task. How do we understand the impact of a published work? Aperson? Aresearch center? Auniversity? What IS impact? How do we scale up and operationalize this process? How do we support it locally from the library?
- 12. leverage ongoing effortson the local level and beyond • ORCID • CASRAI • CERIF • VIVO-ISF ontology • NISO Altmetric work • CV/Biosketch • Large-scale semantic search • Institutional repositories • Research networking systems • Institutional research information and management systems • Analytical and benchmarking efforts • Tracking and evaluation
- 13. VIVO lends itselfto research assessment and impact • Facilitates a more robust research ecosystem – Diffusion of research products – Discovery of shared products – Reproducibility – Translation – Enables credit and visibility by showcasing individual achievements and expertise • Supports team-based science and collaboration • Allows better large-scale understanding of the research enterprise – Temporal relationships (career development, time from publication to research synthesis) – Peer comparisons – Strategic planning and visualization – Identify emerging trends • Enables more efficient means of collecting & representing meaningful outputs en masse • New experimental protocols • Datasets • Instrumentation • Software code • New diagnostic criteria • New standard of care • Curriculum guidelines • Measurement instruments • Continuing education materials • Clinical/practice guidelines
- 14. VIVO lends itselfto research assessment and impact • Facilitates a more robust research ecosystem – Diffusion of research products – Discovery of shared products – Reproducibility – Translation – Enables credit and visibility by showcasing individual achievements and expertise • Supports team-based science and collaboration • Allows better large-scale understanding of the research enterprise – Temporal relationships (career development, time from publication to research synthesis) – Peer comparisons – Strategic planning and visualization – Identify emerging trends • Enables more efficient means of collecting & representing meaningful outputs en masse • New experimental protocols • Datasets • Instrumentation • Software code • New diagnostic criteria • New standard of care • Curriculum guidelines • Measurement instruments • Continuing education materials • Clinical/practice guidelines VIVO can help operationalize this process: Enhance profiles by incorporating meaningful outputs to enhance content and value; can facilitate dissemination and discovery of scholarship The ontology is the key…
- 15. • Plum Analyticsgathers metrics across five categories—usage, mentions, captures, social media and citations. • Metrics are gathered around artifacts • Collected information is displayed in visualizations, dashboards, and widgets. • Customized for the institution or organization. ARTIFACTS articles blog posts book chapters books cases clinical trials conference papers datasets figures grants interviews letters media patents posters presentations source code theses / dissertations videos webpages http://www.ebscohost.com/newsroom/stories/plum-analytics-becomes-part-of-ebsco-information-services IR integration
- 16.
- 17. IR integration Benefits? 1. Createa feedback loop for researchers that gives them timely insight into the impact of their research right from the start 2. Easy access to advanced metrics from PlumX can help build buy-in from researchers, increasing repository support by your contributors 3. Aggregated metrics and reports from the PlumX dashboard puts the repository at the center of communicating about the impacts of your organization’s research Courtesy of Plum
- 18. a spectrum ofpossibilities… http://commons.wikimedia.org/wiki/File:Linear_visible_spectrum.svg Integrate altmetrics in your IR Lead or support RNS project (VIVO, etc) Run citation reports ORCID services and support Encourage prudent use of online tools Welcome deposit of alternative outputs in your IR Keep current Collaborate with TT office to track outputs Develop standard reports/visualiz ations/analyses Hire an impact and evaluation librarian Workshop on strategies for enhancing research impactRun publication data reports Establish a consultation service Help researchers obtain DOIs and understand data options Understand motivations of funding agencies
- 19. local success dependson stakeholder engagement • Stay up to date on scholarly issues and the scholarly workflow • Stay up to date on funding/reporting requirements • Brainstorm to identify and understand motivations of stakeholders • Anticipate need and present solutions • Call on advocates • Think beyond ‘business as usual’ • Be persistent!
- 20. Acknowledgements Support: • Northwestern University Clinicaland Translational Sciences Institute, NIH award 8UL1TR000150-05 • VIVO - DuraSpace Thanks: • Cathy Sarli, MLS, AHIP • Karen Gutzman – NLM Fellow • Andrea Michalek at Plum @amichalek • VIVO Community @VIVOcollab • Galter Health Sciences Library Questions: @kristiholmes kristi.holmes@northwestern.edu Thank you!
Editor's Notes
- #6 How do we scale up and operationalize this process
- #8 Our work in this area really orbits around efforts to develop a library-based framework for understanding meaningful research impact, The Becker Model. Project with Cathy Sarli, originally motivated by work she did for one of our researchers at Washington University, Dr. Mae Gordon.
- #9 Our work in this area really orbits around efforts to develop a library-based framework for understanding meaningful research impact, The Becker Model. Project with Cathy Sarli, originally motivated by work she did for one of our researchers at Washington University, Dr. Mae Gordon.
- #11 Consistency enhances retrieval. A lot of the suggestions lie directly in our wheelhouse – makes this a great opportunity for the library
- #12 How do we scale up and operationalize this process
- #13 Multiple platforms
- #14 VIVO can play a big role in helping us operationalize this process – as well as leverage the data at the enterprise level for higher-order analyses, visualizations, and strategic planning efforts. Much of this is not possible to tease out from bibliographic or grant data Applying the pathways, we are able to uncover outputs and impacts that may not be visible if one only follows the papers – things related to Diagnosis, clinical care, clinical environment. Cost-effective interventions Even materials and improvements related to how we educate doctors to practice medicine
- #15 VIVO can play a big role in helping us operationalize this process – as well as leverage the data at the enterprise level for higher-order analyses, visualizations, and strategic planning efforts. Much of this is not possible to tease out from bibliographic or grant data Applying the pathways, we are able to uncover outputs and impacts that may not be visible if one only follows the papers – things related to Diagnosis, clinical care, clinical environment. Cost-effective interventions Even materials and improvements related to how we educate doctors to practice medicine
- #16 KH Let’s use Plum Analytics as a good example for a deep dive on the types of things you can track…
- #19 Subjects Trade Names Manufacturers Publisher Abstract Author Keywords Index Keywords Date Grant Funding Support Language DOI Source Document Type Peer Review Status Authors Author Affiliations Corresponding Author Group Authors Chemicals Institutions Countries References Citation Counts
- #21 Just a few notes of thanks. First to Cathy Sarli, a dear colleague and collaborator. We work together on the Becker Model. Cathy, Karen Gutzman (NLM second year fellow) and I work together at Becker Library and we are also members of the WU ICTS Tracking and evaluation team. We’re in the midst of a pilot project to apply the Becker Model to ICTS investigators and look forward to learning more from this process. Andrea Michalek from Plum – very cool work – for their work with VIVO and information for this presentation Finally, the VIVO team and open source community – a fantastic group of friends and colleagues from all over the world doing very cool work and spreading Linked Open Data love wherever they go. Librarians may be interested to read more about a complementary effort, Linked Data for Libraries, a recent award from the Mellon Foundation to Cornell, Stanford, and Harvard Libraries.