Going beyond counts: understanding impact

689 views

Published on

Webinar: Librarians & #altmetrics: tools, tips and use cases

Altmetrics are becoming an integral part of looking at the impact and reach of research. Tracking social and online outlets, altmetrics provide quick feedback from a wide range of sources. In this webinar, library experts will discuss how altmetrics work, tools available, and the application of altmetrics in a range of institutions and for various user groups. What steps should librarians be taking to incorporate altmetrics into their workflows and reports? - See more at: http://libraryconnect.elsevier.com/articles/2014-01/webinar-librarians-altmetrics-tools-tips-and-use-cases#sthash.Lfdao6rU.dpuf

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
689
On SlideShare
0
From Embeds
0
Number of Embeds
81
Actions
Shares
0
Downloads
6
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • Share with you some work we’ve done toward trying to understand the impact of biomedical research in our library and on our campus.While our focus is decidedly biomedical in nature, many of these concepts can (and are!) being applied more broadly to a host of other disciplinesMuch if this work is done in the context of the Washington University Institute of Clinical and Translational Sciences.Focus on accelerating bench discoveries into the clinical environ and beyond to result in improved health
  • We know it is important to think about issues in this area – but it is so much bigger than dealing with promotion and tenure activities.Understand how others are building on your work and moving your field forwardQuantify return on investment and justify future funding – a big deal in these lean budget timesDetermine any meaningful outcomes related to health and well-being of the greater communityLet’s think about productivity and outputs of a typical researcher.
  • She is shown on the ladder. She is focused on writing papers.Do research, write papers, do more research, write more papers.The workflow is set up so that the papers seem to be the end goal. Papers aren’t a bad thing – don’t get me wrong!Citation data can tell us a lot, as can the bibliographic metadata.
  • Our researcher’s impact is so much more than her publication outputs.These alternative outputs (and alternative metrics!) can have monumentally significant impact on our health and well-being.TO this point, it seems as though we have focused our energies on counting the things that are easy to count – papers, citations, and grants, primarily.Although there is a lot of emphasis placed in scholarly articles, we should remember they are mere artifacts of the scholarly process – placeholders which cannot tell the true story and impact of research.Alternative outputs…yield alternative metrics for understanding impact
  • We need a better understanding of the effects of an article, person, institute or event an entire institution in the scholarly ecosystem – rankings, ROIMetrics for assessing research performance, quality and impact cover a wide range of the scholarly ecosystem and are used for a variety of purposes, ranging from individual career planning; promotion and tenure; benchmarking to track group or institutional performance; marketing and strategic planning purposes; and reporting of research outcomes to the public.Outcomes which allow you to understand the true value of an effort
  • Applying the pathways, we are able to uncover outputs and impacts that may not be visible if one only follows the papers – things related to Diagnosis, clinical care, clinical environment. Cost-effective interventionsEven materials and improvements related to how we educate doctors to practice medicine
  • Our work in this area really orbits around efforts to develop a library-based framework for understanding meaningful research impact, The Becker Model.Project with Cathy Sarli, originally motivated by work she did for one of our researchers at Washington University, Dr. Mae Gordon.
  • Consistency enhances retrieval.
  • How do we scale up and operationalize this process
  • I think that research information systems can play a big role in helping us operationalize this process – as well as leverage the data at the enterprise level for higher-order analyses, visualizations, and strategic planning efforts.This is an area where we see a lot of library involvement from data to development and implementation, to outreach and ontology work.Many products (VIVO, SciVal Experts, Profiles, etc.)
  • An open source semantic web application An information model – for the VIVO platform, but also other research networking systems and other efforts in the scholarly reputation and representation realm through the VIVO ontologyAn open community* - awesome open-source community with many different areas of interest (development, implementation, outreach, and ontology developmentOk – so how do we use a researcher networking system?
  • First let’s think about providing a space for investigators to represent all of these meaningful outputs and indicators of impactEnhance traditional profile information with the things we’ve mentioned that focus on improved diagnoses, clinical care, cost effective interventions, and so on. This will require extending the ontology (or information model) to accommodate these concepts. Profile data are machine readable and rank highly on search result, enhancing both dissemination and discovery.What about integrating with the tools that Jenny and Mike mentioned? Let’s look at one example from Plum Analytics.
  • The VIVO system aggregates information about researchers at an institutionPlum can Harvest research outputs from the profile Plum’s harvesting engine calculates metrics and provides both an analytics dashboard as well as visualizations. Widgets can be embedded back into the VIVO profile!
  • Plum widget – gives a 30K-foot view of a specific output, a person, a department or research center and more using the types of metrics that Mike and Jenny discussed.Plum widgets are embeddable in your institutional repository as well.Widgets could be a number of different things – Also reflect activity in code or data repositories.
  • Just a few notes of thanks. First to Cathy Sarli, a dear colleague and collaborator. We work together on the Becker Model.Cathy, Karen Gutzman (NLM second year fellow) and I work together at Becker Library and we are also members of the WU ICTS Tracking and evaluation team. We’re in the midst of a pilot project to apply the Becker Model to ICTS investigators and look forward to learning more from this process. Jae Allen, another member of this team very kindly gave permission for me to use some of the images you saw today.Andrea Michalek from Plum – very cool work – for their work with VIVO and information for this presentationFinally, the VIVO team and open source community – a fantastic group of friends and colleagues from all over the world doing very cool work and spreading Linked Open Data love wherever they go. Librarians may be interested to read more about a complementary effort, Linked Data for Libraries, a recent award from the Mellon Foundation to Cornell, Stanford, and Harvard Libraries.
  • Going beyond counts: understanding impact

    1. 1. Kristi Holmes Bioinformaticist Washington University in St. Louis Going beyond counts: understanding impact Webinar #LCwebinar
    2. 2. http://icts.wustl.edu/ Translational Research The bench to the bedside and beyond Webinar Going beyond counts: understanding impact Kristi Holmes
    3. 3. Why do we need to think about research impact? • Quantify and document research impact • Justify future requests for funding • Quantify return on research investment • • Discover how research findings are being used • • Identify similar research projects • Identify possible collaborators • Determine if research findings are duplicated, confirmed, corrected, improved or repudiated https://becker.wustl.edu/impact-assessment/model Going beyond counts: understanding impact • • • • • Determine if research findings were extended (different human populations, different animal models/species, etc.) Confirm that research findings were properly attributed/credited Demonstrate that research findings are resulting in meaningful health outcomes Discover community benefit as a result of research findings Progress reports Tenure Promotion dossiers Webinar Kristi Holmes
    4. 4. How do we think about productivity? Don’t knock publications • Citations – Rates and rankings – Networks – Dissemination patterns • • • • Collaboration Dissemination Grant funding Research trends Webinar http://www.philnel.com/2010/10/14/procrastigrading/ Going beyond counts: understanding impact Kristi Holmes
    5. 5. …but there’s so much more! Webinar http://www.philnel.com/2010/10/14/procrastigrading/ Going beyond counts: understanding impact Kristi Holmes
    6. 6. Research Impact “It is no longer enough to measure what we can – we need to measure what matters.” So how do we measure what matters? Webinar Wells R, Whitworth A. 2007. Assessing outcomes of health and medical research: do we measure what counts or count what we can measure? Australia and New Zealand Health Policy, 4:14 Going beyond counts: understanding impact Kristi Holmes
    7. 7. Going beyond the counts to find meaningful impact In everyone’s CV, we’ll find • Reviews • New funding awarded • New types of funding mechanisms • New research studies • Invited lectures, new focus areas at conferences • Membership on committees • Awards Consider specific pathways* to uncover meaningful impact…  Advancement of Knowledge  Clinical Implementation  Legislation and Policy Enactment  Economic Benefit  Community Benefit *can be adapted to reflect any number of disciplines Webinar Going beyond counts: understanding impact Kristi Holmes
    8. 8. Going beyond the counts to find meaningful impact • New diagnostic criteria • • • • • • • • • http://icts.wustl.edu/ • http://icts.wustl.edu/ Going beyond counts: understanding impact New standard of care Curriculum guidelines Measurement instruments Continuing education materials Clinical/practice guidelines Quality measure guidelines Private healthcare benefit plans Cost-effective intervention Consensus development conferences American Medical Association Current Procedural Terminology (CPT) codes Change in delivery of healthcare services Webinar Kristi Holmes
    9. 9. Slide title https://becker.wustl.edu/impact-assessment Going beyond counts: understanding impact Webinar Kristi Holmes
    10. 10. The Becker Model • Provides a supplement to publication analysis to provide a more robust and comprehensive perspective of biomedical research impact. – reporting templates, glossary of resources and terms, examples of relevant indicators of impact across the research process, and readings • Straightforward framework for tracking diffusion of research outputs and activities to locate indicators that demonstrate evidence of biomedical research impact – individual, core, and institutional-level; modify for different disciplines • Guidance for quantifying and documenting research impact as well as resources for locating evidence of impact. • Strategies for enhancing the impact of research Webinar Going beyond counts: understanding impact Kristi Holmes
    11. 11. Strategies for enhancing the impact of research Optimizing discoverability and access of your research is the surest way to enhance its visibility and impact. The strategies focus upon Preparing for Publication, Dissemination, and Keeping Track of Your Research. Repetition, consistency, and an awareness of the intended audience form the basis of most the strategies. Suggestions for researchers and recommendations to reach out to their library for assistance. Webinar https://becker.wustl.edu/impact-assessment/strategies Going beyond counts: understanding impact Kristi Holmes
    12. 12. Understanding the impact of a single paper can be challenging. How do we understand the impact of a person? A research center? A university? How do we scale up and operationalize this process? Webinar Going beyond counts: understanding impact Kristi Holmes
    13. 13. Research networking and discovery systems • • • • • Enable more efficient means of collecting & representing meaningful outputs en masse Showcase achievements and expertise Facilitate diffusion of research products Support team-based science and collaboration Allow better understanding of the research enterprise – Peer comparisons – Strategic planning – Emerging trends Webinar Going beyond counts: understanding impact Kristi Holmes
    14. 14. What is VIVO? 1. An open source semantic web application 2. An information model 3. An open community* * A big, welcoming OS community! Let us know if you have questions or need information, connections, or materials: http://vivoweb.org/contact Going beyond counts: understanding impact Webinar Kristi Holmes
    15. 15. Tying it all together Continue to represent typical profile information – absolutely! Enhance profiles by incorporating meaningful outputs, as described by the Becker Model or other frameworks, to enhance their content and value http://vivo.ufl.edu/display/n86607 Going beyond counts: understanding impact Can facilitate the dissemination of scholarship and more Webinar Kristi Holmes
    16. 16. Tying it all together PlumX supports VIVO 1. Aggregate information about our researchers 2. Harvest research outputs from the profile 3. Plum’s harvesting engine calculates metrics and provides both an analytics dashboard as well as visualizations. 4. Widgets can be embedded back into the VIVO profile! http://blog.plumanalytics.com/post/57707501083/plumx-now-supports-vivo Going beyond counts: understanding impact Webinar Kristi Holmes
    17. 17. Tying it all together http://vivo.ufl.edu/display/n86607 FigShare, lab notebooks, project outputs, code/data repositories SlideShare, etc. PlumX supports VIVO 1. Aggregate information about our researchers 2. Harvest research outputs from the profile 3. Plum’s harvesting engine calculates metrics and provides both an analytics dashboard as well as visualizations. 4. Widgets can be embedded back into the VIVO profile! Webinar http://blog.plumanalytics.com/post/57707501083/plumxnow-supports-vivo Going beyond counts: understanding impact Kristi Holmes
    18. 18. Acknowledgements Thanks: • • • • Cathy Sarli, MLS, AHIP Karen Gutzman – NLM Fellow Jae Allen Washington University ICTS and WU ICTS Tracking & Evaluation Team • Andrea Michalek at Plum @amichalek • VIVO Community @VIVOcollab • Becker Medical Library Funding: • Washington University Institute of Clinical and Translational Sciences, NIH award UL1 RR024992 • VIVO - DuraSpace Webinar Going beyond counts: understanding impact Kristi Holmes

    ×