Kristi Holmes, PhD
2AM - Amsterdam
7 October 2015
Understanding impact
through alternative metrics
Library-based assessmen...
Northwestern University Clinical and Translational
Sciences (NUCATS) Institute
Mission: Speeding transformative
research d...
What is Translational
Science?
Translation form “Bench to Bedside”
• Circa 2000: 18 years from discovery to
dissemination...
- for successful therapies
•...
CTSAs – Mission and Structure
10/7/15 5
Emphasis on local/institutional
support for
Preclinical developmental
therapeutics...
http://www.academicdepartments.musc.edu_sebin_b_o_Translational_Science.jpg
Translational Medicine
Northwestern University Clinical and
Translational Sciences (NUCATS)
Institute Evaluation and Continuous
Improvement Progr...
Principles to guide ECI
• Open and collaborative processes
• Good data (quality and quantity)
• Look for opportunities for...
Desperately seeking data
What do we want?
a better perspective of both productivity and impact
•  Multiple sources coverin...
Papers published by NU-affiliated authors
1950-present (as indexed in Scopus)
10
Lots of missing data here:
•  Citations t...
What is research impact?
A Definition of Research Impact
Research impact is the demonstrable contribution that research ma...
Context is
everything!
•  Measurement	
  instruments	
  
•  Con.nuing	
  educa.on	
  materials	
  
•  Quality	
  measure	
  guidelines	
  
•  Cos...
Building an
ecosystem
Library as Partner
Opportunity!
Why a library--based support model?
•  Are a trusted, neutral space
•  Have a tradition of service and support
•  Strive t...
Galter Library’s Metrics and Impact Core
The Metrics and Impact Core (MIC), housed in Galter Library, has expertise in
bib...
So why did you name it …CORE?
NUCATS Evaluation Metrics and Approach
19
Workflows, dashboards, system integrations, feedback loops
20
7 steps for success
1.  Identify stakeholders, audience, and champions
2.  Pay attention to local and national/global issu...
…and CONNECT!
• res-impact@googlegroups.com
-  Librarians and information scientists doing assessment and
visualizations
-...
Thank you!
Kristi Holmes
@kristiholmes
kristi.holmes@northwestern.edu
orcid.org/0000-0001-8420-5254
Upcoming SlideShare
Loading in …5
×

Understanding impact through alternative metrics: developing library-based assessment services

2,054 views

Published on

There’s never been a more critical need to better understand the impact of research efforts. The challenging state of funding models (1) and an enhanced pressure on young investigators to stand out from the crowd magnify this need as well as the perceived value of locally based impact services. These services are leveraged by a diverse range of stakeholders, from individuals to university-level decision makers and strategists. Individuals often wish to better demonstrate impact of published works to promotion committees or describe the impact of research studies to funding agencies when applying for funding or complying with institution-level or federal reporting exercises. Research groups, departments, and institutions often wish to discover how research findings are being used to promote science and gain a better overall view of research publications and outputs.

Libraries are particularly well poised to meet the need to understand a more nuanced view of impact. Libraries are trusted, neutral parties with a tradition of service and support and often act as technology hubs on campus with IT and data expertise. Librarians are trained information professionals with information and searching skills and a keen understanding of the research, education, clinical landscape of their institution. This presentation will discuss general trends in the field, including an overview of resources, assessment frameworks and tools; strategies for partnering with stakeholders; and examples of library based service models, from basic services to highly integrated library-based core research units.

(1) http://dx.doi.org/10.1126/scitranslmed.aac5200

Published in: Science
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,054
On SlideShare
0
From Embeds
0
Number of Embeds
401
Actions
Shares
0
Downloads
12
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Understanding impact through alternative metrics: developing library-based assessment services

  1. 1. Kristi Holmes, PhD 2AM - Amsterdam 7 October 2015 Understanding impact through alternative metrics Library-based assessment services
  2. 2. Northwestern University Clinical and Translational Sciences (NUCATS) Institute Mission: Speeding transformative research discoveries to patients and the community http://nucats.northwestern.edu/
  3. 3. What is Translational Science?
  4. 4. Translation form “Bench to Bedside” • Circa 2000: 18 years from discovery to dissemination... - for successful therapies • Circa 2015: 7 years... - and we are smarter about what should go through the pipeline • And we understand that translation has to go beyond the bedside
  5. 5. CTSAs – Mission and Structure 10/7/15 5 Emphasis on local/institutional support for Preclinical developmental therapeutics Enhanced clinical trial infrastructure Community-engaged research Incorporated aspects of training and career development of translational scientists (K/T awards)
  6. 6. http://www.academicdepartments.musc.edu_sebin_b_o_Translational_Science.jpg Translational Medicine
  7. 7. Northwestern University Clinical and Translational Sciences (NUCATS) Institute Evaluation and Continuous Improvement Program Mission: Speeding transformative research discoveries to patients and the community http://nucats.northwestern.edu/ Some sample metrics… OUTPUT Metrics Time to Publication or other output Number of technology transfer products ROI of pilot awards ROI of TL1, T32, KL2 scholars Time from publication to research synthesis IMPACT Metrics Influence of a research output Researcher and institutional collaborations Career development and path/ trajectory PROCESS Metrics Time from IRB submission to approval Time from grant award to start of study Volume of investigators who use services, take training, and other activities
  8. 8. Principles to guide ECI • Open and collaborative processes • Good data (quality and quantity) • Look for opportunities for data reuse • Limited human interaction with data whenever possible • Leverage common workflows, look for opportunities
  9. 9. Desperately seeking data What do we want? a better perspective of both productivity and impact •  Multiple sources covering multiple subjects and disciplines; authoritative sources •  Disambiguated •  De-duplicated 1.  quality metadata 2.  a perspective of attention 3.  a way to tie these outputs to institutional efforts, systems, and workflows in a meaningful way grants and to our IR 9
  10. 10. Papers published by NU-affiliated authors 1950-present (as indexed in Scopus) 10 Lots of missing data here: •  Citations to these works •  Valuable bibliographic metadata (collaborators, journal titles, affiliations, keywords and indexing, etc.) •  Non-NU affiliated papers, but written by NU people (e.g., before they arrived) •  Reliable links to funding •  Open access status? Compliance status? •  Attention metrics •  Only traditional outputs are accounted for…so several other outputs are missed! Low hanging fruit – already addressing this with the Galter Metric and Impact Core and other key initiatives.
  11. 11. What is research impact? A Definition of Research Impact Research impact is the demonstrable contribution that research makes to the economy, society, culture, national security, public policy or services, health, the environment, or quality of life, beyond contributions to academia. Academic impact The demonstrable contribution that excellent research makes to academic advances, across and within disciplines, including significant advances in understanding, methods, theory and application. Economic and societal impacts The demonstrable contribution that excellent research makes to society and the economy. Economic and societal impacts embrace all the extremely diverse ways in which research- related knowledge and skills benefit individuals, organizations and nations Research Impact Principles and Framework, Australian Research Council http://www.arc.gov.au/general/impact.htm Pathways to Impacts, Research Councils UK http://www.rcuk.ac.uk/ke/impacts/
  12. 12. Context is everything!
  13. 13. •  Measurement  instruments   •  Con.nuing  educa.on  materials   •  Quality  measure  guidelines   •  Cost-­‐effec.ve  interven.on   •  Consensus  development  conferences     •  American  Medical  Associa.on  Current   Procedural  Terminology  (CPT)  codes   •  Change  in  delivery  of  healthcare  services   •  Gray  literature   Going beyond the counts to find evidence of meaningful impact •  New  experimental  methods,  data   models,  databases,  or  soFware  tools   •  New  diagnos.c  criteria     •  New  standards  of  care   •  Biological  materials  or  animal  models   •  Consent  documents   •  Clinical/prac.ce  guidelines   •  Quality  measure  guidelines   https://becker.wustl.edu/impact-assessment http://nucats.northwestern.edu/ Diverse outputs Diverse impacts Diverse roles Each a critical component of the research process
  14. 14. Building an ecosystem
  15. 15. Library as Partner Opportunity!
  16. 16. Why a library--based support model? •  Are a trusted, neutral space •  Have a tradition of service and support •  Strive to serve all missions of the institution •  Are technology centers and have IT and data expertise •  Have skills—information organization, instruction, usability, subject expertise, ontologies and controlled vocabularies •  Have close relationships with their clients (buy in) •  Understand user needs •  Understand the importance of collaboration and know how to bring people together •  Have knowledge of institution, research, education, clinical landscape Library  Staff:   Libraries:  
  17. 17. Galter Library’s Metrics and Impact Core The Metrics and Impact Core (MIC), housed in Galter Library, has expertise in bibliometrics, data visualization, continuous improvement, information systems and alternative metrics. The core provides extensive advisory services for researchers, groups or departments on topics such as: •  developing successful publishing strategies •  managing or tracking publications •  maintaining an impactful online identity •  measuring or assessing research impact by discipline •  communicating research impact to audiences MIC uses a wide collection of resources, including Scopus, Web of Science, Essential Science Indicators, Google Scholar, NU Scholars, Journal Citation Reports, and more, to provide services and reports for: -  Researchers or clinicians to demonstrate impact of published works to promotion or tenure committees, or the impact of research studies to funding agencies when applying for funding -  Research groups/institutions/departments to discover how research findings are being used to promote science, or an overall view of research publications and outputs by a specific group
  18. 18. So why did you name it …CORE?
  19. 19. NUCATS Evaluation Metrics and Approach 19 Workflows, dashboards, system integrations, feedback loops
  20. 20. 20
  21. 21. 7 steps for success 1.  Identify stakeholders, audience, and champions 2.  Pay attention to local and national/global issues 3.  What’s missing? What’s needed? 4.  What can you do today? Next week? In a year? 5.  What do you have in hand? What do you need? 6.  Communicate – develop a communication plan to encourage engagement during development and launch 7.  Don’t wait until you (fill in the blank)! You can do something terrific NOW!
  22. 22. …and CONNECT! • res-impact@googlegroups.com -  Librarians and information scientists doing assessment and visualizations -  Very practical group – sharing ideas, strategies, etc. • Mosaic: Be Part of the Big Picture joint meeting – to be held May 13–18, 2016, in Toronto, ON. -  Medical Library Association (MLA) -  Canadian Health Libraries Association/Association des bibliothéques de la santé du Canada (CHLA/ABSC) -  International Clinical Librarian Conference (ICLC)
  23. 23. Thank you! Kristi Holmes @kristiholmes kristi.holmes@northwestern.edu orcid.org/0000-0001-8420-5254

×