The document describes how the CDC's Science Impact Framework can be used to measure the impact of scientific work beyond just citation data. It provides three case studies that will illustrate how the framework can be applied. The framework uses a combination of quantitative and qualitative indicators to measure outcomes across five levels of influence: disseminating science, creating awareness, catalyzing action, effecting change, and shaping the future. The case studies will demonstrate how scientific work can have a complex path of impact that does not necessarily follow a linear progression through these levels of influence.

1. Measuring Science Impact Beyond Citations (case studies)
Traditional journal metrics help us understand how widely the article content is disseminated. But then what?
Three case studies will illustrate how CDC’s Science Impact Framework can describe the importance of your
work beyond citation data. The Framework utilizes a combination of quantitative and qualitative indicators to
measure outcomes, through five levels of influence: disseminating science, creating awareness, catalyzing
action, effecting change, and shaping the future. The complex dynamics between the levels of influence and the
intricate environment in which influence materializes create a path of impact which does not necessarily follow
a linear progression.
……………………………………………………………………………………………………………………………………………………
Christine Casey, MD FAAP |CAPT, US Public Health Service
Editor, Morbidity and Mortality Weekly Report (MMWR) Serials
Centers for Disease Control and Prevention, ccasey@cdc.gov, 404-498-6621
https://www.cdc.gov/mmwr/staff/staff.html#Christine_G_Casey
Disclaimer: The findings and conclusions in this presentation are those of the author and do not necessarily represent the official position of the Centers for
Disease Control and Prevention / the Agency for Toxic Substances and Disease Registry.
Conflicts: none to disclose.
Acknowledgements: CDC and NISO colleagues
NISO Virtual Conference, Is This Still Working? Incentives to Publish, Metrics and New Reward Systems
Date and Time: Wednesday, February 20, 2019, 12:00 Noon – 4:00pm (Eastern)

2. Best poster
2014 Annual Meeting
Council of Science Editors
https://www.councilscienceeditors.org/

3. www.cdc.gov | Contact CDC at: 1-800-CDC-INFO or www.cdc.gov/info
The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.
Beyond Citations: Introducing a Story-Based Framework to Assess Science Impact
Christine Casey, MD1 ; John Araujo, PhD, MHSA, MA2; Richard Hunt, MD3; Scott Sasser, MD3,4; David Sugerman, MD, MPH3; Mark Faul, PhD, MA3; Terica Scott, MA3; Jeffrey Sokolow, MA1; and Tanja Popovic, MD, PhD2
1-Centers for Disease Control and Prevention (CDC), Morbidity and Mortality Weekly Report; 2-CDC, Office of the Associate Director for Science; 3-CDC, National Center for Injury Prevention and Control; and 4- Emory University School of Medicine
Morbidity and Mortality Weekly Report (MMWR)
Background
Editors rely on various journal metrics (bibliometrics, webometrics, and
altmetrics), but no single indicator can adequately measure the impact of
scientific publications.
The Morbidity and Mortality Weekly Report (MMWR) series, often called
“the Voice of CDC,” publishes scientific reports on urgent public health
concerns with recommendations for immediate application and action.
MMWR content is seen by ~1 million persons/month and is cited widely in
the scientific literature and media. MMWR cannot obtain certain journal
metrics due to government restrictions and indexing considerations.
Therefore, alternate methods are needed to describe and assess the impact
of MMWR content on public health practice.
CDC Science Impact Framework
Source: http://www.iom.edu/About-
IOM/~/media/Files/About%20the%20IOM/2
013-Presidents-Supplement.pdf

4. www.cdc.gov | Contact CDC at: 1-800-CDC-INFO or www.cdc.gov/info
The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.
Beyond Citations: Introducing a Story-Based Framework to Assess Science Impact
Christine Casey, MD1 ; John Araujo, PhD, MHSA, MA2; Richard Hunt, MD3; Scott Sasser, MD3,4; David Sugerman, MD, MPH3; Mark Faul, PhD, MA3; Terica Scott, MA3; Jeffrey Sokolow, MA1; and Tanja Popovic, MD, PhD2
1-Centers for Disease Control and Prevention (CDC), Morbidity and Mortality Weekly Report; 2-CDC, Office of the Associate Director for Science; 3-CDC, National Center for Injury Prevention and Control; and 4- Emory University School of Medicine
Morbidity and Mortality Weekly Report (MMWR)
Methods
In June 2011, we selected “Guidelines for field triage of injured
patients: recommendations of the National Expert Panel on Field
Triage” to pilot a story-based case study using minimal
bibliometrics. Findings were initially classified by dissemination,
evaluation, implementation, and uptake and adoption.
In January 2012, the Guidelines were updated and republished
with the case study findings. In May 2012, the findings were re-
organized using the CDC Science Impact Framework (adapted
from the Institute of Medicine’s [IOM] Degrees of Impact
model), which includes 5 categories: disseminating science,
creating awareness, catalyzing action, effecting change, and
shaping the future. A single example from each category of the
Framework is presented.
Source: http://www.cdc.gov/mmwr/pdf/rr/rr5801.pdf
Watch CDC Grand Rounds at: http://www.youtube.com/watch?v=EDxBT7ll8Oc&feature=youtu.be

5. www.cdc.gov | Contact CDC at: 1-800-CDC-INFO or www.cdc.gov/info
The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.
Beyond Citations: Introducing a Story-Based Framework to Assess Science Impact
Christine Casey, MD1 ; John Araujo, PhD, MHSA, MA2; Richard Hunt, MD3; Scott Sasser, MD3,4; David Sugerman, MD, MPH3; Mark Faul, PhD, MA3; Terica Scott, MA3; Jeffrey Sokolow, MA1; and Tanja Popovic, MD, PhD2
1-Centers for Disease Control and Prevention (CDC), Morbidity and Mortality Weekly Report; 2-CDC, Office of the Associate Director for Science; 3-CDC, National Center for Injury Prevention and Control; and 4- Emory University School of Medicine
Morbidity and Mortality Weekly Report (MMWR)
CDC Science Impact Framework
Source: http://www.iom.edu/About-IOM/~/media/Files/About%20the%20IOM/2013-
Presidents-Supplement.pdf

6. https://www.cdc.gov/grand-rounds/pp/2014/20140121-presentation-science-impact-H.pdf

7. www.cdc.gov | Contact CDC at: 1-800-CDC-INFO or www.cdc.gov/info
The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.
Results
Disseminating science: The Guidelines were reprinted in Journal of
Emergency Medical Services (circulation ~51,000).
Creating awareness: In 2010, the National Association of Emergency
Medical Services Physicians and American College of Surgeons Committee
on Trauma issued a joint position paper recommending adoption of the
Guidelines.
Catalyzing action: Using the National Trauma Databank and the National
Hospital Ambulatory Medical Care Survey, a cost impact analysis
concluded that full implementation of the Guidelines would produce
savings of ~$568 million/year.
Effecting change: The National Registry of Emergency Medical Technicians
(EMT) adopted the Guidelines as a standard for the certification
examination for ~270,000 EMTs.
Shaping the future: Authors emphasized the importance of linking
nationally representative databases and called for the application of
uniform case definitions and data standards to assess and improve all
phases of care.
Beyond Citations: Introducing a Story-Based Framework to Assess Science Impact
Christine Casey, MD1 ; John Araujo, PhD, MHSA, MA2; Richard Hunt, MD3; Scott Sasser, MD3,4; David Sugerman, MD, MPH3; Mark Faul, PhD, MA3; Terica Scott, MA3; Jeffrey Sokolow, MA1; and Tanja Popovic, MD, PhD2
1-Centers for Disease Control and Prevention (CDC), Morbidity and Mortality Weekly Report; 2-CDC, Office of the Associate Director for Science; 3-CDC, National Center for Injury Prevention and Control; and 4- Emory University School of Medicine
Morbidity and Mortality Weekly Report (MMWR)

8. www.cdc.gov | Contact CDC at: 1-800-CDC-INFO or www.cdc.gov/info
The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.
Beyond Citations: Introducing a Story-Based Framework to Assess Science Impact
Christine Casey, MD1 ; John Araujo, PhD, MHSA, MA2; Richard Hunt, MD3; Scott Sasser, MD3,4; David Sugerman, MD, MPH3; Mark Faul, PhD, MA3; Terica Scott, MA3; Jeffrey Sokolow, MA1; and Tanja Popovic, MD, PhD2
1-Centers for Disease Control and Prevention (CDC), Morbidity and Mortality Weekly Report; 2-CDC, Office of the Associate Director for Science; 3-CDC, National Center for Injury Prevention and Control; and 4- Emory University School of Medicine
Morbidity and Mortality Weekly Report (MMWR)
Centers for Disease Control and Prevention. Guidelines for Field
Triage of Injured Patients Recommendations of the National
Expert Panel on Field Triage, 2011. MMWR 2012;61(No. RR-
61):[1-21]. https://www.cdc.gov/mmwr/pdf/rr/rr6101.pdf

9. www.cdc.gov | Contact CDC at: 1-800-CDC-INFO or www.cdc.gov/info
The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.
Conclusion
The Science Impact Framework is a viable alternative
to (or can augment) conventional bibliometrics.
We recommend that authors, editors and publishers
consider adopting the framework as a vehicle to
assess and communicate scientific impact.
Evaluation and revision of the Framework should be
ongoing to improve its utility.
Beyond Citations: Introducing a Story-Based Framework to Assess Science Impact
Christine Casey, MD1 ; John Araujo, PhD, MHSA, MA2; Richard Hunt, MD3; Scott Sasser, MD3,4; David Sugerman, MD, MPH3; Mark Faul, PhD, MA3; Terica Scott, MA3; Jeffrey Sokolow, MA1; and Tanja Popovic, MD, PhD2
1-Centers for Disease Control and Prevention (CDC), Morbidity and Mortality Weekly Report; 2-CDC, Office of the Associate Director for Science; 3-CDC, National Center for Injury Prevention and Control; and 4- Emory University School of Medicine
Morbidity and Mortality Weekly Report (MMWR)

10. https://www.cdc.gov/od/science/impact/index.htm

11. https://www.cdc.gov/od/science/impact/framework.html

12. https://www.cdc.gov/od/science/impact/testing.html
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC114149/

13. In 2001, CDC published an article in
the Journal of Virology showing that a single dose of a
DNA vaccine could prevent WNV infection in mice and
horses
In 2002, the California Condor
Recovery Team learned that
an experimental CDC DNA
WNV vaccine was proven to
protect against WNV infection
in several bird species
CDC expedited the delivery of
the equine WNV vaccine
In 2004, the vaccine was shown
to be safe and effective in
protecting captive condors from
naturally circulating WNV
Disseminating
Science
Creating
Awareness
Catalyzing
Action
Effecting
Change
Shaping the
Future
In 2005, the CDC equine DNA vaccine was
licensed by the USDA
This led to a Phase 1 clinical trial of a similar
DNA vaccine in humans
The vaccine was shown to induce T-cell and
antibody responses at levels shown to be
protective in studies of horses
In 2011, the CDC/NIH DNA vaccine
was further modified
It demonstrated an enhanced T-cell
response in Phase 1 clinical trials
Quick Stats
Total Citations 193
2nd Generation Citations 4,210
5 Year Impact Factor 5.257
Average Cites Per Year 16.17
WEST NILE VIRUS VACCINE
CASE STUDY
https://www.cdc.gov/od/science/impact/docs/wnv-case-study.pptx

14. https://www.cdc.gov/od/science/impact/use-framework.html
FRAMEWORK USES
Prospectively
-Funding announcement
-Guidelines
Retrospectively
-Manuscript
-Projects

15. https://www.cdc.gov/od/science/impact/application.html
“Degree of impact is
not necessarily a
progression…”

16. https://www.cdc.gov/od/science/impact/faq.html

17. WEBSITE
https://www.cdc.gov/grand-rounds/pp/2014/20140121-science-impact.html
SLIDE DECKS
https://www.cdc.gov/grand-rounds/pp/2014/20140121-presentation-science-impact-H.pdf

18. Public health plays an important role in ensuring access to interventions that can prevent disease, including the implementation of evidence-based genomic recommendations. We
used the Centers for Disease Control and Prevention (CDC) Science Impact Framework to trace the impact of public health activities and partnerships on the
implementation of the 2009 Evaluation of Genomic Applications in Practice and Prevention (EGAPP) Lynch Syndrome screening recommendation
and the 2005 and 2013 United States Preventive Services Task Force (USPSTF) BRCA1 and BRCA2 testing recommendations. The EGAPP and USPSTF
recommendations have each been cited by >300 peer-reviewed publications.
CDC funds selected states to build capacity to integrate these recommendations into public health programs, through education, policy, surveillance, and partnerships. Most state
cancer control plans include genomics-related goals, objectives, or strategies. Since the EGAPP recommendation, major public and private payers now provide coverage for Lynch
Syndrome screening for all newly diagnosed colorectal cancers. National guidelines and initiatives, including Healthy People 2020, included similar recommendations and cited the
EGAPP and USPSTF recommendations. However, disparities in implementation based on race, ethnicity, and rural residence remain challenges. Public health achievements in
promoting the evidence-based use of genomics for the prevention of hereditary cancers can inform future applications of genomics in public health.
Genetics in Medicine (2018) https://doi.org/10.1038/s41436-018-
0028-2

19. Abstract: In 2011, the Division of Cancer Prevention and Control (DCPC), at the United States Centers for Disease Control and Prevention (CDC),
released a three-year funding opportunity announcement (FOA) for a competitive, non-research cooperative agreement. The agreement
enhanced the capacities of state health departments to promote the application of best practices for evidence-based breast cancer genomics
through education, surveillance, and policy activities. The FOA required that applicants focus on activities related to hereditary breast and ovarian
cancer (HBOC). The DCPC funded three states: Georgia, Michigan, and Oregon. Georgia was a first-time recipient of cancer genomics funding,
whereas Michigan and Oregon had long standing activities in cancer genomics and had received CDC funding in the past. By the end of the funding
period, each state had well-functioning and impactful state-based programs in breast cancer genomics. This article highlights the impact of a few
key state activities by using CDC’s Science Impact Framework. There were challenges to implementing public health genomics programs, including
the need to develop relevant partnerships, the highly technical nature of the subject matter, a lack of genetic services in certain areas, and the
difficulty in funding genetic services. Georgia, Michigan, and Oregon have served as models for others interested in initiating or expanding cancer
genomics programs, and they helped to determine what works well for promoting and integrating public health genomics into existing systems.
Healthcare
ISSN 2227-9032
www.mdpi.com/journal/healthcare