Digital Scholar Webinar: Getting to Outcomes: Supporting Implementation of Evidence-Based Programs

Getting To Outcomes®: Supporting
Implementation of Evidence-Based Programs
Matthew Chinman
3/3/2021
USC Digital Scholar lecture

Agenda
Background in
Implementation Science
The Getting To Outcomes
approach
Early Getting To
Outcomes Studies
Later Getting To
Outcomes Studies

EBPs not
often used,
or not
applied
equally
across
groups
• Drug prevention - DOE survey of 6K
schools:
– only 8% used EBPs
– less than half of those
implemented with fidelity
Department of Education. Prevalence and
implementation fidelity of research-based prevention
programs in public schools: Final report. Washington,
DC, 2011.
• Disparities
– Blacks and Hispanics received
worse care than Whites for about
40% of quality measures
– Poor people received worse care
than high-income people for
about 60% of quality measures.
AHRQ. 2013 National Healthcare Disparities Report

Definitions of Implementation Science
“…The scientific study of methods to
promote the systematic uptake of research
findings and other evidence-based practices
into routine practice…It includes the study
of influences on healthcare professional and
organizational behavior.”
(Eccles & Mittman, 2006)

National Research Council and Institute of Medicine, 2009, p. 326
Phases of Clinical Research
How to deliver?
What to deliver?
What to target?

Implementation
Outcomes
Feasibility
Fidelity
Penetration
Acceptability
Sustainability
Uptake
Costs
*IOM Standards of Care
What?
QIs
ESTs
How?
Implementation
Strategies
Implementation Research Methods
Service
Outcomes*
Efficiency
Safety
Effectiveness
Equity
Patient-
centeredness
Timeliness
Patient
Outcomes
Clinical/health
status
Symptoms
Function
Satisfaction
CONTEXT
CONTEXT
CONTEXT
CONTEXT
The Usual
The Core of
Implementation
Science
Conceptual model of implementation
research
Proctor et al. (2009)

• Acceptability
• Adoption
• Appropriateness
• Feasibility
• Fidelity
• Implementation cost
• Penetration
• Sustainability
Implementation outcomes

Agenda
Background in
Implementation Science
The Getting To
Outcomes approach
Early Getting To
Outcomes Studies
Later Getting To
Outcomes Studies

Getting to ‘Getting To Outcomes’
No
outcome
s
 Community Collaboratives for Drug
Prevention
 $.5 Billion
 251 communities
 South Carolina
 4 communities
 $8 Million
 Well-intentioned practitioners
 Bad choices

Objectives
Can we help
community-based
organizations
implement better?
What methods
can we use to
research this
question?

GTO designed to build capacity for
high quality program
implementation
AND
GTO is a model
that poses ten steps that must
be addressed in order to obtain
positive results
GTO is an implementation support
provides practitioners with the
guidance necessary to complete
those steps with quality

GTO model supports high quality program
implementation in many domains
Choose which
problem(s) to
focus on.
1
Identify goals,
target population,
and desired
outcomes.
2
Find existing
programs and
best practices
worth copying.
3
Modify the program
or best practices
to fit your needs.
4
Assess capacity
(staff, financing,
etc.) to implement
the program.
5
Make a plan
for getting started:
who, what, when,
where, and how.
6
Steps 1-6
PLANNING
DELIVERING PROGRAMS
Evaluate
planning and
implementation.
How did it go?
7
Evaluate
program’s success
in achieving
desired results.
8
Make a plan for
Continuous
Quality
Improvement.
9
Consider how to
keep the program
going if it is
successful.
10
Steps 7-10
EVALUATING AND
IMPROVING

#1
Needs/
Resources
#2
Goals &
Objectives
#3
Best
Practices #4
Fit
#5
Capacities
#6
Plan
#7
Implementation/
Process
Evaluation
#8
Outcome
Evaluation
#9
Improve /
CQI
#10
Sustain
RESULTS
GTO is not always liner

Use GTO to plan ANYTHING
Like a vacation

Getting To Outcomes has 3 parts
Tools Training Technical Assistance (TA

Manuals
have text
& tools
that guide
users
through
the 10
steps

“Goals Tool” helps organizations set
concrete benchmarks
Behavior or Determinant SMART Desired Outcome Statement
Aligned with:
 Recent sexual activity
 Number of sexual partners
 Frequency of sexual activity
 Contraceptive use and/or use consistency
 Sexual initiation and abstinence
 Pregnancy or birth
 STIs (including HIV)
SMART Checklist
S
M
A
R
T
Intention to practice abstinence At the completion of the program, 80% of participants’ will report
that they plan to abstain from sex for the next 90 days.
Aligned with:
 Recent sexual activity
 Number of sexual partners
 Frequency of sexual activity
 Contraceptive use and/or use consistency
 Sexual initiation and abstinence
 Pregnancy or birth
 STIs (including HIV)
SMART Checklist
Specific- Plans to abstain from sex for
the next 90 days
Measurable- 80% at post survey
Achievable- Abstinence promotion is in
line with program goals
Realistic- Similar youth have achieved
this Desired Outcome before
Time-bound- By the completion of the
program

Face to face
training
builds
knowledge
and basic
skills
• Involves learning about how
to apply 10 steps in their
programs
• Walk through manual
• Learn about various tools to
accomplish 10 steps
• Tailor to local program needs
• OLD. Often lasts a full-day or
longer, often with follow-up
• NEW. 3 blocks: GTO 1-3, 4-6,
7-10

Technical
Assistance (TA) is
like having an
expert in the
driver’s seat
• Who provides TA?
From PhD to BA. From
full to part time.
• Who receives TA?
Usually program
coordinators
• When? Varies. Weekly
to 2X/month via
meetings, phone,
email. Ongoing over 1
to 2 yrs.
• What work is done?
Depends on the focus.
Make existing
programs better vs.
start new evidence-
based program
20

Early GTO studies focused on
capacity and performance
Implementation
Support
(i.e., GTO)
Capacity
(knowledge,
attitudes,
skills)
Performance
(e.g., goal setting,
ensuring fit &
capacity, planning,
evaluation, quality
improvement)
Chinman, M., Acosta, J., Ebener, P., Malone, P.S., Slaughter M. (2016). Can implementation-support help community-based
settings better deliver evidence-based sexual health promotion programs: A randomized trial of Getting To Outcomes®.
Implementation Science, 11, 78.
GTO logic model

GTO Study GTO pilot Assets GTO
Sites Two coalitions (SC, CA):
6 GTO programs (SC=2, CA=4)
vs
4 non-GTO programs (SC=2, CA=2)
12 coalitions in Maine:
6 prevention coalitions (5 programs each)
vs
6 prevention coalitions (5 programs each)
Design Quasi-experimental by state Cluster-randomized by coalition
Measures • Prevention capacity (individual level)
• Program performance (program level)
• Acceptability (qualitatively)
TA Two, .5 FTE PhD psychologists Two, 1.0 FTE BA prevention specialists
Program
type, quality
pre-existing programs of varying type,
quality
pre-existing programs of varying type,
quality (pos. youth development or
‘Assets’)
GTO step
completion
Select steps via consensus between TA and site
Key findings
• Program performance – GTO superior (evaluation steps improved most)
• Prevention capacity – ITT – ns (modest GTO use); More GTO use = higher
capacity
• liked GTO, but required time
Additional
findings
Relationship between greater capacity
(i.e., knowledge) and better performance
while controlling for leadership, ability to

Lessons learned and implications for
implementation support
 Local communities, with support, can implement
GTO
 Existing capacity/resources can make it difficult to
adopt
 Got to be in it to win it: Users get more benefit
 Evaluation support became highly valued
 GTO spurred evaluation skills being sustained
through a community-university partnership
 Difficult to affect de-implementation of poor programs
 Ongoing implementation support is helpful, but
accountability and consistent resources are also
needed
 IMPACT ON OUTCOMES?

Bottom line: Communities could run
programs better, but…..

Later GTO studies link support to
outcomes
Implementation
Support
(i.e., GTO)
Capacity
(knowledge,
attitudes,
skills)
Performance
(e.g., goal setting,
ensuring fit &
capacity, planning,
evaluation, quality
improvement)
Fidelity
(adherence,
delivery,
dose)
Individual
outcomes
Chinman, M., Acosta, J., Ebener, P., Malone, P.S., Slaughter M. (2016). Can implementation-support help community-based
settings better deliver evidence-based sexual health promotion programs: A randomized trial of Getting To Outcomes®.
Implementation Science, 11, 78.
GTO logic model

Two year GTO training and TA
process
•GTO training
(Steps 1-3)
•EBP training
Work w/ TA staff to
set Desired
Outcomes on Goals
tool
• GTO
training
(Steps 4-6)
Work w/ TA staff to
complete Fit,
Capacity, & Plan
tools • Implement
EBP
Data collection
Fidelity
Outcomes
• GTO
Evaluation and
CQI workshop
(Steps 7-9)
Work w/ TA staff to
revise plans for
second cycle
Year 2
•GTO training
(Step 10)
REPEAT

GTO
Study
GTO and Teen Pregnancy GTO and Drug Prevention
Sites Boys & Girls clubs in two states (GA, AL):
16 GTO club sites
vs
16 non-GTO club sites
Boys & Girls clubs in greater Los Angeles:
15 GTO club sites
vs
14 GTO club sites
Design Cluster-randomized by state (EBP v EBP+GTO) Cluster-randomized (EBP v EBP+GTO)
Measures Implementation:
• Fidelity (Curriculum adherence, generic
delivery quality, attendance)
Youth Outcomes: Sex behaviors & knowledge;
attitudes on sex and condoms
Implementation:
• Fidelity (Curriculum adherence, Motiv Interv
delivery quality, attendance)
Youth Outcomes: Drug behaviors, knowledge,
attitudes
TA Two, .5 FTE BA prevention specialists Two, .5 FTE MA prevention specialists
Program
type, quality
Making Proud Choices – evidence-based teen
pregnancy prevention program
Project CHOICE – evidence-based drug
prevention program
GTO step
completion
Complete all steps over 2 year period: Steps 1-6/run program/7-9/redo 1-6/run program/7-10
Key findings • Program performance – GTO superior
• By Year 2, GTO sites superior on
Adherence, Delivery quality
• By Year 2, GTO youth had better condom
attitudes/intentions; other outcomes – n.s.
• Program performance – GTO superior
• By Year 2, GTO sites superior on
Adherence, Delivery quality
• No differences on youth outcomes
Additional
findings
• 2 years post GTO, more GTO sites were
running CHOICE than control sites
• Better GTO performance predicted better
CHOICE fidelity

• GTO sites had better
Knowledge/Performance/ Fidelity
– Highly structured EBP gets modest fidelity, need more
support for high fidelity (only 26 hours over 2 years)
– Despite having sites that were less implementation-friendly
– Intervention bleed – 3rd no contact group—NO
implementation
• Youth outcomes
– Proximal Outcomes
• Teen pregnancy: GTO sites did better on condom attitudes
• Drug use: n.s.
– Behaviors: low base rates make comparisons difficult
• Replicated findings using same design but in
different domain, different measures
Conclusions

Going Digital
• All tools are in digital form
• What is needed is to turn GTO into an
interactive website
– Users would have an account
– Complete tools online

For More Information
http://www.rand.org/gto
Matthew Chinman
RAND Corporation
4570 5th Avenue
Pittsburgh, PA 15213
(412) 683-2300 x 4287
chinman@rand.org

Digital Scholar Webinar: Getting to Outcomes: Supporting Implementation of Evidence-Based Programs

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Digital Scholar Webinar: Getting to Outcomes: Supporting Implementation of Evidence-Based Programs

Similar to Digital Scholar Webinar: Getting to Outcomes: Supporting Implementation of Evidence-Based Programs (20)

More from SC CTSI at USC and CHLA

More from SC CTSI at USC and CHLA (20)

Recently uploaded

Recently uploaded (20)

Digital Scholar Webinar: Getting to Outcomes: Supporting Implementation of Evidence-Based Programs