Samantha Harden discuss provides an overview of the RE_AIM framework which evaluates the effectiveness of interventions based on the following five dimensions: Reach into the target population Effectiveness or efficacy Adoption by target settings, institutions and staff Implementation - consistency and cost of delivery of intervention Maintenance of intervention effects in individuals and settings over time. We will also practice using RE-AIM in planning, implementation, and evaluation and share resources available on RE-AIM.org. Learning Objectives 1. Understand the five RE-AIM dimensions 2. Practice using RE-AIM for planning, implementation, and evaluation 3. Explore available resources found at RE-AIM.org

1. Planning, Implementation, and Evaluation
Using the RE-AIM Framework
HPLive.org Presentation: March 13, 2015
Contributors: Russ E. Glasgow, PhD and Paul A. Estabrooks PhD
Presenter: Samantha M. Harden, PhD, Human Nutrition, Foods and Exercise,
Virginia Tech

2. RE-­‐AIM
Dimensions
Example
Studies
History
of
RE-­‐AIM
Outline

3. Outline
“If we want more evidence-
based practice, we need more
practice-based evidence.”
Green LW. Am J Pub Health 2006

4. • Internal
validity
perspec=ve
§ The
magnitude
of
effect
as
the
key
indicator
of
readiness
for
transla=on
and
adheres
to
the
principles
of
evidence
ra1ng
for
determining
efficacy
• External
validity
perspec=ve
§ ACen=on
to
interven=on
features
that
can
be
adopted
and
delivered
broadly,
have
the
ability
for
sustained
and
consistent
implementa1on
at
a
reasonable
cost,
reach
large
numbers
of
people,
especially
those
who
can
most
benefit,
and
produce
replicable
and
long-­‐las1ng
effects
Brief History of RE-AIM
Glasgow RE, Vogt TM, Boles SM. Evaluating the Public Health Impact…Am J Public Health, 1999;89:1322-1327

5. Original RE-AIM
• First
published
ar=cles
in
1999
• Originally
intended
to
increase
balance
between
internal
and
external
validity
• First
used
to
evaluate
preven=on
and
health
behavior
change
programs
• RE-­‐AIM
Trivia:
was
going
to
be
called
ARIEM
Gaglio B & Glasgow RE. (2012). Evaluation approaches for dissemination and implementation
research. In: R. Brownson, G. Colditz, & E. Proctor (Eds.), Dissemination and implementation
research in health…1st Edition pp. 327-356). New York: Oxford University Press.

6. 6
www.RE-­‐AIM.org

7. A CHOICE?
– Program A: 16 session physical activity
program that produces a 150 minute per
week change in moderate intensity physical
activity for 8 out of every 10 participants
– Program B: 16 session physical activity
program that produces a 150 minute per
week change in moderate physical activity for
2 out of every 10 participants
Which program do you think is better?

8. Answer: It Depends!
— Why does it depend?
– Who delivers?
÷ Program A: Trained master’s level health educators
÷ Program B: Administrative assistants in community health
center
– How easy is it to implement?
÷ Program A: Moderately difficult
÷ Program B: Moderately easy
– What resources?
÷ Program A: Group exercise area and counseling rooms
÷ Program B: Email access and participants can do activities at
home or in neighborhood.
Which is Better?

9. Answer: It Still Depends!
– How scalable is it?
÷ Program A: 20 people can participate per class session which
includes 90 minute counseling session and 3 one hour classes
each week.
÷ Program B: 100 people can participate per session which
includes monitoring of physical activity and sending out
weekly newsletters.
– What does it cost?
÷ Program A: 33 hours/week for 6 months from health educator
for every 16 successes (20 people per group).
÷ Program B: 8 hours/week for 6 months from administrative
assistant for every 20 successes.
– How sustainable are the effects?
÷ Program A: 50% return to baseline activity after 6 months
÷ Program B: 50% return to baseline level after 6 months
Which is Better?

10. RE-AIM: Goals for translating useful interventions into
regular practice
10
• Move from a paradigm that emphasizes:
• The magnitude of effect as the key indicator of readiness
for translation and adheres to the principles of evidence
rating for determining efficacy
• Move to one that emphasizes:
• Attention to intervention features that can be adopted
and delivered broadly, have the ability for sustained and
consistent implementation at a reasonable cost, reach
large numbers of people, especially those who can most
benefit, and produce replicable and long-lasting behavior
changes

11. • Reach
• Effectiveness
• Adoption
• Implementation
• Maintenance
The RE-AIM Framework:
Glasgow et al, AJPH, 1999
What is the RE-AIM Framework?

12. 12
Definition: The number, percent of target
audience, and representativeness of those who
participate.
Data Needed:
Denominator—number of eligible contacted for
potential participation
Numerator—number of eligible that participate
Comparative information on target population
E-AIM ELEMENTS:

13. 13
Example: Move More.
Data:
Denominator—Inactive or insufficiently active adults
going to the doctor for a physical (n=1518 total; 607
eligible; 218 referred)
Numerator—number of eligible that participate
(n=115)
Participation Rate: 115/607=19%
Almeida et al. JSEP 2005
E-AIM ELEMENTS:

14. 14
Almeida et al. JSEP 2005
Enrolled Participants Census
TractTotal
Sample
(n=115)
Stimulus
Control
(n=44)
Standard
Care
(n=71)
Female 60.9% 65.9% 57.7% 51.1%
48.8
(±11.9)
48.6
(±11.3)
48.9
(±12.3)
30.4
White 58.9% 56.8% 60.3% 55.3%
Black 22.3% 27.3% 19.1% 19.8%
Latino 12.5% 6.8% 16.2% 14.8%
Asian 0% 0% 0% 5.6%
E-AIM ELEMENTS:

15. 15
Definition: Change in outcomes and impact on quality
of life and any adverse outcomes
Data needed:
Primary Outcome
Quality of life
Potential negative outcomes
R -AIM ELEMENTS:

16. 16
Example: Family Connections
Data:
Primary Outcome: Significant reductions in
BMI z-score.
Quality of life: Improvement in quality of life
with lower weight status
Potential negative outcomes: No evidence of
heightened eating disordered symptoms
Estabrooks et al. AJPM 2009
Shoup et al. QLR 2008
R -AIM ELEMENTS:

17. Definition: Number, percent and
representativeness of settings and educators who
participate.
Data needed:
Denominator—number of eligible sites contacted
for potential participation
Numerator—number of eligible sites that
participate
Comparative information on target population of
sites
RE- IM ELEMENTS:

18. Example:
Data:
Denominator—105 counties in Kansas eligible to
participate
Numerator—48 agreed; 48/105=46%
Representativeness—Less active agent, less likely
to deliver; Smaller population counties, more
likely to deliver
Estabrooks, Bradshaw, Fox, et al. , AJHP, 2004
Estabrooks, Bradshaw, Dzewaltowski , & Smith-Ray, ABM, 2008
RE- IM ELEMENTS:

19. 19
Definition: Extent to which a program or policy is
delivered consistently, and the time and costs of
the program.
Data needed:
Information on program components and
essential elements
Information on resource use
RE-A M ELEMENTS:

20. Example:
Data:
Variability in delivery of program components
based upon local tailoring of the program.
On average, 80% of program components were
delivered as intended
2.5 hours of delivery agent time per participant
compared to 36 hrs per participant in control
RE-A M ELEMENTS:

21. 21
Definition:
Individual/member target: Long-term effects and
attrition.
Setting/educator: Extent of discontinuation,
modification, or sustainability of program.
Data needed:
Primary outcome assessment 12 months post
intervention
Documented sustained delivery
RE-AI ELEMENTS:

22. 22
Data:
Decreased BMI z-scores sustained 12
months after intervention complete
Data:
RE-AI ELEMENTS:

23. Individual level factors balancing internal
and external validity:
— Shift from focus on the numerator to the denominator
— Generalizability to target population
— Avoid contributing to disparities
— Common comparison for decision making including
unintended consequences
— Robustness when combined with adoption: what
works best for whom, and under what conditions
Summary: Key RE-AIM Issues to Improve
Translational Research

24. Setting level factors balancing internal and
external validity:
— Will the intervention fit in a typical practice setting?
— Generalizability to who will deliver the program
— Initial start-up and ongoing costs
— Understanding structure and who makes adoption
decisions (and how they are made)
— Characteristics of the intervention, setting, culture,
and organization that help or hurt implementation
Summary: Key RE-AIM Issues to Improve
Translational Research

25. RE-AIM: Goals for translating useful interventions into
regular practice
Develop and translate research and practice-
based interventions that can:
be adopted and delivered broadly,
have the ability for sustained and consistent
implementation at a reasonable cost
reach large numbers of people especially those
who can most benefit,
produce replicable and long-lasting behavior
changes

26. One more example!
Integrated Research-Practice Developed versus Pipeline Model in
Physical Activity Programming: A comparative analysis.
Samantha M. Harden, Sallie Beth Johnson, Fabio A. Almeida, Paul A.
Estabrooks
In Preparation

27. Adoption-Based Randomized Control Trial
Integrated Research-Practice Model Efficacy to Effectiveness to
Demonstration to
Dissemination Model
Fit Extension Active Living Everyday

28. Untitled

29. Implementation
0
100
200
300
400
500
600
Fit Ex ALED
Cost assessed as implementation
hours
• Degree delivered as intended
• ALED ~90%
• Fit Ex ~ 80%
• Adaptation
• ALED None reported
• Fit Ex Numerous small changes in feedback timing and
structure

30. Effectiveness

31. — Reach
¡ Fit Ex-75 participants per program
¡ ALED 15 participants per program
¡ Both underrepresented by men
— Maintenance- Delivered for 3 years post initial evaluation
Descriptive Information