1. Measuring Change
Moving from Outputs to Outcomes to Impact
10:00am – 11:15am
Facilitator: Lisa Kuzma, Ri h d Ki M ll F d ti
F ilit t Li K Richard King Mellon Foundation
Speakers: Mike Bangser, MDRC / Junlei Li, Office of Child
Development / Art Maxwell and Pam Meadowcroft, Meadowcroft
& Associates / Tim Weidemann, Rondout Consulting

2. Converting YOUR Program to a
Valid EBP: Fidelity Management
V lid EBP Fid li M
Meadowcroft & Associates and
Wesley Spectrum Services
For more information, please contact Pamela
Meadowcroft, Ph.D. at pmeadowcroft@aol.com
Or 412.683.7275
Meadowcroft&Associates and
5/20/2011 Wesley Spectrum (c) 2

3. Evolution of what payers want AND ways
providers improved programs
id i d
Old days: Then Came:
PROCESS OUTCOMES
Near F t
N Future: EBP+ Then EBP:
process+
continuous improvement outcomes
(assess-plan-monitor-
improve)
i )
Meadowcroft&Associates and
5/20/2011 Wesley Spectrum (c) 3

4. What is the Difference?
Levels of confidence in the outcomes
Evidence-based practice (rigorously
evaluated; most often proven via RCT)
Evidence-informed practice/research-based
E id i f d ti / hb d
(existing research support)
Best Practices (expert opinion)
Promising practice (acceptable treatments,
anecdotal) )
Innovations
Intuition, “the way it’s always done”
, y y
Meadowcroft&Associates and
5/20/2011 Wesley Spectrum (c) 4

5. WE KNOW A LOT ABOUT WHAT
WORKS!!!!!
Meta-analyses on thousands of studies
Many programs ARE using research-
based practices
They just haven’t MEASURED and
TRACKED their work!!!
Meadowcroft&Associates and
5/20/2011 Wesley Spectrum (c) 5

6. Mark Lipsey, “Evidence-based Practice More than One Approach.” MST
and FFT (two brand-names) show positive results the dark boxes but even
results, boxes,
“generic” interventions showed better results.
From http://cjjr.georgetown.edu/pdfs/ebp/ebppaper.pdf
Meadowcroft&Associates and
5/20/2011 Wesley Spectrum (c) 6

7. Wesley Spectrum In Home: History of
Tracking Outcomes
Clients Discharged to Same or Less Restrictive
Environments
100% 100%
100%
91% 93% 92% 93%
89% 90% 90%
90% 86% 86%
82%
80%
70%
60%
50% Allegheny
40% Westmoreland
30%
20%
10%
0%
FY 2009 Qtr 1 FY 2009 Qtr 2 FY 2009 Qtr 3 FY 2009 Qtr 4 FY 2010 Qtr 1 FY 2010 Qtr 2
Qtr. Qtr. Qtr. Qtr. Qtr. Qtr.
5/20/2011 Meadowcroft&Associates and 7
5/20/2011 Wesley Spectrum (c) 7

8. But… Wh G d O t
B t Why Good Outcomes?
?
Easier population? OR
Something we are DOING (our
interventions/program model)?
/ d l)
In other words: TRACKING
OUTCOMES IS NOT ENOUGH
Meadowcroft&Associates and
5/20/2011 Wesley Spectrum (c) 8

9. Ideal Results
Id l R lt
High “fidelity” to the model leads to
fidelity
the best outcomes
45.00
45 00
40.00
35.00
30.00
25.00 Model Adherence Scores
20.00
Change in Outcomes
15.00 Scores
10.00
5.00
0.00
Family 1 Family 2 Family 3 Family 4
5/20/2011 Meadowcroft&Associates and 9
5/20/2011 Wesley Spectrum (c) 9

10. Steps for Building a local EBP: Fidelity
Management
Define the program
f
Develop and Track Model Fidelity
(outputs)
( )
Develop and Monitor Outcomes
Validate the Locally-Developed
Program Model (link outputs to
outcomes) )
Build-in CQI
Meadowcroft&Associates and
5/20/2011 Wesley Spectrum (c) 10

11. Tools for A
T l f Assuring Model Fid lit
i M d l Fidelity
Therapist and S
Th i d Supervisor Ch kli
i Checklists
(Intake, Monthly, Discharge) include:
Who we are serving (population assessments)
What are we doing (outputs related to key
activities, intensity of services)
How’d we do (client outcomes)
Consumer Satisfaction Surveys
Items relate to key program activities;
e s e a e o ey p og a ac es;
additional output measures
Embed in CQI (participating in QII)
5/20/2011 Meadowcroft&Associates and 11
5/20/2011 Wesley Spectrum (c) 11

12. Model Adherence Scores for Wesley Spectrum
In Home Model Elements: Two Sites
Compared to Be Used for CQI
4 4 4
3.75 3.75 3.75 3.75
3.6
36
3.5 3.5 3.5 3.5 3.5
3.5
3.25
3
3
2.5
25
2
1.5 Westmoreland n=4
Allegheny n=10
1
0.5
0
5/20/2011 Meadowcroft&Associates and 12
5/20/2011 Wesley Spectrum (c) 12

13. Higher Model Fidelity Improved Child
Well-Being: Strong Relationship
g g p
between Outputs and Outcomes
NCFAS-G Child Well-Being Change Scores
0.7
0.6
0.5
0.4 Lower Adherence
Group
0.3
03
Higher Adherence
0.2 Group
0.1
0
Overall Child Child's School Child's Motivation to
Well-Being Behavior Performance Relationship Maintain Family
with Parents
Overall Child Well-Being and Child's Behavior significant at p<.05 level
Meadowcroft&Associates and
5/20/2011 Wesley Spectrum (c) 13

14. Wesley Spectrum In Home VS Multi-
Systemic Therapy (
y py (name-brand EBP) with
)
Comparable Populations
Percent of Consumers Who Achieved Outcomes
0.96
0.92
0.88
0.84 MST
0.8 WSIH-All Cases
0.76 WSIH-Higher Adherence
Cases
0.72
0.68
Completed No Arrests / Child in Home Child in School
Therapy / Planned Planned Discharge
Discharge
Child In School significant for MST vs. WSIH Higher (p<.05).
Meadowcroft&Associates and
5/20/2011 Wesley Spectrum (c) 14

15. Brand-name EBP vs Local-developed EBP
Purchased EBP Model Home Grown Model
Building Process
$millions for research Low-cost research and
and evaluation evaluation in short time
short-time
Many decades Moderate level program
research/development requirements
Highly prescribed Lower program cost
Low adaptability Greater utility across
populations
High effort
Embedded in CQI
Ongoing high program
Cost (e.g.,
C t( Tools for incorporating
new practices
recertification)
Staff commitment
Meadowcroft&Associates and
5/20/2011 Wesley Spectrum (c) 15

16. Key Conclusions
K C l i
Evidence b
E id based models pose li it ti
d d l limitations th t our
that
model building process does not
Our model building process is replicable so other
programs could do the same
The process gives programs supervision and
monitoring tools for continuous improvement AND
for making the case of value to stakeholders
Meadowcroft&Associates and
5/20/2011 Wesley Spectrum (c) 16