Pittsburgh Nonprofit Summit - Measuring Change - Supplemental Slides

Measuring Change Moving from Outputs to Outcomes to Impact 10:00am – 11:15am Facilitator: Lisa Kuzma, Ri h d Ki M ll F d ti F ilit t Li K Richard King Mellon Foundation Speakers: Mike Bangser, MDRC / Junlei Li, Office of ChildDevelopment / Art Maxwell and Pam Meadowcroft, Meadowcroft & Associates / Tim Weidemann, Rondout Consulting

Converting YOUR Program to a Valid EBP: Fidelity Management V lid EBP Fid li M Meadowcroft & Associates and Wesley Spectrum Services For more information, please contact Pamela Meadowcroft, Ph.D. at pmeadowcroft@aol.com Or 412.683.7275 Meadowcroft&Associates and5/20/2011 Wesley Spectrum (c) 2

Evolution of what payers want AND ways providers improved programs id i d Old days: Then Came: PROCESS OUTCOMES Near F t N Future: EBP+ Then EBP: process+ continuous improvement outcomes (assess-plan-monitor- improve) i ) Meadowcroft&Associates and5/20/2011 Wesley Spectrum (c) 3

What is the Difference? Levels of confidence in the outcomes Evidence-based practice (rigorously evaluated; most often proven via RCT) Evidence-informed practice/research-based E id i f d ti / hb d (existing research support) Best Practices (expert opinion) Promising practice (acceptable treatments, anecdotal) ) Innovations Intuition, “the way it’s always done” , y y Meadowcroft&Associates and5/20/2011 Wesley Spectrum (c) 4

WE KNOW A LOT ABOUT WHAT WORKS!!!!! Meta-analyses on thousands of studies Many programs ARE using research- based practices They just haven’t MEASURED and TRACKED their work!!! Meadowcroft&Associates and5/20/2011 Wesley Spectrum (c) 5

Mark Lipsey, “Evidence-based Practice More than One Approach.” MSTand FFT (two brand-names) show positive results the dark boxes but even results, boxes, “generic” interventions showed better results. From http://cjjr.georgetown.edu/pdfs/ebp/ebppaper.pdf Meadowcroft&Associates and5/20/2011 Wesley Spectrum (c) 6

Wesley Spectrum In Home: History of Tracking Outcomes Clients Discharged to Same or Less Restrictive Environments 100% 100% 100% 91% 93% 92% 93% 89% 90% 90% 90% 86% 86% 82% 80% 70% 60% 50% Allegheny 40% Westmoreland 30% 20% 10% 0% FY 2009 Qtr 1 FY 2009 Qtr 2 FY 2009 Qtr 3 FY 2009 Qtr 4 FY 2010 Qtr 1 FY 2010 Qtr 2 Qtr. Qtr. Qtr. Qtr. Qtr. Qtr. 5/20/2011 Meadowcroft&Associates and 75/20/2011 Wesley Spectrum (c) 7

But… Wh G d O t B t Why Good Outcomes? ? Easier population? OR Something we are DOING (our interventions/program model)? / d l) In other words: TRACKING OUTCOMES IS NOT ENOUGH Meadowcroft&Associates and5/20/2011 Wesley Spectrum (c) 8

Ideal Results Id l R lt High “fidelity” to the model leads to fidelity the best outcomes 45.00 45 00 40.00 35.00 30.00 25.00 Model Adherence Scores 20.00 Change in Outcomes 15.00 Scores 10.00 5.00 0.00 Family 1 Family 2 Family 3 Family 4 5/20/2011 Meadowcroft&Associates and 95/20/2011 Wesley Spectrum (c) 9

Steps for Building a local EBP: Fidelity Management Define the program f Develop and Track Model Fidelity (outputs) ( ) Develop and Monitor Outcomes Validate the Locally-Developed Program Model (link outputs to outcomes) ) Build-in CQI Meadowcroft&Associates and5/20/2011 Wesley Spectrum (c) 10

Tools for A T l f Assuring Model Fid lit i M d l Fidelity Therapist and S Th i d Supervisor Ch kli i Checklists (Intake, Monthly, Discharge) include: Who we are serving (population assessments) What are we doing (outputs related to key activities, intensity of services) How’d we do (client outcomes) Consumer Satisfaction Surveys Items relate to key program activities; e s e a e o ey p og a ac es; additional output measures Embed in CQI (participating in QII) 5/20/2011 Meadowcroft&Associates and 115/20/2011 Wesley Spectrum (c) 11

Model Adherence Scores for Wesley Spectrum In Home Model Elements: Two Sites Compared to Be Used for CQI 4 4 4 3.75 3.75 3.75 3.75 3.6 36 3.5 3.5 3.5 3.5 3.5 3.5 3.25 3 3 2.5 25 2 1.5 Westmoreland n=4 Allegheny n=10 1 0.5 0 5/20/2011 Meadowcroft&Associates and 125/20/2011 Wesley Spectrum (c) 12

Higher Model Fidelity Improved Child Well-Being: Strong Relationship g g p between Outputs and Outcomes NCFAS-G Child Well-Being Change Scores0.70.60.50.4 Lower Adherence Group0.303 Higher Adherence0.2 Group0.1 0 Overall Child Childs School Childs Motivation to Well-Being Behavior Performance Relationship Maintain Family with Parents Overall Child Well-Being and Childs Behavior significant at p<.05 level Meadowcroft&Associates and5/20/2011 Wesley Spectrum (c) 13

Wesley Spectrum In Home VS Multi- Systemic Therapy ( y py (name-brand EBP) with ) Comparable Populations Percent of Consumers Who Achieved Outcomes0.960.920.880.84 MST 0.8 WSIH-All Cases0.76 WSIH-Higher Adherence Cases0.720.68 Completed No Arrests / Child in Home Child in School Therapy / Planned Planned Discharge Discharge Child In School significant for MST vs. WSIH Higher (p<.05). Meadowcroft&Associates and5/20/2011 Wesley Spectrum (c) 14

Brand-name EBP vs Local-developed EBP Purchased EBP Model Home Grown Model Building Process $millions for research Low-cost research and and evaluation evaluation in short time short-time Many decades Moderate level program research/development requirements Highly prescribed Lower program cost Low adaptability Greater utility across populations High effort Embedded in CQI Ongoing high program Cost (e.g., C t( Tools for incorporating new practices recertification) Staff commitment Meadowcroft&Associates and5/20/2011 Wesley Spectrum (c) 15

Key Conclusions K C l i Evidence b E id based models pose li it ti d d l limitations th t our that model building process does not Our model building process is replicable so other programs could do the same The process gives programs supervision and monitoring tools for continuous improvement AND for making the case of value to stakeholders Meadowcroft&Associates and5/20/2011 Wesley Spectrum (c) 16

Pittsburgh Nonprofit Summit - Measuring Change - Supplemental Slides

by GPNP on May 20, 2011

Accessibility

Tags

Upload Details

Usage Rights

Statistics

Pittsburgh Nonprofit Summit - Measuring Change - Supplemental Slides — Presentation Transcript