Your SlideShare is downloading. ×
  • Like
Pittsburgh Nonprofit Summit - Measuring Change - Supplemental Slides
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Pittsburgh Nonprofit Summit - Measuring Change - Supplemental Slides


This set of slides contains supplemental information shared by Pam Meadowcroft of Meadowcroft & Associates.

This set of slides contains supplemental information shared by Pam Meadowcroft of Meadowcroft & Associates.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads


Total Views
On SlideShare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. Measuring Change Moving from Outputs to Outcomes to Impact 10:00am – 11:15am Facilitator: Lisa Kuzma, Ri h d Ki M ll F d ti F ilit t Li K Richard King Mellon Foundation Speakers: Mike Bangser, MDRC / Junlei Li, Office of ChildDevelopment / Art Maxwell and Pam Meadowcroft, Meadowcroft & Associates / Tim Weidemann, Rondout Consulting
  • 2. Converting YOUR Program to a Valid EBP: Fidelity Management V lid EBP Fid li M Meadowcroft & Associates and Wesley Spectrum Services For more information, please contact Pamela Meadowcroft, Ph.D. at Or 412.683.7275 Meadowcroft&Associates and5/20/2011 Wesley Spectrum (c) 2
  • 3. Evolution of what payers want AND ways providers improved programs id i d Old days: Then Came: PROCESS OUTCOMES Near F t N Future: EBP+ Then EBP: process+ continuous improvement outcomes (assess-plan-monitor- improve) i ) Meadowcroft&Associates and5/20/2011 Wesley Spectrum (c) 3
  • 4. What is the Difference? Levels of confidence in the outcomes Evidence-based practice (rigorously evaluated; most often proven via RCT) Evidence-informed practice/research-based E id i f d ti / hb d (existing research support) Best Practices (expert opinion) Promising practice (acceptable treatments, anecdotal) ) Innovations Intuition, “the way it’s always done” , y y Meadowcroft&Associates and5/20/2011 Wesley Spectrum (c) 4
  • 5. WE KNOW A LOT ABOUT WHAT WORKS!!!!! Meta-analyses on thousands of studies Many programs ARE using research- based practices They just haven’t MEASURED and TRACKED their work!!! Meadowcroft&Associates and5/20/2011 Wesley Spectrum (c) 5
  • 6. Mark Lipsey, “Evidence-based Practice More than One Approach.” MSTand FFT (two brand-names) show positive results the dark boxes but even results, boxes, “generic” interventions showed better results. From Meadowcroft&Associates and5/20/2011 Wesley Spectrum (c) 6
  • 7. Wesley Spectrum In Home: History of Tracking Outcomes Clients Discharged to Same or Less Restrictive Environments 100% 100% 100% 91% 93% 92% 93% 89% 90% 90% 90% 86% 86% 82% 80% 70% 60% 50% Allegheny 40% Westmoreland 30% 20% 10% 0% FY 2009 Qtr 1 FY 2009 Qtr 2 FY 2009 Qtr 3 FY 2009 Qtr 4 FY 2010 Qtr 1 FY 2010 Qtr 2 Qtr. Qtr. Qtr. Qtr. Qtr. Qtr. 5/20/2011 Meadowcroft&Associates and 75/20/2011 Wesley Spectrum (c) 7
  • 8. But… Wh G d O t B t Why Good Outcomes? ? Easier population? OR Something we are DOING (our interventions/program model)? / d l) In other words: TRACKING OUTCOMES IS NOT ENOUGH Meadowcroft&Associates and5/20/2011 Wesley Spectrum (c) 8
  • 9. Ideal Results Id l R lt High “fidelity” to the model leads to fidelity the best outcomes 45.00 45 00 40.00 35.00 30.00 25.00 Model Adherence Scores 20.00 Change in Outcomes 15.00 Scores 10.00 5.00 0.00 Family 1 Family 2 Family 3 Family 4 5/20/2011 Meadowcroft&Associates and 95/20/2011 Wesley Spectrum (c) 9
  • 10. Steps for Building a local EBP: Fidelity Management Define the program f Develop and Track Model Fidelity (outputs) ( ) Develop and Monitor Outcomes Validate the Locally-Developed Program Model (link outputs to outcomes) ) Build-in CQI Meadowcroft&Associates and5/20/2011 Wesley Spectrum (c) 10
  • 11. Tools for A T l f Assuring Model Fid lit i M d l Fidelity Therapist and S Th i d Supervisor Ch kli i Checklists (Intake, Monthly, Discharge) include: Who we are serving (population assessments) What are we doing (outputs related to key activities, intensity of services) How’d we do (client outcomes) Consumer Satisfaction Surveys Items relate to key program activities; e s e a e o ey p og a ac es; additional output measures Embed in CQI (participating in QII) 5/20/2011 Meadowcroft&Associates and 115/20/2011 Wesley Spectrum (c) 11
  • 12. Model Adherence Scores for Wesley Spectrum In Home Model Elements: Two Sites Compared to Be Used for CQI 4 4 4 3.75 3.75 3.75 3.75 3.6 36 3.5 3.5 3.5 3.5 3.5 3.5 3.25 3 3 2.5 25 2 1.5 Westmoreland n=4 Allegheny n=10 1 0.5 0 5/20/2011 Meadowcroft&Associates and 125/20/2011 Wesley Spectrum (c) 12
  • 13. Higher Model Fidelity Improved Child Well-Being: Strong Relationship g g p between Outputs and Outcomes NCFAS-G Child Well-Being Change Scores0. Lower Adherence Group0.303 Higher Adherence0.2 Group0.1 0 Overall Child Childs School Childs Motivation to Well-Being Behavior Performance Relationship Maintain Family with Parents Overall Child Well-Being and Childs Behavior significant at p<.05 level Meadowcroft&Associates and5/20/2011 Wesley Spectrum (c) 13
  • 14. Wesley Spectrum In Home VS Multi- Systemic Therapy ( y py (name-brand EBP) with ) Comparable Populations Percent of Consumers Who Achieved Outcomes0.960.920.880.84 MST 0.8 WSIH-All Cases0.76 WSIH-Higher Adherence Cases0.720.68 Completed No Arrests / Child in Home Child in School Therapy / Planned Planned Discharge Discharge Child In School significant for MST vs. WSIH Higher (p<.05). Meadowcroft&Associates and5/20/2011 Wesley Spectrum (c) 14
  • 15. Brand-name EBP vs Local-developed EBP Purchased EBP Model Home Grown Model Building Process $millions for research Low-cost research and and evaluation evaluation in short time short-time Many decades Moderate level program research/development requirements Highly prescribed Lower program cost Low adaptability Greater utility across populations High effort Embedded in CQI Ongoing high program Cost (e.g., C t( Tools for incorporating new practices recertification) Staff commitment Meadowcroft&Associates and5/20/2011 Wesley Spectrum (c) 15
  • 16. Key Conclusions K C l i Evidence b E id based models pose li it ti d d l limitations th t our that model building process does not Our model building process is replicable so other programs could do the same The process gives programs supervision and monitoring tools for continuous improvement AND for making the case of value to stakeholders Meadowcroft&Associates and5/20/2011 Wesley Spectrum (c) 16