Nc live 8 3-10 presentation


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Nc live 8 3-10 presentation

  1. 1. Improving the Quality of Services Through Measurement and Feedback Supported by the National Institute of Mental Health grant (RO1 MH068589) and a grant from the Leon Lowenstein Foundation . Leonard Bickman, Ph.D., Susan Douglas Kelley, Ph.D., and Susan Casey, MSW
  2. 2. Outline • The challenge and opportunity • The theory of change • Contextualized Feedback Systems and its components • Preliminary results of evaluation • Benefits • Plans 1
  3. 3. The Challenge: A Quality Chasm Exists for Mental Health Services 2
  4. 4. “The greatest obstacle to discovery is not ignorance - it is the illusion of knowledge” Daniel Boorstin, 1984 Librarian of Congress 3
  5. 5. Little Research Support for the Effectiveness of Services in Community Settings & Little Change in a Decade • “The findings to date offer little support for the effectiveness of conventional clinical intervention for young people.” (Weisz & Jensen, 1999, p. 133) • Some common elements of effective practice have been observed in treatment as usual but not in the depth and breadth needed for effective treatment. (Garland, Bickman, & Chorpita, 2010) • But lack of evidence doesn’t necessarily mean lack of effectiveness 4
  6. 6. Little Accuracy About Client Progress & Focus of Treatment • No correlation between standardized measures and clinicians’ perceptions of progress (Love et al., 2007) • Difficulty predicting and detecting worsening of symptoms and functioning over the course of treatment (Hannan et al., 2005; Hatfield et al., 2009) – Yet, 89% of clinicians said they would know based on their clinical judgment • Little agreement on target problems by parents and therapists, and children and therapists (Hawley, KM & Weisz, JR. (2003). • Chance Agreement on Level of Care Assignment (Bickman, L., Karver, M., & Schut, L. J. A. (1997). 5
  7. 7. Typical Community Based Care has Significant limitations • “The findings to date offer little support for the effectiveness of conventional clinical intervention for young people.” (Weisz & Jensen, 1999, p. 133) • Some common elements of effective practice have been observed in treatment as usual but not in the depth and breadth needed for effective treatment. (Garland, Bickman, & Chorpita, 2010) • But lack of evidence doesn’t necessarily mean lack of effectiveness 6
  8. 8. Clinical Intuition and Regulations are Insufficient to Produce Good Outcomes • Fifty years of research does not support relying only on clinical judgment for effective practices • Reliance on presumed standards of quality such as licensing and accreditation may also impede development of effective services • Unmonitored “single shot” training on EBTs are not effective 7
  9. 9. Systematic and Accurate Feedback Can Improve Outcomes 8
  10. 10. “Thus, use of patient monitoring and feedback in routine practice is imperative. The focus on the individual rather than the average patient empowers the patient and encourages dialogue about progress, the direction of treatment and achievement of treatment goals”. Newnham & Page. (2010). Bridging the gap between best evidence and best practice in mental health. Clinical Psychology Review, 30, 127–142 (p.138). 9
  11. 11. Strong Support for Feedback Concept • Measurement and feedback are the core of all management and learning theories. • Thousands of studies outside of mental health show that improvement is minimal without measuring performance and providing feedback. • Direct feedback occupations show improvement with experience. However, clinician experience alone is not a good predictor of client outcomes. 10
  12. 12. Providence Service Corporation delivers home and community based social services to government beneficiaries and privatized social service projects. Providence Service Corporation
  13. 13. . Providence Overview PRSC Service Offering Home and Community Management Based Services Services Foster Care Services 12
  14. 14. • Providence Service Corporation was aware of the need for feedback and was committed to implementing a quality enhancement initiative • In 2004 we began a collaboration with Vanderbilt University to combine their efforts to create an evidence-based feedback system 13
  15. 15. Providence Core Values • Community-based and Multi-systemic Services • Identifying and Building on Strengths • Local Viability that is Nationally Supported • Respectful Organizational Culture • Best Practice Models • Cultural Diversity Providence Service Corporation
  16. 16. Quality Enhancement Initiative • 12 states, 38 sites, 287 clinicians • 836 youth 11-18 years of age and their caregivers • 10,431 client weeks of Providence provided services 15
  17. 17. CFS - A Concurrent, Systematic Monitoring and Formative Feedback Practice Improvement Tool Practice Without Feedback Does Not Lead To Improvement 16
  18. 18. CFS is a Practice Improvement Strategy • is part of treatment – provides an opportunity for brief reflection at the close of a session • is primarily designed to support clinicians – provides guidance for those moments when little seems to be working • supports supervisors – helps supervisors identify areas where clinicians need extra guidance to ensure they feel confident • is flexible – optional custom measures, schedules, reports can be tailored for office workflow and QI initiatives • can be used for administrative and reporting functions – provides tools for leadership to become or remain successful at meeting the needs of their funders and clients 17
  19. 19. CFS Theory of Change is on Two Levels • The individual or psychological level • The group or organizational level 18
  20. 20. Key Domains of CFS • System Development • Measurement • Feedback • Training in Common Factors • Comprehensive Support 19
  21. 21. Dashboard-driven System Dashboards are designed for clients and Easy to navigate Highlights common actions by user caregivers, clinicians, supervisors, administrative assistants, program directors, etc. 20
  22. 22. Sample Clinical Alerts New = most recent alerts; not yet viewed Link to the feedback report containing the Pending = oldalert; Archive alerts alert; not yet viewed 21
  23. 23. Sample Client Login After logging in, client is presented with questionnaire to be completed. 22
  24. 24. Barry’s Sample Case – Clinician Judgment is Necessary - What Do You Notice? 23
  25. 25. What do you see here? 24
  26. 26. Weeks 1 2 3 4 5 6 7 25
  27. 27. Measurement: The Peabody Treatment Progress Battery (PTPB) • Domains include processes and outcomes • Includes strength-based measures • Brief – each takes 15 seconds to 2 minutes • Reliable • Show convergent and divergent validity • Information from youth, clinician, and caregiver • Easy to score and interpret • Sensitive to change • Available in English and Spanish • Free for anyone to use – now in second edition • 26
  28. 28. CFS Measures Map onto the Relevant Questions • How is the youth doing overall? – Severity of symptoms and functioning (y, cg, cl) – Life satisfaction (y) – Hope (y) • How is treatment going with the youth? – Therapeutic alliance (y, cl) – Motivation for treatment (y) – Counseling impact (y) – Service satisfaction (y) • What are important caregiver issues? – Caregiver strain – Life satisfaction – Therapeutic alliance – Motivation for treatment – Service satisfaction 27
  29. 29. Organizational Measures • Services, staffing, any data already collected; experience with other initiatives, etc. • Culture, climate, adaptability, leadership, organizational learning, & initial perceptions of CFS • Current attitudes toward CFS, barriers/supports, value of reports, self- efficacy, & goal commitment 28
  30. 30. CFS Consultation and Training Model • CFS is a tool for transforming your agency into a learning organization • Ongoing support to – Promote sustainability through local ownership – Contextualize CFS operations and feedback to your agency’s needs and resources – Apply feedback to inform clinical sessions, supervision, program planning, professional development, and more – Empower practice improvement leadership at all levels of your agency 29
  31. 31. Analyses of Current Project • Randomized experiment in 33 sites - Analyses ongoing • Randomly assigned sites to feedback and no feedback conditions • Started with most direct question: Does feedback appear to affect clinician behavior? • The answer to this question does not depend upon the effectiveness of the treatment provided 30
  32. 32. CFS Feedback Influences Clinician Behavior in Sessions • Cases selected where clinician had not previously reported a problem with youth symptoms and functioning in a specific domain (e.g., behavioral issues, emotional problems, friend/peer issues) • When a client or caregiver alert was present for that domain – Viewing clinical feedback reports associated with addressing that domain in a future session – More feedback viewed more times addressed – Not explained by more reporting of problems by clients or caregivers 31
  33. 33. Feedback Reduces Symptoms and Improves Functioning • Used HLM random effects with new clients, controlled for nesting within sites, and number of sessions. • Youth respondent (ages 11-18): 340 clients • Caregiver respondent: 308 clients • Clinician respondent: 294 clients • There were no significant differences at baseline between the two groups for any respondent • Results showed Feedback Group improved more than the non-Feedback Group according to all three respondents. 32
  34. 34. Modeled Values of Youths’ SFSS p = 0.025 80 75 No Feedback 70 Youths’ SFSS in CBCL Units 65 60 55 50 45 40 35 Feedback 30 0 1 2 3 4 5 6 7 8 9 10 11 12 13 Months in CFS 33
  35. 35. Modeled Values of Caregivers’ SFSS p = 0.05 80 No Feedback 75 Caregivers’ SFSS in CBCL Units 70 65 60 55 50 45 Feedback 40 35 30 0 1 2 3 4 5 6 7 8 9 10 11 12 13 Months in CFS 34
  36. 36. Modeled Values of Clinicians’ SFSS p = .0005 80 No Feedback 75 Clinicians’ SFSS in CBCL Units 70 65 60 55 50 45 Feedback 40 35 30 0 1 2 3 4 5 6 7 8 9 10 11 12 13 Months in CFS 35
  37. 37. What Can CFS Do For Clients and Caregivers? • Provide a way to raise issues they may not feel comfortable addressing aloud or in the presence of caregiver or youth • Reassure them that the clinician is paying attention to their needs and is contextualizing services to their concerns • Demonstrate that effective services are a priority • CFS is a tool that takes the consumer’s voice beyond the session and into all aspects of care 36
  38. 38. What Can CFS Do For Clinicians? • Better identify thoughts, events, and feelings of clients and caregivers • Identify successes and problem areas to focus sessions • Provide consistent and systematic feedback of clinically relevant information • Inform treatment planning and goal setting • Focus sessions on clients’ issues to show that you are attending to their concerns • Check on how well treatment is working • CFS is a tool that enhances the clinician’s ability to tailor treatment as it progresses 37
  39. 39. What Can CFS Do For Clinical Supervisors? • Provide clinical data on each case’s progress and areas that seem to be improving or declining • Provide a tangible framework from which to guide clinical supervision that is independent of the clinician • Provide the needed information for a continuous quality improvement effort that facilitates accountability • Most of all…CFS is a tool that provides the supervisor with needed resources to promote evidence-based practice 38
  40. 40. What Can CFS Do For an Agency? • Serve as evidence to funding agencies that quality of services and effectiveness are a priority • Manage clinical services on their impact as well as their cost • Provide data on the effectiveness of services, how clients are improving, the typical problems being encountered, and where needs are not being met • CFS is a tool that supports overall practice improvement 39
  41. 41. What CFS Can Do for the Funder? • Determine who is being served and their progress • Provide a Quality Assurance mechanism • Compare services and agencies on cost effectiveness to optimize value • Reduce costs of some regulations and documentation activities such as excessively detailed treatment planning and progress notes 40
  42. 42. Plans • Continued partnership between Vanderbilt and Providence in training, development, and research • Continue software development • Use with adults as well as children and youths • Continue development of new measures • Proposal pending to test CFS with different types of feedback with youth with substance abuse problems • Integrate CFS with other mental health EBTs (new NIMH grant for Functional Family Therapy combined with CFS) • Develop web-based “treatment” intervention with CFS (Australia Project) • Use CFS in primary care settings to bridge medical and mental health fields 41
  43. 43. Web Site: 42