Dr. Barry L. Duncan, Director_______________________________________________________________________                      ...
2Construct ValidityConstruct validity rests on the assumption of an underlying trait or state that the questionnaire purpo...
The CORS/ORS was significantly related to the YOQ in all cells of the matrix demonstrating moderate     concurrent validit...
4   •   The use of evidence based treatments (EBT) does not guarantee success. In recognition of the inability       of an...
About the Heart and Soul of Change Project     The Heart and Soul of Change Project (hereafter the Project) is a practice-...
Upcoming SlideShare
Loading in …5

Youth Outcome Managment and the Partners for Change Outcome Management System


Published on

This handout details psychometrics of the ORS for adolescents and the CORS for children 6-12. It also highlights new relevant research regarding children and PCOMS.

1 Comment
1 Like
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Youth Outcome Managment and the Partners for Change Outcome Management System

  1. 1. Dr. Barry L. Duncan, Director_______________________________________________________________________ CDOI Training and Implementation of the Partners for Change Outcome Management System Youth Outcome Management and the Partners For Change Outcome Management System Feasibility is a critical issue in outcome management. Though it may be distressing to researchers, the ease with which an instrument can be explained, completed, interpreted, and then integrated into ongoing care is much more likely to influence utilization than either validity or reliability. In the real world of delivering services, finding the right outcome measure means striking a balance between the competing demands of validity, reliability, and feasibility. The development of the Outcome Rating Scale (ORS), and subsequently the Child Outcome Rating Scale (CORS), both just 4 items taking only minutes to score, interpret, and integrate, reflects an attempt to find such a balance. Intimately related to feasibility is the issue of the immediacy of feedback—whether the measure has an intended clinical use to improve the effectiveness and efficiency of rendered services. Most if not all other youth outcome measures were developed primarily as pre-post and/or periodic outcome measures. Such instruments provide an excellent way to measure program effectiveness but are not feasible to administer frequently, and therefore, do not provide real-time feedback for immediate treatment modification before clients drop out or suffer a negative outcome—in short they are not clinical tools as much as they are management or oversight tools. The ORS and CORS were designed as clinical and outcome tools to provide immediate feedback to both clients and providers to improve the effectiveness of services, and as a way to measure outcome at individual, program, and agency levels. There is robust empirical support of the CORS (for children 6-12) and the ORS (for adults and adolescents 13-17) as reliable, valid, and feasible measures of youth benefit from mental health and substance abuse services. After four years of systematic data collection at three clinical sites serving youth, the preliminary validation study was published in a peer reviewed journal (available at www.heartandsoulofchange.com) : Duncan, B., Sparks, J., Miller, S., Bohanske, R., & Claud, D. (2006). Giving youth a voice: A preliminary study of the reliability and validity of a brief outcome measure for children. Journal of Brief Therapy, 5(1), 5-22. Reliability Reliability, based on 1495 adolescents and 1961 children (over 20,000 administrations of the CORS and ORS) was estimated using Cronbach’s coefficient alpha, a measure of the internal consistency of the measure. The ORS and the CORS displayed strong evidence of reliability, with coefficient alpha estimates of .95 and .87 respectively. These are very high coefficients of reliability for such brief measures, suggesting that all four of the items tap the factor that most if not all outcome measures tap, global distress.PO Box 6157, Jensen Beach., FL 34957; 772.204.2511; 561.239.3640; barrylduncan@comcast.net
  2. 2. 2Construct ValidityConstruct validity rests on the assumption of an underlying trait or state that the questionnaire purportsto measure. In the case of outcome questionnaires, the underlying state, based on factor analysis ofmany outcome measures, is global distress. More specific constructs such as “depression,” “anxiety,”and “interpersonal problems” can be shown to share a large percentage of variance with the globaldistress factor. Constructs based on diagnostic nomenclature such as “anxiety disorder symptoms” or“symptoms of depression” appear to have very little predictive value, as both anxiety and depressionsymptoms load heavily on the global distress factor. The high coefficient alphas provide strongevidence that the ORS/CORS is measuring a single factor. Studies of concurrent validity invariablyconfirm that measures with similar item content assessing symptoms of depression, anxiety, socialdistress, and impairment in daily functioning are highly correlated. Given the substantial body ofresearch supporting the existence of the global distress factor, it is reasonable to suspect that theCORS/ORS is likewise a valid measure of global distress and will correlate with other establishedmeasures like the YOQ-30. And it does, as will be shown below. Construct validity is alsodemonstrated if the measures prove to be sensitive to change over time for youth receiving mentalhealth or substance abuse services. Both samples showed significant pre-post change. First Last Pre-post Effect Size Significance Assessment Assessment change (Change/SD (One tailed Mean SD Mean SD Mean SD at intake) t-test) ADOL-ORS 25.9 8.1 33.6 6.5 7.9 8.3 0.98 p<.001 (n=1495)CARETAKER- CORS 21.1 7.8 24.4 7.7 3.25 7.9 0.42 p<.001 (n=1961)Construct and Concurrent ValidityCorrelations between the CORS/ORS and Caretaker CORS/ORS scores and scores from the well-validated YOQ-30 provide further evidence of construct validity as well as concurrent validity. Thefollowing concurrent validity correlation matrix provides results for children and adolescents from anormative sample that received 3 concurrent administrations of the ORS/CORS and the YOQ for bothyouth and caretakers (CT). Note that correlations between the ORS and YOQ will be negative sincethe low score reflect low distress on the YOQ while on the ORS high scores are low distress.CORS/ORS and YOQ correlation matrixN=354 (all administrations combined) CORS CT-CORS CT-YOQ CORS 1 0.63* -0.43* CT-CORS 0.63* 1 -0.61* CT-YOQ -0.43* -0.61* 1* represents significance
  3. 3. The CORS/ORS was significantly related to the YOQ in all cells of the matrix demonstrating moderate concurrent validity with the well researched but much longer YOQ. Interestingly, the correlation was also significant between youth and caretaker ratings, suggesting that giving voice to youth via outcome measures is supported by the evidence. Although adolescents have long had this opportunity, the CORS is the first outcome measure that taps into the perspective of children ages 6-12. Other indications of construct validity are reported in the validation article: the ability of the measures to differentiate between normative and clinical samples, between different levels of severity, and the demonstration of stability in non-clinical populations vs. change sensitivity, beyond regression to the mean, in clinical populations. Conclusions The ORS/CORS provides a brief measure of global distress suitable for assessing treatment outcomes. The reliability compares favorably to well-established outcome measures containing many more items. The high coefficient alpha and pre-post differences provide evidence of construct validity; the significant correlations with the YOQ provide evidence for concurrent validity—that both measures appear to measure the broad construct of global distress. Reliable, valid, and feasible measures were the first step in developing the Partners for Change Outcome Management System (PCOMS). What Is PCOMS? PCOMS incorporates the most robust predictors of therapeutic success into an outcome management system that partners with clients while honoring the daily pressures of front-line clinicians. Unlike other methods of measuring outcome, this system truly gives clients the voice they deserve and assigns consumers key roles in determining how services are delivered and perhaps funded. PCOMS uses two brief scales, the Outcome Rating Scale (ORS and CORS) and the Session Rating Scale (SRS and CSRS) to measure the clients perspective of benefit and the alliance, respectively. It is the only system that includes a transparent discussion of the results with clients and the only system to include routine measurement of the therapeutic alliance. PCOMS has been shown in 3 randomized clinical trials (RCTs), all conducted by Heart and Soul of Change Project researchers and published in top tier journals, to significantly improve effectiveness in real clinical settings. Because of the 3 RCTs, PCOMS is recognized in SAMHSA’s National Registry of Evidence-based Programs and Practices (NREPP). PCOMS has been implemented by hundreds of organizations, public and private, by thousands of behavioral healthcare professionals in all 50 states and 20 countries serving over 100,000 clients a year. PCOMS is a-theoretical and may be added to or integrated with any model of practice. Why Is PCOMS a Good Idea? • There is a growing worldwide movement, both private and governmental, to involve consumers in mental health and substance abuse care and improve the outcome or value of rendered services. Both the Presidents New Freedom Commission and SAMHSA call for consumer centered care with services tailored to the individual’s unique recovery journey. PCOMS proactively partners with consumers to improve the value of the care they receive and is the only system that by design includes clients in all aspects of outcome management.PO Box 6157, Jensen Beach., FL 34957; 772.204.2511; 561.239.3640; barrylduncan@comcast.net
  4. 4. 4 • The use of evidence based treatments (EBT) does not guarantee success. In recognition of the inability of any model to predict success for the individual client, the APA Task Force on evidence based practice (EBP) suggested that “ongoing monitoring of patient progress and adjustment of treatment as needed are essential.” PCOMS provides a method to combine EBP with “practice-based evidence” to ensure success at the individual client level. PCOMS is included in SAMHSA’s NREPP. • Although psychosocial intervention is successful for many clients, a portion of clients do not derive benefit. In addition, public behavioral health drop-out rates average 47%. Making matters worse is the fact that clinicians often fail to identify people at risk for dropping out or unsuccessful outcome. PCOMS rectifies these problems by providing an early warning system to identify failing clients based on the best known predictors of outcome and retention. • Two factors are strongly predictive of retention, progress, and the eventual success of treatment: The consumer’s rating of the alliance with the provider of services and the consumer’s rating of early progress in response to the provider, level, and type of treatment offered. PCOMS monitors these two predictive variables with reliable, valid, and feasible outcome and alliance measures. • Providing clinicians with ongoing consumer feedback regarding the alliance and progress in treatment dramatically increases success rates as the cost-effectiveness (reduces cancellations, no shows, length of stay, etc) of provided services. • Three randomized clinical trials (RCT) used PCOMS (the ORS and SRS) to investigate the effects of feedback versus treatment as usual (TAU). First, Reese, Norsworthy, & Rowlands (2009; Psychotherapy) found that individuals who attended therapy at a university counseling center and a graduate training clinic demonstrated significant treatment gains for feedback when compared to TAU. Second, our study in Norway (Anker, Duncan, & Sparks, 2009; Journal of Consulting and Clinical Psychology), the largest (N = 410) RCT of couple therapy ever done, found that feedback clients reached clinically significant change nearly four times more than non-feedback couples. The feedback condition maintained its advantage at 6 month follow-up and achieved a 46% lower separation/divorce rate. Feedback improved the outcomes of nine of ten therapists in this study. Finally Reese, Toland, Slone, & Norsworthy, 2010; Psychotherapy) replicated the Norway study with couples. These studies, as well as all the references cited in this summary are available at www.heartandsoulofchange.com. Two other RCTs are underway, one in a group treatment setting for substance and PTSD problems with returning Iraq and Afghanistan veterans, and the other in school based setting with children and adolescents. • Study of PCOMS in the schools in the UK found that the demonstrated benefits of feedback with adults extend to 7-11 year olds too (Cooper, Stewart, Bunting, & Sparks, 2012: Psychotherapy Research). This study found outcomes were significantly better with children who had PCOMS as part of their treatment than those who did not, and that this benefit extended to caregiver and teacher ratings as well. Stay tuned for the results of the large RCT being conducted now in the schools (Gillaspy, Murphy, & Duncan, Bohanske, & Zartoga, in preparation). • Download the ORS/CORS and SRS/CSRS at www.heartandsoulofchange.com. They are free for personal use with only a nominal fee a lifetime license for agencies.Successful Implementation of PCOMSSuccessful implementation of PCOMS requires organizational commitment at all levels (see ReadinessChecklist at www.heartandsoulofchange or www.slideshare.com) . Implementation also requires anattention to front-line clinicians. For some who have been in the field for a while, outcomemanagement is a totally foreign concept while others have been turned off by cumbersome measuresthat seem far removed from their day-to-day work with clients. Still others are fearful that "pay forperformance" or similarly motivated strategies will punish those who do not measure up to somearbitrary standard. Implementation is enhanced when it makes sense to therapists and appeals to theirnearly universal desire to do good work. This is the approach to implementation of the Heart and Soulof Change Project.
  5. 5. About the Heart and Soul of Change Project The Heart and Soul of Change Project (hereafter the Project) is a practice-driven, training and research initiative that focuses on what works in therapy, and more importantly, how to deliver it on the front lines via client based outcome feedback, or what is called PCOMS. The Project features an international community of providers of all stripes and flavors as well as researchers and professors, all dedicated to privileging consumers and improving psychotherapy and substance abuse outcomes. Researchers at the Project conducted all three RCTs described above as well as the noted child and adolescent study and upcoming veteran and inner city school investigations. The RCTs led to the designation of PCOMS as an evidence based practice by SAMSHA. In addition to the RCTs addressing PCOMS and the benefits of consumer feedback, researchers and scholars at the Project have published 15 other recent studies and papers regarding improving psychotherapy outcomes and the training of mental health and substance abuse professionals. The Project is distinguished by its commitment to ongoing research and dissemination to front line practitioners. About the Director, Barry Duncan, Psy.D. Barry L. Duncan, Psy.D., is a therapist, trainer, and researcher with over 17,000 hours of clinical experience. Dr. Duncan has over one hundred publications, including fifteen books addressing systematic client feedback, consumer rights and involvement, the power of relationship, and a risk/benefit analysis of psychotropic medications. His work regarding consumer rights and client feedback has been implemented across the US and in 20 countries including national implementation in couple and family centers in Norway. His latest books: the 2nd edition of the Heart and Soul of Change (APA, 2010); and On Becoming a Better Therapist (APA, 2010). Because of his self help books, he has appeared on "Oprah," "The View," and several other national TV programs. Barry co-developed the ORS/SRS family of measures and PCOMS to give clients the voice they deserve as well as provide clients, clinicians, administrators, and payers with feedback about the clients response to services, thus enabling more effective care tailored to client preferences. He is the developer of the clinical process of using the measures and PCOMS, first articulated in the first edition of Heroic Clients, Client Agencies (Duncan & Sparks, 2002). Barry implements PCOMS in small and large systems of care and conducts agency trainings, workshops, and keynote presentations on all of the topics listed above for both professional and general audiences. Drawing upon his extensive clinical experience and passion for the work, Barrys trainings speak directly to the front line clinician. His presentations not only cover consumer based outcome feedback or PCOMS—which improves outcomes more than anything since the beginning of psychotherapy—Barry also talks about what it means to be a therapist and how each of us can re-remember and achieve our original aspirations to make a difference in the lives of those we serve. His trainings integrate the nuances of the work, our dependence on the resources of clients, and an appreciation of the hard work required for strong alliances across clients with the systematic use of outcome and alliance feedback. Video examples from a wide variety of clients demonstrate both the ideas and practices of CDOI and PCOMS—and moreover show that Barry doesnt just talk the talk.PO Box 6157, Jensen Beach., FL 34957; 772.204.2511; 561.239.3640; barrylduncan@comcast.net