The document discusses the history and current state of evidence-based practices in children's mental health. It notes that while research has identified hundreds of evidence-based therapies and interventions, many children still have unmet mental health needs. It summarizes the levels of evidence for different psychosocial and pharmacological treatments, as well as home- and community-based services. However, it states that significant challenges remain in implementing evidence-based practices into real-world mental health systems and services.

1. Building a Science on Implementation of Evidence-based Practices in Children’s Mental Health Kimberly Eaton Hoagwood, Ph.D. Columbia University December 11, 2005

2. Turning Points in Child & Adolescent Services Research: A Very Brief History Unclaimed Children (Knitzer, 1982) documents lack of community-based care (soon to be updated) Systems of Care Monograph published (Stroul & Friedman, 1986); CASSP established, 1986 Tripling of funding for research on children’s mental health at NIMH (1989-2001) Meta-analyses of psychotherapies document effect sizes (Weisz et al, 1995, 1998, 2003; Kazdin et al., 1998, 2003) System of Care study results published (Bickman, 1996) Healthcare reform in US and Britain spurs growth of evidence-based practice movement (1996-present) Surgeon General’s Reports (1999; 2000; 2001) highlight disparities between research and practice Identification of >550 psychosocial therapies (Kazdin, 2003); medication trials for ADHD, OCD, aggression, depression Methods developed for assessing organizational context applied to youth mental health services and found to predict child outcomes (Glisson, 2002, 2005; Schoenwald et al, 2003) EBPs spread into state and federal policy planning (2004-present)

3. Yet…sobering facts about mental health services for children Unmet need as high now as 20 years ago Unmet need highest among minority youth Receipt of mh services increased but only 1/5 of children with the most severe needs receive mh services (Kessler et al 2005; Foster, Rollefson, Doksum et al., 2005) Onset by age 14 for 90% of adults with SMI (Kessler et al., 2005) Lack of availability and access a major barrier for most families There is no system: use and need don’t match up

4. Ringel &Sturm, 2000; NIMH, 2001 Unmet Need for Mental Health Services 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% White African- American Latino Other

5. National Averages of Use and Need Don’t Match National Average MH Need for Children at 6-17: 7.09% National Average MH Use for Children at 6-17: 7.45% Data Source: NSAF wave 1 and 2, Sturm, 2001

7. Beyond the Linear Model Basic Research Clinical Trial (Efficacy) Treatment Development Effectiveness Trial Treatment Deployment

8. Schoenwald & Hoagwood, 2001

9. The Rise in Popularity of the term “Evidence-Based” (Hoagwood & Johnson, 2003) EBT EBP EBM 1900-1990 0 0 0 1990-1995 3 7 76 1995-2002 63 459 5,425

10. Psychotherapies provided in routine clinical care have little to no effect (Weisz et al., 1995) Mean Effect Sizes Weisz et al., 1995 Children & Adolescents Adults University Clinic settings

11. What is Evidence? Lonigan, Ebert & Johnson, 1998; Chambless et al., 1998 At least two controlled group design studies or a large series of single-case design studies Minimum of two investigators (for well-established) Use of a treatment manual Uniform therapist training and adherence Tested with clinical samples of youth Tests of clinical and functional outcomes Long-term outcomes beyond termination of treatment .

12. Grading the Quality of Evidence Biglan, Mrazek, Carnine, Flay (Am Psych 2003) Grades 1-7 1 = multiple RCTs or multiple time series experiments by 2 or more indep teams + implementation effectiveness 2 = 1 without implementation effectiveness 3 = no indep teams 4 = 1 RCT or time series 5 = comparisons w/o randomization 6 = pre-post 7 = endorsement by authorities

13. Kazdin (2004) criteria Not evaluated Evaluated but unclear, no or possibly negative effects at this time Promising (some evidence) Well-established (parallel to well-established in conventional schemes) Better/Best Treatments (treatments shown to be more effective than other evidence-based treatments)

14. 14 Major Reviews of Evidence-based Interventions for Children (1998-2004) Chambless & Hollon (1998) Defining empirically-supported therapies. Journal of Consulting & Clinical Psychology Surgeon General’s Mental Health Report (1999) Weisz & Jensen (1999) Mental Health Services Research Journal of the Am. Academy of Child/Adol. Psychiatry , 1999 Olds et al., (1999) Review of Preventive Interventions, Center for Mental Health Services Burns, Hoagwood, & Mrazek (2000) Effective treatments for mental disorders in children and adolescents, Child Clinical and Family Psych Rvw Rones & Hoagwood (2000) School based mental health services review. Clinical Child and Family Psychology Review

15. Webster-Stratton & Taylor (2001) Preventing violence in adolescence with interventions for young children Greenberg, et al., (2001) Prevention of mental disorders in school-aged children. Prevention & Treatment Surgeon General’s Youth Violence Report (2001) Burns & Hoagwood (2002) Community treatments for youth : Oxford University Press Kazdin & Weisz (2003) Evidence-based psychotherapy for children and adolescents Weisz, JR. (2004) Psychotherapy for children and adolescent: Evidence-based treatments and case examples. Cambridge University Press. Burns, BJ & Hoagwood, K (in press) Update on evidence-based practices. Two Volumes. Psychiatric Clinics of North America 14 Reviews of EBPs

16. More than 1500 published clinical trials on outcomes of psychotherapies for youth More than 550 different named psychotherapies (Kazdin, 2000; Kazdin & Weisz, 2003; Weisz, 2004) 6 meta-analyses of their effects: effects as robust as for adults More than 300 published clinical trials on safety/efficacy of psychotropic medications and growing Approx 50 field trials of community-based services 47 effective school-based interventions cited by Rones & Hoagwood (2000) 34 effective preventive interventions cited by Greenberg et al, 2001 Strength of the evidence

17. EVIDENCE-BASED PSYCHOSOCIAL TREATMENTS Well-Established Probably Efficacious DEPRESSION None Self-Control (children) Coping with Depression (adolescents) IPT (adolescents) ADHD Behavioral Parent Training Behavioral Management Training Behavioral Interventions in the Classroom Behavioral Modification in Classroom ANXIETY None Cognitive-Behavioral Therapy Phobia Participant Modeling Imaginal and In Vivo Desensitization Reinforced Practice Live and Filmed Modeling DISRUPTIVE BEHAVIOR Living with Children Delinquency Prevention Program Videotape Modeling Parent-Child Interaction Therapy Parent Training Program Time-Out Plus Signal Seat Treatment Anger Coping Therapy Problem Solving Skills Training Anger Control Training w/ Stress Innoc Assertiveness Training Multisystemic Therapy Rational-Emotive Therapy Preschool Adolescent School Age Source: Journal of Clinical Child Psychology, Volume 27, Number 2, 1998 + additional studies for depression and PTSD

18. Evidence-Based Psychosocial Treatments Well-Established Probably Efficacious PTSD None Cognitive-Behavioral Therapy for sexual & physical abuse Source: Journal of Clinical Child Psychology, Volume 27, Number 2, 1998 + additional studies for depression and PTSD

19. Ineffective Psychosocial Treatments Non-behavioral interventions for disruptive behavior disorders and/or ADHD (Weisz et al., 1995; Pelham et al., 1998) Group, peer-based interventions for disruptive disorders

20. DARE (5th and 6th grade curriculum) Gun Buyback programs Boot Camps Peer counseling programs Summer job programs (at risk youth) Home detention with electronic monitoring Wilderness / challenge programs Casework / counseling What produces negative (iatrogenic) outcomes Waivers to adult (criminal courts) Scared Straight Shock Probation / Parole Ineffective programs to prevent youth violence (Elliot, 2000)

21. Psychopharmacology Evidence for Childhood Disorders STRONG ADHD Stimulants TCAs MODERATE WEAK DEPRESSION SSRIs AUTISM Antipsychotics OCD SSRIs, TCAs ODD/CD Antipsychotics, Mood stabilizers, Stimulants ANXIETY SSRIs AGGRESSION Atypical antipsychotics BIPOLAR Lithium TOURETTE’S Antipsychotics

22. Evidence-based Home & Community-based Services Multisystemic Therapy (Henggeler et al., Schoenwald et al) Intensive Case Management (including Wraparound) (Evans; Burchard; Burns) 8 RCTs and 1 quasi-exper. fewer arrests fewer placements decreased aggression cost-savings 4 RCTs and 3 quasi-exper. less restrictive placements some increased functioning

23. Evidence-based Home & Community-based Services Treatment Foster Care (Chamberlain) Nurse home visitation Program (Olds et al) Functional family therapy (Alexander & Sexton) 4 RCTs more rapid improvement decreased aggression better post-discharge outcomes 6 RCTs Long term improvements in reducing child abuse 3 RCTs Reduced recidivism 6-42 months out

24. Evidence-based Home & Community-based Services (cont’d) Parent Empowerment (fam ed) (Heflinger & Bickman) Mentoring (Vance) Respite Services (Bruns & Burchard) Crisis Services 1 RCT increased knowledge and self-efficacy 1 RCT less substance use and aggression better school, peer, and family func 2 wait-list control studies fewer placements reduced family stress 0 controlled, 1 pre-post placement prevented in 60-90% of cases

25. Evidence for Institutionally-Based Care Hospital Residential Treatment Center Group Home Partial Hospitalization 3 RCTs findings in favor of community comparison conditions 2 quasi-experimental Project Re-Ed: gains versus untreated Gains in residential treatment center were equal to treatment foster care (TFC @ one-half cost) 2 quasi-experimental mixed findings -- gains and deterioration (arrest rates) 1 RCT partial hospital versus wait-list controls benefits at 6 months for behavior symptoms, and family

27. Challenges to Putting EBPs into Practice No science to guide implementation Policies are way ahead of the knowledge base Pressures to hold providers accountable for outcomes Poorly trained workforce Incentives misaligned

28. 5 Core Components for State System Change Train practicing clinicians to deliver clinical EBPs with proven effectiveness Engage families in services by removing barriers to access: Target clinician outreach Empower families with tools, guidelines, and support: Target families and advocates Support core organizational processes Create system-wide incentives to support change

29. New York State Implementation Model System & Policy Context Financial policies, methods of reimbursement, state policies Organizational Context Culture Climate Structure Clinical Care Improvement Training on EBP’s, supervision, consultation and support Engagement Empowerment Attitudes, Beliefs & Expectancies of Families & Youth Improved Child & Family Outcomes Attitudes, Beliefs & Expectancies of Clinicians and Supervisors Improved Implementation Efficiency & Effectiveness

30. Implementation of Trauma-focused CBT for Children: The CATS Study Developed in response to the World Trade Center Disaster

32. Children and Youth participating in the CATS research project ID’s Generated 1068 Assigned 650 Declined 50 Ineligible 165 CATS 450 Conferenced-In 51 Comparison 149 Children 287 Adolescents 163 Children 108 Adolescents 41 Unassigned 204

33. Acknowledgements CATS Consortium The CATS Consortium is a cooperative multi site treatment study performed by nine independent teams in collaboration with the New York State Office of Mental Health. The New York State collaborators are Kimberly Eaton Hoagwood, Ph.D., Chip Felton, M.S.W., Sheila Donahue, Ph.D., Anita Appel, M.S.W., James Rodriguez, Ph.D., (NYSPI), Laura Murray, Ph.D., (NYSPI), David Fernandez, M.A., (NYSPI), Joanna Legerski, B.S. (NYSPI), Michelle Chung, B.A., Jacob Gisis, B.S., Jennifer Sawaya, B.A., Sudha Mehta, M.P.H. (OMH), Jessica Mass Levitt, Ph.D. (NYSPI). The Principal investigators and co-investigators from the nine sites are Robert Abramovitz, M.D., (JBFCS),), Reese Abright, M.D., (St. Vincents Hospital), Peter D’Amico, (North Shore/Long Island Jewish), Giussepe Constantino, Ph.D., (Lutheran Hospital), Carrie Epstein, C.S.W.-R., (Safe Horizon), Jennifer Havens, M.D., (Columbia University), Sandra Kaplan, M.D., (North Shore/LIJ), Jeffrey Newcorn, M.D., (Mt. Sinai), Moises Perez, Ph.D., (Alianza Dominicana), Raul Silva, M.D., (NYU/Bellevue), Heike Thiel de Bocanegra, Ph.D., (Safe Horizon),), Juliet Vogel, Ph.D. (North Shore/LIJ). The Scientific Advisors to the project are: Leonard Bickman, Ph.D., (Vanderbilt University), Peter S. Jensen, M.D., (Columbia University), Mary McKay, Ph.D., (Mount Sinai Medical School), Susan Essock, Ph.D., (Mount Sinai Medical School), Sue Marcus, Ph.D., (Mount Sinai Medical School), Wendy Silverman, Ph.D. (Florida International University), Robert Pynoos, M.D. (University of California, Los Angeles); Allan Steinberg, Ph.D. (University of California, Los Angeles); Lawrence Palinkas, Ph.D. (University of California at San Diego); and Joseph Cappelleri, Ph.D., (Pfizer Corporation). The Treatment Developers and Scientific Consultants to the project are: Judy Cohen, M.D., (University of Pittsburgh), Anthony Mannarino, Ph.D., (University of Pittsburgh), Christopher Layne, Ph.D., (Brigham Young University), William Saltzman, Ph.D ., (UCLA).

34. CATS Design 81 routine practice therapists working in schools and clinics trained in 2 EBP trauma models Ongoing consultation/support provided by treatment developers Children and families offered EBP trauma or treatment as usual Baseline, 3, 6, 12 month follow-up Assessments of PTSD, other anxiety, depression, behavior problems, strengths, school functioning Regression discontinuity design + propensity analyses to assess predictors of outcome improvement

35. Child Trauma-focused Treatment Cohen, Judith; Mannarino, A, Deblinger, E. et al. (2002) Child and Parent Trauma-Focused Cognitive Behavioral Therapy Treatment Manual. Trauma Focused Interventions: Psychoeducation Stress inoculation/Relaxation Cognitive triangle Trauma Narrative/Gradual exposure Cognitive processing Parallel parent treatment sessions

36. Adolescent Treatment Layne, Christopher M.; Saltzman, William R.; Pynoos, Robert S.; (2002) Trauma/Grief-Focused Group Intervention for Adolescents. Module I (6 sessions): Psychoeducation Coping Skills Cognitive work Communication skills Module II (8-12 sessions): Trauma narrative/Exposure Cognitive restructuring Module III (3 sessions): Resuming developmental progression Problem-solving

37. Intensive consultation 81 clinicians trained on these models 3 day training + 2 booster sessions + bi-weekly consultation calls for 1 year Bi-weekly site visits Weekly steering committee calls with PIs Weekly site coordination meetings

38. Engaging Families in Treatment “ Triple threat condition”: poverty, single parent status, and stress Rates of service use are lowest in low-income, urban communities. No show rates can be as high as 50% (Armbruster & Kazdin) Trained all teams on McKay’s engagement protocol (McKay & Bannon, 2005)

39. Treatment as Usual Show Rates McKay et al., 2005

40. Empirically supported engagement interventions Reminders reduced missed appointments by 32% (Kourany et al., 1990; McLean et al., 1989; Shivack et al., 1989) Intensive family-focused telephone engagement associated with 50% decrease in initial show rates and a 24% decrease in premature terminations (Szapocznik, 1988; 1997) Combined telephone and first interview engagement interventions associated with attendance rates of 74%, representing a 16 to 25% increase above the clinic comparison families (McKay et al., 1998).

41. First Interview Results

42. Key elements of engagement training Help clinicians and intake staff examine their perceptions of barriers Practice skills related to the initial face-to-face interview with a child and their family Support clinicians and intake staff abilities to form collaborative working relationships with adult caregivers and youth Help them identify an immediate and practical concern that can be addressed in the first interview Learn skills related to the development of a shared commitment, language and understanding with the family

43. CATS Assessment vs. Treatment Show Rates

44. Assessment Rate by Site

45. Treatment Show Rate by Site

46. Treatment Settings

47. Average Number of Treatment Sessions for Treatment Completers

48. Treatment Sessions N=446

49. Implementation challenges: Matching EBPs to individual cases Strategy: Bruce Chorpita and State of Hawaii level of evidence 5 levels of evidence Detailed information about sample demographics Practical and flexible menu of options

50. Autism Conduct Depression Oppositional Substance None None CBT Parent/Teacher Training CBT None Multisystemic Therapy CBT + parents; IPT; Relaxation Anger Coping; Assertiveness; PSST Behavior Tx; Family Tx ABA; FCT None None None None Play Therapy; GIST Juvenile Justice; Individual Tx Family Tx; Individual Tx Relaxation; Individual Tx Individual Therapy ADHD Behavior Therapy None None Biofeedback; Play Tx; GIST None None Group Therapy None Group Therapy Group Therapy Anxiety CBT; Exposure; Modeling CBT+ parents; Ed support None EMDR; Play Tx; GIST None Problem Level 1 best support Level 2 good support Level 3 some support Level 4 no support Level 5 known risks Example: Chorpita (2002) EBT Analysis

51. Example: Clinical application 14 year old Depressed Puerto Rican Male Late in semester

52. Level 2 CBT + parents Interpersonal Relaxation 88% 85% 100% MA; PhD MA; PhD; MD MA; PhD clinic clinic school CBT 94% MA; PhD Clinic; school 1.74 1.40 1.51 1.48 Level 1 Intervention Finish 14 to 18 12 to 18 11 to 18 9 to 18 Age Staff Setting Effect NS 49% PR; 41% HA; 10% C NS 84% NS; 18%PR; 3%AA Ethn 7 to 8 weeks 12 weeks 5 to 8 weeks 5 to 16 weeks Length Evidence: Interventions for Depression

53. Level 2 CBT + parents Interpersonal Relaxation 88% 85% 100% MA; PhD MA; PhD; MD MA; PhD clinic clinic school CBT 94% MA; PhD Clinic; school 1.74 1.40 1.51 1.48 Level 1 Intervention Finish 14 to 18 12 to 18 11 to 18 9 to 18 Age Staff Setting Effect NS 49% PR; 41% HA; 10% C NS 84% NS; 18%PR; 3%AA Ethn 7 to 8 weeks 12 weeks 5 to 8 weeks 5 to 16 weeks Length Evidence: Interventions for Depression

54. Level 2 CBT + parents Interpersonal Relaxation 88% 85% 100% MA; PhD MA; PhD; MD MA; PhD clinic clinic school CBT 94% MA; PhD Clinic; school 1.74 1.40 1.51 1.48 Level 1 Intervention Finish 14 to 18 12 to 18 11 to 18 9 to 18 Age Staff Setting Effect NS 49% PR; 41% HA; 10% C NS 84% NS; 18%PR; 3%AA Ethn 7 to 8 weeks 12 weeks 5 to 8 weeks 5 to 16 weeks Length Evidence: Interventions for Depression

55. Example 16 year old Female Anxiety problems Both parents available

56. Level 2 CBT + parents Edu support 93% 85% MA; PhD N/A clinic clinic CBT 95% UG; MA; PhD Clinic; school 1.05 1.78 N/A Level 1 Intervention Finish 14 to 18 6 to 17 2 to 17 Age Staff Setting Effect NS 92% C 54% NS; 33% C; 7% Arm; 6%AA Ethn 12 weeks 12 weeks 3 to 16 weeks Length Evidence: Interventions for Anxiety

57. Level 2 CBT + parents Edu support 93% 85% MA; PhD N/A clinic clinic CBT 95% UG; MA; PhD Clinic; school 1.05 1.78 N/A Level 1 Intervention Finish 7 to 12 6 to 17 2 to 17 Age Staff Setting Effect NS 92% C 54% NS; 33% C; 7% Arm; 6%AA Ethn 12 weeks 12 weeks 3 to 16 weeks Length Evidence: Interventions for Anxiety

58. Strategy: Distillation approach (Chorpita & Weisz, 2005; Dalaiden & Chorpita, 2005) Cross tabulate studies with intervention elements Use all studies; code each study Yields a matrix demonstrating protocol overlaps Implementation Challenges: Too many models to choose among

59. Example (Chorpita et al., 2005) Internalizing Externalizing All Ext Int

60. Anxiety and Phobias (Chorpita et al) Depression All Dep A/P Ext Int

61. Strategies for making EBP implementation practical Integrated Psychotherapy Consortium (Center for the Advancement of Children’s Mental Health– P. Jensen, E. Goldman) Michigan’s clinical outcome assessments (K. Hodges, J. Wotring) Casey Blue Sky Project (P. McCarthy, S. Henggeler, S. Schoenwald, T. Sexton, P. Chamberlain) Hawaii’s clinical decision-making system (B. Chorpita, E. Dalaiden)

62. New York State Implementation Model System & Policy Context Financial policies, methods of reimbursement, state policies Organizational Context Culture Climate Structure Clinical Care Improvement Training on EBP’s, supervision, consultation and support Engagement Empowerment Attitudes, Beliefs & Expectancies of Families & Youth Improved Child & Family Outcomes Attitudes, Beliefs & Expectancies of Clinicians and Supervisors Improved Implementation Efficiency & Effectiveness

63. Family-based services Hoagwood (2005) review of 4,000 articles since 1980 identified 41 rigorous studies of family-based services 3 categories: Families as recipients of services Families as co-therapist Processes of involvement, engagement, empowerment

64. Conclusions from family-based services review Broader view of outcomes is needed Absence of robust literature on process variables limits conclusions Evidence ambiguous as to whether these services improve child outcomes Linkage of these services to EBP implementation may be needed to amplify effects

65. Parent Empowerment in New York 4 year process: scientific review of the literature Identified one controlled trial of empowerment (Bickman et al, 1998) Adapted for parent advocates and for multi-ethnic families Added modules about EBPs for child mental health (ADHD, depression, conduct, treatment efficacy) Collaborative partnership: Mental Health Assn., Columbia University, NYS Office of Mental Health Added engagement strategies Developed 4 manuals for advocates and parents Conducting 2 NIMH-funded effectiveness trials to examine impact of program on knowledge, skills, self-efficacy and use of services (behavior)

67. PEP Manual Content Parent Advocate Manual Introduction Getting Ready Building Engagement, Listening, and Boundary Setting Skills Building Your Teaching and Group Management Skills Developing Priority Setting Skills Specific Disorders and Their Treatments The Mental Health System of Care: What to Expect and How to Prepare Services and Options Through the School System Teaching Tools for Parent Advocates Parent Handbook Introduction Knowing Yourself Knowing Your Child Treatment Management Skills: How to be Your Child’s Case Manager Specific Disorders and Their Treatments The Mental Health System of Care: What to Expect and How to Prepare Services and Options Through the School System Helpful Tools for Parents

68. Parent Empowerment Research Study Basic Design 40 Parent Advocates/Family Support Specialists (PA/FSS) 20 PA/FSS PEP Training 20 PA/FSS Training as Usual 120 Parent/Caregivers Receiving PA/FS Services 6 per PA/FSS 120 Parent/Caregivers Receiving PA/FS Services 6 per PA/FSS

69. Post Training Self Efficacy N=31 =p< .05 5 = Greater Efficacy 1 = Less Efficacy

70. Post Training Self-Efficacy N=31 + = p< .10 = p< .05 ** = p<.001 5 = Greater Efficacy 1 = Less Efficacy

71. Embedding effective clinical practices in settings and systems Family support services (e.g., engagement, empowerment) and effective clinical treatments are part of larger work environments Studies of environmental contexts have identified characteristics that improve or interfere with service delivery

72. Key Constructs in Measurement of Organizational Contexts Organizational climate reflects perceptions of the work environment and has been linked with child outcomes in studies of child welfare agencies (Glisson & Himmelgarn, 1998) Organizational culture refers to the ways things are done in a work environment—the norms and shared expectations Organizational structure refers to the hierarchy of power

73. Organizational context affects uptake of EBPs and outcomes Three decades of studies by Glisson and colleagues Glisson & Himmelgarn’s (1998) study of child welfare agencies found that the strongest predictor of child improvement was organizational climate Organizational culture , not climate, explained variations in service quality (Glisson & James, 2002) Organizational level interventions can improve climate and reduce staff turnover (Glisson, in press) Organizational factors affect youth outcomes (Schoenwald et al., 2003)

74. Glisson & Himmelgarn (1998) Parameter Estimates for Hypothesized Six-Variable Model Service Quality County Demographics Service Outcomes (problem levels) -.13* .12* -.05 -.24* -.03 .02 -.36* .01 .06 -.20* * p < .05 Organizational Climate Interorganizational Services Coordination Interorganizational Relationships

75. Organizational Context: Implications for the Transport of Evidence-Based Treatments To Mental Health Provider Organizations Sonja K. Schoenwald, Ph.D. Family Services Research Center Psychiatry & Behavioral Sciences Medical University of South Carolina

76. Organizational structure & climate (Schoenwald et al., 2003) Multi-site study of 40+ community clinics delivering MST Examine impact of organizational context on therapist adherence and outcomes Organizational structure and climate factors were not associated with adherence scores Organizational factors were associated directly with youth outcomes. And some associations were in unexpected directions

77. Aims of Transportability Study To examine : the association of MST adherence to outcomes in field sites organization’s impact on adherence extra-organizational factors’ impact on organizational factors affecting adherence the impact of clinician training & experience on adherence a mediation model of effectiveness

78. Social Ecological Model of TreatmentTransportability Extra-Organizational Context (Referral, Reimbursement, Disposition) Organization Clinician Child Adherence Outcomes Clinician Variables Professional Training & Experience

79. Transportability of MST - Evidence of Multi-Level Treatments* First 666 referred youth (juvenile justice, child welfare, and mental health) 14.7 years old, 67% male, 61% Caucasian 57% one bio parent, 15% both bio parents 48% less than 20k/yr *Schoenwald, Sheidow, Letourneau, & Liao (2003). Mental Health Services Research

80. Treatment Outcomes Significant pre-post reductions in child behavior problems and functioning Discharge was based on achievement of treatment goals in 73% of cases Discharge decisions were made by the therapist and family (versus external entity) in 64% of cases

81. Adherence-Outcomes Linkages Higher adherence predicted post-treatment decreases in child behavior problems and child functioning problems . Higher adherence predicted positive discharge circumstances .

82. Therapist Effects Pre-Post Differences In CBCL Total Scores by Adherence

83. Organizational Structure & Climate Findings Organizational structure and climate factors were not associated with adherence scores

84. Organizational Factors Predicted Youth Outcomes Organizational factors were associated with youth outcomes. And, some associations were in unexpected directions.

85. Adherence & Organization: Direct Effects on Outcomes

86. Moderation of Organizational Effects by Adherence Level (1) Sample of adherence scores was split into upper and lower adherence quartiles Advancement & Reward x low adherence = increased child problems Advancement & Reward x high adherence = unrelated to child problems Greater Procedural Specification x high adherence = increased child problems Greater Procedural Specification x low adherence = unsuccessful discharge

87. Adherence Moderates Organization Effects on Outcomes Opportunities for Advancement & Reward appear to matter little when adherence is high, but translates into poorer outcomes when adherence is low Hierarchical Authority and Procedural Specification may interfere with positive outcomes when therapists are adhering to MST, but matters little when adherence is low.

88. Implications Need to better understand criteria used in organizations for advancement and reward, and to consider including adherence and outcome indicators in those criteria Need to better understand how organizational hierarchy and procedures may interfere with adherence to a specific EBP

89. Key Factors Associated with Adoption and Diffusion (Greenhalgh et al, 2004): A Meta-Narrative Synthesis of Evidence Characteristics of the innovation Characteristics of the individual adopter Sources of communication and influence Structural and cultural characteristics of potential organizational adopters Characteristics of the external environment Innovation uptake practices Linkage among components of the model

90. Organizational change is personal Interpersonal influence through social networks is the dominant mechanism for diffusion (Valente, 1996) Champion roles: Organizational maverick Transformational leader Organizational buffer Network facilitator Boundary spanners (social networkers) Organizations that promote boundary-spanning roles are more likely to assimilate innovations (Barnsley, Lermieux-Charles & McKinnet, 1998; Ferlie et al., 2001)

91. Sustaining organizational change (Gustafson et al., 2003; Rogers 1995; Plsek 2003; Champagne et al., 1991) Tension for change: Staff want a change Innovation-system fit: Innovation fits norms and values of organization Assessment of implications of innovation: Implications are thought about in advance Support and advocacy—Existence of champions and boundary spanners Dedicated time and resources Capacity to evaluate the innovation: Ability to monitor and evaluate the impact of the innovation

92. Areas for further study What are the key factors that improve the uptake and sustainability of efficacious treatments? What factors improve the fidelity of implementation efforts? What are the most effective outcome measures and suitable methodologies for dissemination and implementation? How do different stakeholder perspectives about EBPs affect organizational readiness to adopt new practices? What are the mediators and moderators of organizational effects? Can organizational context be changed to improve adoption of new practices? What are effective interventions for changing organizational culture and climate? How do family and consumer perspectives affect organizational readiness to adopt new practices?

93. Important Contributing Fields Social Marketing—packaging EBPs? Behavioral Change—why use an EBP? Anthropology—fit in different communities? Organizational behavior—can organizational environments be changed? Finance/Economics—is there an economic argument for EBPs? Technology development: EBPs are a kind of technology—how to efficiently incorporate new technologies in new environments?

94. Culture Structure Psychological Climate Organizational Climate Attitudes Social Norms Self-Efficacy Beliefs & Expectations Behavioral Intention Models of Diffusion, Organizational Implementation & Social Processes Systems Context Organizational Properties Individual & Shared Perceptions Behavior Structural Determinants of Organizational Innovation Social Determinants of Organizational Innovation Adapted from Glisson 2002

95. Concluding thoughts “There is no practice without theory, however much that theory is suppressed, unformulated, or perceived as obvious.” Northrup Frye/Belsey

96. Closing thought “New technologies alter the structure of our interests: the things we think about. They alter the character of our symbols: the things we think with. And they alter the nature of community: the arena in which thoughts develop.” Neil Postman, Technopoly

98. Dimensions of Organizational Readiness What factors are considered important to the uptake of evidence-based practices? Do all stakeholders agree on the relative importance of specific factors?

99. Service System Organization Service Delivery Client Practitioner Intervention Schoenwald & Hoagwood, 2001 Organiza-tional mandates Ethnicity/ cultural iden Salary level/ Criteria for increases Clarity of intervention Organiza-tional mission Gender Endorsement of intervention Complexity of intervention Interagency working relationship Organiza-tional climate Source of payment Age and dev- elopmental status Training of practitioner Similarity of int to std practice Legal mandate for referrals Organiza- tional culture Physical location of sessions Source of referral Supervisor/ Researcher Intervention specification Manual? Financing methods Personnel policies Length of sessions Family context Adherence monitoring Focus of intervention Policies of referral source, pay Structure, hierarchy Frequency of sessions Nature of referral problems Specialized training Nature of intervention theory

100. Intervention Characteristics Theoretical foundation, strength of research support, clinical foundation, precision, availability of manual, specificity of manual, clarify of model Practitioner Characteristics Clinical adherence to model, frequency of clinical supervision, structure of clinical supervision, type of clinician, treatment orientation of clinician Client Characteristics Referral problem(s), family context, client’s ethnicity/cultural identification Service Delivery Characteristics Referral source, frequency of treatment sessions, length of treatment sessions, setting/location of treatment sessions, setting/location of the clinic or school Service System Characteristics Salary incentives to adopt EBPs, policies and practices of referral sources, source of payments for the specific EBP, financing/payment mechanisms, legal mandates of referral sources, strength of interagency relationships DOMAIN 1 DOMAIN 2 DOMAIN 3 DOMAIN 4 DOMAIN 5 DOMAIN 6 Key Readiness Factors Service Agency Characteristics Endorsement by site leadership, structure of organization, size of organization, culture and climate of organization, policies and practices within the organization

101. Dimensions of Organizational Readiness (DOOR) Question: how important are the following factors: Frequency of clinical supervision required to deliver the EBT (e.g., consultations between clinicians & supervisors) Length of each treatment session required to deliver the EBT Strength of the research supporting the EBT … outside agencies … state mental health authorities … consumer advocacy … families or youth in the service setting … clinical staff (e.g., therapists, social workers, psychologists, psychiatrists) Support for the EBT by …. Clinic Directors Clinicians Consumers Researchers To Me

102. Mean Ratings of Readiness Factors Scale

103. Total Mean Ratings of Readiness Factors

104. Comparison of Respondent Group Ratings Scale