Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Measuring Sustainment of Multiple EBPs
Fiscally Mandated in Children’s Mental
Health Services: Knowledge Exchange on
Evide...
Objectives
1. Provide an overview of system driven EBP
implementations and community context and the
4KEEPS Study.
2. Desc...
Public Mental Health Care System (Local, State)
Organizations contracted to provide MH Care
Therapists providing direct se...
Public Mental Health Care System (Local, State)
Organizations contracted to provide MH Care
Therapists providing direct se...
Statewide Reform: California’s
Mental Health Services Act
Prevention and
Early Intervention
(PEI)
Los Angeles County
• Nation’s largest county
mental health
department
• 75 directly operated sites
• 288 contracted agenci...
Applying Aarons et al (2011) Model to LACDMH PEI Timeline
Applying Aarons et al (2011) Model to LACDMH PEI Timeline
Characteristics of Practices
Practice Age Range
(years)
Target Problem(s)
Triple P 2-12 conduct
CPP 0-6 trauma; attachment...
Applying Aarons et al (2011) Model to LACDMH PEI Timeline
Applying Aarons et al (2011) Model to LACDMH PEI Timeline
4KEEPS Study Aims
• A neutral, observational study investigating the
sustainment of original 6 practices with
implementati...
Multiple Sources of Data
LACDMH
Administrative
Claims Data
LACDMH Site
Visit
Documents
Online Surveys
of Program
Leaders a...
Today
2014 2015 2016 2017 2018
4KEEPS Study
launch
1/1/2014
Online survey 1
launch
3/1/2015
In-depth sample
launch
5/1/201...
Sustainment outcomes
• # & % agencies continuing to be reimbursed for the practice
• # & % Therapists continuing to claim ...
Practice Volume/Penetration
94
agencies
8,514
providers
87,100 children
2,331,000
psychotherapy claims
3,014,353 total cla...
Cumulative # of Claims Per
Practice
• MAP = 905,395
• TF-CBT = 662,184
• Seeking Safety = 515,208
• Triple P = 140,147
• C...
Volume Over Time
Initial increases in raw volume
for most practices in the first
quarters with peak and leveling
off after...
Gross Penetration Over Time
During the initial ramp-up
period, some practices
ramped up more quickly than
others.
Reorderi...
Characteristics of Practices
Higher Penetration (MAP, TF-
CBT, CPP)
• Apply to a broad age range of
clients
• Can be deliv...
Sustainment outcomes
• # & % agencies continuing to be reimbursed for the practice
• # & % Therapists continuing to claim ...
Measuring Sustainment: EBP
Concordant Care
Addresses concerns about the feasibility and
appropriateness of using tradition...
EBP Concordant Care Assessment
(ECCA) Development Process
Reviewed existing practice inventories.
Adapted/selected/gener...
Current ECCA
Purpose Assesses the extent to which a
therapist delivered individual
psychotherapeutic strategies
considered...
ECCA Data Collection To Date
• 710 therapists from 54 agencies
completed a Bi-Monthly ECCA
Full Sample
• 71 therapists sub...
ECCA Preliminary Findings
High internal consistency for each EBP Target Composites
for Anxiety, Conduct, Depression and T...
ECCA Preliminary Findings
Continued
Concordance between Session and Bi-monthly
version supporting by significant and larg...
Next Steps
1) Finish validation of therapist-report ECCA with
observational coding system
1) Projected sample of 1080 sess...
Potential Inner Context
Determinants of Sustainment
• Changes in state regulatory requirements
• Changes in implementation...
Therapist and Practice Characteristics
as Facilitators of Multiple EBP
Implementation
Practices Therapists
• The PEI Conte...
Demographics M (SD) or %
Age 37.00 (9.28)
Gender (female) 88%
Hispanic 43%
Race
White 50%
Asian/ Pacific Islander 11%
Afri...
Measuring Therapist Attitudes
Practice-Specific Attitudes General Attitudes towards EBP
Perceived Characteristics of Inter...
Attitudes Differ by Practice
F b
Intercept (grand mean) 9667.23*** 3.29***
TF-CBT 183.10 *** .45***
CPP 29.73*** .28***
Tr...
Practice Characteristics
Practice
PCIS
Alphas
Narrow Age
Range
Consultation
Required
Prescribed
Session
Content/ Order
CBI...
Therapist Attitudes Differ by
Practice Characteristics
1
2
3
4
5
Prescribed Session Content/
Order
Consultation Required N...
• Use sequential QUANqual
design to understand:
• The types of adaptations that
therapists make in community
mental healt...
Characterizing Adaptations
1
2
3
4
5
MeanExtensiveness
Fidelity Consistent
CBITS CPP SS TFCBT TP
1
2
3
4
5
Fidelity Incons...
Types of Adaptations
• Therapists reported more
fidelity consistent adaptations
(EMM= 9.30; SE = 1.79) than
fidelity incon...
Reasons for Adaptations
Culture
Crises
Adjust Order
Extend Pacing
Modify Presentation
Omit
Developmental
Level I think the...
Summary of Preliminary Findings
The scope and size of the PEI transformation provides a critical
opportunity to examine mu...
Funding Source: NIMH R01MH100134
Upcoming SlideShare
Loading in …5
×

Measuring Sustainment of Multiple EBPs in Children's Mental Health Services

246 views

Published on

Presentation:
"Measuring Sustainment of Multiple EBPs Fiscally Mandated in Children's Mental Health Services: Knowledge Exchange on Evidence-based Practice Sustainment (4KEEPS) Study"

Speaker:
Lauren Brookman-Frazee
Southern California Regional Dissemination, Implementation and Improvement Science Webinar Series

June 1, 2016

Sponsored by UCLA CTSI

Published in: Education
  • Be the first to comment

  • Be the first to like this

Measuring Sustainment of Multiple EBPs in Children's Mental Health Services

  1. 1. Measuring Sustainment of Multiple EBPs Fiscally Mandated in Children’s Mental Health Services: Knowledge Exchange on Evidence-Based Practice Sustainment (4KEEPS) Study Lauren Brookman-Frazee Southern California Regional Dissemination, Implementation and Improvement Science Webinar Series June 1, 2016
  2. 2. Objectives 1. Provide an overview of system driven EBP implementations and community context and the 4KEEPS Study. 2. Describe study methods and initial findings: • Characterizing sustainment outcomes • Characterizing potential inner context determinants of sustainment outcomes
  3. 3. Public Mental Health Care System (Local, State) Organizations contracted to provide MH Care Therapists providing direct services to consumers Exploration Phase Preparation Phase Implementation Phase Sustainment Phase Phases and Levels of EBP Implementation Adapted from Aarons, Hurlburt & Horwitz, 2011
  4. 4. Public Mental Health Care System (Local, State) Organizations contracted to provide MH Care Therapists providing direct services to consumers Exploration Phase Preparation Phase Implementation Phase Sustainment Phase Sustained Delivery of Multiple EBPs in System-Driven Implementation Efforts Adapted from Aarons, Hurlburt & Horwitz, 2011
  5. 5. Statewide Reform: California’s Mental Health Services Act Prevention and Early Intervention (PEI)
  6. 6. Los Angeles County • Nation’s largest county mental health department • 75 directly operated sites • 288 contracted agencies (120 with child MH services) • 8 Service Provision Areas range from rural to inner city communities • Serves an ethnically diverse, disadvantaged population Hispanic/Latino Asian/Pacific Islander African American
  7. 7. Applying Aarons et al (2011) Model to LACDMH PEI Timeline
  8. 8. Applying Aarons et al (2011) Model to LACDMH PEI Timeline
  9. 9. Characteristics of Practices Practice Age Range (years) Target Problem(s) Triple P 2-12 conduct CPP 0-6 trauma; attachment MAP 0-21 anxiety; trauma; depression; conduct TF-CBT 3-18 trauma CBITS 11-15 trauma Seeking Safety 13-18 trauma; substance use
  10. 10. Applying Aarons et al (2011) Model to LACDMH PEI Timeline
  11. 11. Applying Aarons et al (2011) Model to LACDMH PEI Timeline
  12. 12. 4KEEPS Study Aims • A neutral, observational study investigating the sustainment of original 6 practices with implementation support. • Aim 1: Characterize sustainment outcomes • EBP Concordant Care • Volume/penetration of each practice over time • Aim 2: Use mixed methods to characterize inner context factors and early implementation conditions that potentially predict EBP sustainment. • Aim 3: Identify inner context and early implementation conditions that determine sustainment outcomes Funded by NIMH Grant # R01 MH100134 MPI Anna Lau and Lauren Brookman-Frazee
  13. 13. Multiple Sources of Data LACDMH Administrative Claims Data LACDMH Site Visit Documents Online Surveys of Program Leaders and Therapists Semi- Structured Interviews Session Recordings Sustainment outcomes and potential inner context determinants of outcomes
  14. 14. Today 2014 2015 2016 2017 2018 4KEEPS Study launch 1/1/2014 Online survey 1 launch 3/1/2015 In-depth sample launch 5/1/2015 Online survey 1 supplement 2/1/2016 In-depth sample wrap-up Spring 2017 Online survey 2 launch Spring 2017 4KEEPS StudyTimeline 4KEEPS Events Timeline
  15. 15. Sustainment outcomes • # & % agencies continuing to be reimbursed for the practice • # & % Therapists continuing to claim to the practice • # & %Unique clients served by the practice • # $ % Units of services being provided within each practice Practice Volume/Penetration • Degree to which a therapist’s practice resembles the essential strategies one would expect within an evidence-based protocol for a given problem focus. EBP Concordant Care
  16. 16. Practice Volume/Penetration 94 agencies 8,514 providers 87,100 children 2,331,000 psychotherapy claims 3,014,353 total claims 2009 2015
  17. 17. Cumulative # of Claims Per Practice • MAP = 905,395 • TF-CBT = 662,184 • Seeking Safety = 515,208 • Triple P = 140,147 • CPP = 105,231 • CBITS = 2835
  18. 18. Volume Over Time Initial increases in raw volume for most practices in the first quarters with peak and leveling off after the initial few years Highlight the rapid impact of a fiscal policy change restricting reimbursement to specific practices and training in these practices
  19. 19. Gross Penetration Over Time During the initial ramp-up period, some practices ramped up more quickly than others. Reordering and stabilization after 2 years MAP, TF-CBT, SS > Triple P, CPP, CBITS
  20. 20. Characteristics of Practices Higher Penetration (MAP, TF- CBT, CPP) • Apply to a broad age range of clients • Can be delivered in multiple settings • Addresses common presenting problem -trauma • MAP also covers a range of presenting problems • Train-the-trainer capacity (MAP, SS) • Minimal training required for billing (SS) Lower Penetration (Triple P, CPP, CBITS) • Apply to a narrower age range of clients (CPP, CBITS) • Restrictions on settings/format • Group (CBITS, Triple P) • School setting (CBITS) • Caregiver directed (CPP, Triple P) • Additional requirements to deliver such as MOAs between programs and schools (CBITS)
  21. 21. Sustainment outcomes • # & % agencies continuing to be reimbursed for the practice • # & % Therapists continuing to claim to the practice • # & %Unique clients served by the practice • # $ % Units of services being provided within each practice Practice Volume/Penetration • Degree to which a therapist’s practice resembles the essential strategies one would expect within an evidence-based protocol for a given problem focus. EBP Concordant Care
  22. 22. Measuring Sustainment: EBP Concordant Care Addresses concerns about the feasibility and appropriateness of using traditional fidelity instruments to asses ongoing delivery of multiple EBPs
  23. 23. EBP Concordant Care Assessment (ECCA) Development Process Reviewed existing practice inventories. Adapted/selected/generated items (strategies). Collected data from practice experts about how essential/interfering strategies are to their practice. Delphi rating system used to determine item selection for therapist-report ECCA (alpha version). Collected data from large sample of therapists using ECCA (alpha version). Examine the properties of the therapist-report ECCA (alpha version). Validate therapist-report ECCA with observational coding system. Finalize therapist-report ECCA based on #6 and 7.
  24. 24. Current ECCA Purpose Assesses the extent to which a therapist delivered individual psychotherapeutic strategies considered essential for a given EBP Target Online Versions Session (therapist report, observer rated) and Bi-monthly (therapist report) Items 38 psychotherapeutic content and techniques EBP Targets #Content #Techniques Conduct 18 9 Trauma 10 5 Anxiety 7 7 Depression 9 6
  25. 25. ECCA Data Collection To Date • 710 therapists from 54 agencies completed a Bi-Monthly ECCA Full Sample • 71 therapists submitted 459 Session ECCAs with corresponding audio recordings of sessions as well as Bi-Monthly versions for each client Validation Sub-Sample
  26. 26. ECCA Preliminary Findings High internal consistency for each EBP Target Composites for Anxiety, Conduct, Depression and Trauma (alphas ranged from .86 to .95; M=.90). Discriminant validity of the scales supported by significant differences in EBP Target composites by EBP delivered  When therapists delivered an EBP that targeted Conduct or Trauma, their ratings on the content items from the corresponding composite were significantly higher than for other EBP targets Construct Validity supported by Item Response Theory Analyses indicating that items with high item difficulty appeared to include strategies that reflect high integrity EBP concordant care (Ignoring/Differential Reinforcement of Other Behaviors > Praise)
  27. 27. ECCA Preliminary Findings Continued Concordance between Session and Bi-monthly version supporting by significant and large correlations between the average Session ECCA with the Bi-Monthly version on the EBP Target composites (range= .62 to .71; M=.68). Concordance between therapist report and direct observation in process  Early data based on a small subset of therapist reports with corresponding observer ratings indicate greater concordance for strategies that are part of the practice type being delivered.
  28. 28. Next Steps 1) Finish validation of therapist-report ECCA with observational coding system 1) Projected sample of 1080 sessions with therapist report and observer ratings 2) Examine inner context factors associated with ECCA composite scores. 3) Refine ECCA instrumentation based on concordance analysis and end user feedback on utility.
  29. 29. Potential Inner Context Determinants of Sustainment • Changes in state regulatory requirements • Changes in implementation strategies used • Adoption, de-adoption of EBPs over time System Level • Implementation support • Organizational climate • Early implementation condition Organizational Level • Therapist attitudes: perceptions of effectiveness and fit • Therapist clinical adaptations to practices Therapist Level
  30. 30. Therapist and Practice Characteristics as Facilitators of Multiple EBP Implementation Practices Therapists • The PEI Context and fiscal mandate for EBP reform presents a unique context • EBP vs. Practice as Usual • EBP1 vs. EBP2 vs. EBP3… • Allows for parsing the variance in attitudes associated with • Therapists and therapist characteristics • Practices and practice characteristics
  31. 31. Demographics M (SD) or % Age 37.00 (9.28) Gender (female) 88% Hispanic 43% Race White 50% Asian/ Pacific Islander 11% African American 7% American Indian/Alaska Native 1% Multiracial 8% Other 23% Deliver Services in >1 Language Spanish 47% Other Language 10% Survey: Therapist Characteristics (n=790) Training Background M (SD) or % Years Practiced as Therapist 7.21 (6.16) Years Worked at Current Agency 5.12 (4.56) Current # Clients 14.65 (10.48) MH Discipline MFT 55% Social Work 30% Clinical Psychology 11% Counseling 2% School Psychology <1% Other 2%
  32. 32. Measuring Therapist Attitudes Practice-Specific Attitudes General Attitudes towards EBP Perceived Characteristics of Intervention Scale, Cook et al., 2014 • Relative Advantage • Complexity • Compatibility • Potential for Reinvention • Total Evidence-Based Practice Attitudes Scale, Aarons, 2004 • Openness • Divergence
  33. 33. Attitudes Differ by Practice F b Intercept (grand mean) 9667.23*** 3.29*** TF-CBT 183.10 *** .45*** CPP 29.73*** .28*** Triple P 11.77** .16** MAP 5.28* .08* SS 11.47** -.12** CBITS 117.16*** -.84*** *p<.05; **p<.01; ***p<.001
  34. 34. Practice Characteristics Practice PCIS Alphas Narrow Age Range Consultation Required Prescribed Session Content/ Order CBITS (N = 65) .96    CPP (N = 140) .94   MAP (N = 527) .92  SS (N = 491) .93  TF-CBT (N = 582) .93   Triple P (N = 184) .94 
  35. 35. Therapist Attitudes Differ by Practice Characteristics 1 2 3 4 5 Prescribed Session Content/ Order Consultation Required Narrow Age Range PCISITEMMEAN PRACTICE CHARACTERISTIC Yes No * ** * = p < .001 Very Great Extent Not at All
  36. 36. • Use sequential QUANqual design to understand: • The types of adaptations that therapists make in community mental health settings. • Which therapist characteristics predict types of adaptations (fidelity consistent vs. inconsistent). • The reasons therapists make adaptations. Therapist Adaptations Fidelity Consistent Modify presentation Integrate supplemental content Lengthen/exten d Pacing Fidelity Inconsistent Remove/skip components Shorten/condense pacing Adjust order of sessions/components
  37. 37. Characterizing Adaptations 1 2 3 4 5 MeanExtensiveness Fidelity Consistent CBITS CPP SS TFCBT TP 1 2 3 4 5 Fidelity Inconsistent
  38. 38. Types of Adaptations • Therapists reported more fidelity consistent adaptations (EMM= 9.30; SE = 1.79) than fidelity inconsistent (EMM= 5.88; SE = 1.79) adaptations • F(1, 1,075.59) = 1,332.00, p < .001 • Therapist-reported attitudes towards a practice: • Did not predict fidelity consistent adaptations B = - .02, t = -1.47, p = .14 • Did predict fidelity inconsistent adaptations • B = -.07, t = -5.67, p < .001 0 1 2 3 4 5 6 7 8 9 10 Fidelity Consistent Fidelity Inconsistent MeanComposite CBITS CPP SS TFCBT TP
  39. 39. Reasons for Adaptations Culture Crises Adjust Order Extend Pacing Modify Presentation Omit Developmental Level I think the length of treatment is part of that too, especially with the Hispanic families. Like you can’t just jump right in; you need to spend a little more time building rapport. Some of the examples that they have there I think, you know, might not necessarily fit like this particular culture so I’ll find something that they could relate to a little bit better, a situation they can relate to a little bit better. I did work with one client that was developmentally delayed. So for some of the cognitive behavioral piece it was just really difficult to understand…I did my best to teach it, but I didn’t feel like I focused a lot on it because dude wasn’t really understanding it. Oh, we’ll still go back to some of the relaxation components, or we’ll kind of focus on what we can do to manage our anxiety about it and whatnot. You know, with people that are developmentally delayed or whether there needs to be more repetition, I’ll repeat it more. I think it’s just different working with a child that young. So we do a lot of art. So instead of like a trauma narrative, we’ve been doing a lot of art on the chalkboard. You do have those clients that maybe might become homeless, or maybe parents are having a really hard time, maybe with other kids, older kids. So you might have a lot of crises that might surface throughout treatment, and that might postpone certain parts of it
  40. 40. Summary of Preliminary Findings The scope and size of the PEI transformation provides a critical opportunity to examine multiple EBP implementation in a large and diverse natural laboratory Sustainment varies by practice and over time Inner context factors are critical to examine within EBP implementation efforts • Therapist attitudes vary across practices. • Therapists report more extensive fidelity consistent adaptations than fidelity inconsistent adaptations
  41. 41. Funding Source: NIMH R01MH100134

×