Defines treatment integrity and discusses ways to approach it and barriers

1. Treatment Integrity:
What it is and How to do it
Presented by:
Karen Umstead-Yosmanovich, M.Ed, B.C.B.A, LBS
Beautiful Minds of Princeton
“Teach, Reach, & Expand Potential”
www.beautifulmindsofprinceton.com
kyosmanovich@beautifulmindsofprinceton.com
1

2. Who am I?
Who am I ?
– Teacher (Special/General Education) in private and public
schools
– Board Certified Behavior Analyst (BCBA)
– Executive Director of BMOP
– Adjunct Professor
• Populations worked with
– Schools
– Agencies
– Community Providers
– Families
2

3. Beautiful Minds of Princeton
“Teach, Reach, & Expand Potential”
For more information:
Call: 1-800-675-2709
Email:
kyosmanovich@ beautifulmindsofprinceton.com
or Visit us:
www.beautifulmindsofprinceton.com
3

4. Who are you
• BCBA
• University/Researcher
• Teacher
• Child Study
• Related Services
• Administrator
• Parent
4

5. Warm Up Activity
What do you know: (thumbs up, in middle, down)
Definition of treatment integrity
Multi-dimensional approach to treatment integrity
Barriers to treatment integrity
Variables or interventions that increase treatment
integrity
What do you want to know (why are you here)
More info on defining integrity
Dealing with barriers
Specific variables or interventions to increase integrity
5

6. Our Objectives
• Participants will be able to:
 Label and define multidimensional components of
treatment integrity, including related variables
 Identify at least four methods of assessing
treatment integrity
 Label at least 3 solutions to common challenges to
high treatment integrity in school and home
settings
6

7. Proposed definition
• Multidimensional construct (from several fields)
• Not as simple as “Did they do it?”
• Not just percentage of components implemented as planned
• Content related (quantity)
– “How much of the intervention was implemented?”
• Process related (quality)
– “How well was the intervention implemented?”
7

8. Process-related
8

9. Creating a Treatment Integrity Tool
• Define treatment integrity
• Pick an appropriate assessment method
• Delineate the intervention into operational
steps
• Decide how intervention steps will be rated
and the measure will be summed
• Determine how often treatment integrity will
be assessed
9
(Collier-Meek, Fallon, Sanetti, & Maggin, 2013; Gresham, 1989; Gresham, 2013; Sanetti, Fallon, & Collier-
Meek, 2011; Sheridan, Swanger-Gagne, Welch, Kwon, & Garbacz, 2009)
From PRIME: Planning Realistic Implementation and Maintenance by Educators

10. Content-Related
• Adherence
– This is the most common one we are used to
– Requires operational definitions of the intervention
steps
• Exposure
– Is it being delivered as often as it supposed to?
– For the length of the time it is supposed to happen?
• Another view or key term in research literature
– Errors of Omission (not doing something)
10

11. Content-Related
• Program Differentiation
– Just like we consider it when doing research
– Especially important when comparing two
approaches or programs or when doing a
component analysis of a multi-component plan
– Often times implementers start to incorporate other
elements into existing treatment
– Need to understand the “MUST do” versus
“Would LIKE to do”
• Another view or key term in research literature
– Errors of Commission (doing something wrong) 11

12. 12
From PRIME: Planning Realistic Implementation and Maintenance by Educators

13. Example
NET (natural environment teaching)- Procedures for Early, Intermediate, & Advanced Students
Staff:____________________ Student:____________________ Observer:____________________ Date:____________________
Section 1: Setting Section 4: Behaviors
____Area is sanitized ____Staff uses appropriate antecedent strategies
____Materials needed are organized and ready ____Staff takes accurate behavior data
____Variety of reinforcers are available ____Staff implements behavior interventions correctly
____NET lesson plan/targets are updated and visible ____Staff pairs social reinforcement with tangible
reinforcers
Section 2: NET Teaching Procedures
Section 5: Data Collection
____Staff follows student’s MO
____Begins session with manding ____Any necessary mand data sheets are updated & ready
____Appropriate level of enthusiasm ____Mand data is taken
____Staff’s voice is natural ____ Behavior data sheets are updated & ready
____Staff provides adequate number of manding trials ____Behavior data is taken
____Staff fades prompts as quickly as possible ____NET lesson plans/target data sheets are updated &
____Staff follows appropriate variable ratio of reinforcement (VR) schedule ready
____Number of demands is faded in ____Data on targets is taken
____Difficulty of demands is faded in
____Staff generalizes skills already taught at IT across:
____Environments
____People
____Stimuli
____Staff mixes verbal operants
____Staff uses errorless teaching with appropriate time delay prompts:
____0 second time delay for new targets
____2-3 second time delay for acquisition or maintenance targets
____Staff uses transfer trials (re-present Sd after error and prompts)
____Staff differentially reinforces novel behaviors/appropriate play skills
____Skills are taught to fluency
____Extinction is used for problem behavior
Rating: 1= Yes/Consistently 2= Sometimes/Inconsistent 3= No/Not Occurring 4= N/A Not Applicable
13
From PATTAN Resource Guide for Autism Initiative

14. 14
Natural Environment Teaching Evaluation form
Teacher: ______________________________ Date: ___________________________
Observer: _____________________________ Activity: _________________________
Area 1: Organization Area 4: Reinforcement
____ Instructional area is neat and clean ____ Sr+ Reinforcer competes with Sr-/SrA+
____ All materials needed are organized and ready ____ Pairs social reinforcement w/ tangible items
____ Begins promptly/avoids wasted time
____ Lesson plan current and visible Area 5: Behavior Management
____ Correctly implements extinction procedures
Area 2: Instructional Delivery ____ Maintains composure during procedures
____ Follows EO of student ____ Accurately records behavior data
____ Begins NET session with manding ____ Implements effective antecedent interventions
____ Positioned at child’s eye level
____ Appropriate level of enthusiasm Area 6: Error Correction
____ Mixes verbal operants ____ Re-presents after error w/ Sd and 0 sec time delay
____ Uses errorless teaching with appropriate time delay ____ Returns to target several trials later
____Averages 4-5 responses per minute ____ Requires correct response
____Spends recommended time per day in NET ____ Uses non-verbal stimulus to evoke responses
Area 3: Data Collection Additional Comments: __________________________
____ NET data sheet is available and data is taken
____ Data is graphed by verbal operant ______________________________________________
______________________________________________
Rating: 1 = consistently 2 = Sometimes/inconsistently
3 = Not occurring N/A = Not applicable
Responses Across Verbal Operants: 3 Minute Sample
Mands Tacts Receptive Intraverbals Motor Imitation Echoics
Responses Per Minute: 1 Minute Timing
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 18 20
From PATTAN Resource Guide for Autism Initiative

15. Process-Related
• Harder to operationally define (should still
attempt to though)
• Qualitative
• Quality of delivery
– How well was the treatment delivered
– Can tie into some of the content-related elements
– Often times a rating scale
• Participants responsiveness
15

16. 16
From PRIME: Planning Realistic Implementation and Maintenance by Educators

17. Participant Responsiveness
• Implementer and intended receipt of intervention
• Enlisting help to ensure integrity
– Fruit helper of the day (study on increasing fruit)
• Implementers wanted to quit, were able to identify solution
by incorporating kids
• Treatment acceptability research has not shown a
functional relationship between acceptability of
intervention and integrity of implementation
• Responsiveness relates engagement in the
program/intervention vs. acceptability 17

18. Example: Usage Rating Profile
18
Chafouleas, S.M., Briesch, A.M., Neugebauer, S. R., & Riley-Tillman, T.
C. (2011). Usage Rating Profile – Intervention (Revised). Storrs, CT:
University of Connecticut.

19. Example: Usage Rating Profile
• Factor I: ACCEPTABILITY Items - 1, 7, 9*, 11, 12, 18, 21,
22, 23
• Factor II: UNDERSTANDING Items – 4, 6, 25
• Factor III: HOME SCHOOL COLLABORATION Items – 5,
15, 28
• Factor IV: FEASIBILITY Items – 3, 8, 13, 17, 19*, 27
• Factor V: SYSTEM CLIMATE Items – 10, 14, 16, 20, 26
• Factor VI: SYSTEM SUPPORT Items – 2, 24, 29
19
Chafouleas, S.M., Briesch, A.M., Neugebauer, S. R., & Riley-Tillman, T.
C. (2011). Usage Rating Profile – Intervention (Revised). Storrs, CT:
University of Connecticut.

20. Intervention characteristics
Consider for implementation, include…
•Perception of intervention benefit, as compared to current
practice
•Intervention complexity
•Time or duration required
•Materials and resources required
•Number of interventionists required
•Compatibility
•Rate of behavior change
•Extent to which the intervention can be adapted
20
Sanetti & Kratochwill (2009)
From PRIME: Planning Realistic Implementation and Maintenance by Educators

21. Please do the following
• Draw a 2 dimensional picture that represents a
plane with a parabola and include a line
indicating the vertex
•
• Extend your adductor pollicis while
simultaneously lifting your flexor digitorum
superficialis above your ossicles
21

22. 22

23. Interventionist Characteristics
• Comprehension– does the implementer
actually know what the plan is saying?
• Feasibility – can the implementer actually do
the components of the plan?
• Acceptability – will the plan itself and related
student behavior be considered appropriate to
the implementer and others in the setting?
23

24. Readability and Comprehension
• Ease of understanding or comprehension
• National Average is 8th
grade reading level
– Know your audience and it’s background
• SMOG formula (McLaughlin 1969)
• Readability Assessment Instrument (RAIN)
(Singh, 2003)
24
Singh et al 2009

25. RAIN Suggestions
• Text must have titles and subtitles with
highlighting
• Introductory or summary statement;
• Pronoun references and connectives (e.g.
conjunctions) must be clear
• New words highlighted and accompanied by a
definition or synonym
• Use 12 – 14 pt print size
25
Singh et al 2009

26. Suggestions
• Can check readability with Microsoft Word
• Under Options for Spelling/Grammar
– Check readability
– Run spelling and grammar check
– At the end it will give you a reading level and ease
of reading
• Get feedback from people who will implement
– Change wordings if need be or clarify
• If you must use technical jargon, pair it with
easy to understand terms or definitions 26

27. Don’t forget to consider culture
• Your OWN
– Ex: Your priorities versus others (the case of eating
with forks and knives versus hands)
• Organizational
– Ex: We can’t allow that here
• Implementers
– Ex: What does agreement looking (e.g. nodding
head does not indicate agreement in some cultures,
but means I heard you)
27

28. Collecting Data
• Several ways to do this:
– Direct observation
– Permanent Product
– Self-monitoring, self-reporting, behavioral
interviews
There are Pros and Cons to each approach!
28

29. Direct observation
Pros Cons
•See exactly what is being done
•Most accurate representation
•Can account for program
differentiation issues or external
variables
•Reactivity
•Time consuming
•May be cost prohibitive
(funding source)
29

30. Permanent Product
Pros Cons
•Less chance of reactivity
•More efficient
•Potentially more accurate than other
assessments
•You don’t see what is going on
•Not all interventions result in a
permanent product
30Wilkinson 2007

31. Self-monitoring, self-reporting,
and behavioral interviews
Pros Cons
•Not as time consuming for the evaluator
•Treatment implementers take ownership
for their actions
•Honesty?
•Don’t actually see what is happening
31

32. Self Monitoring Example
32

33. Self Monitoring Example
33

34. Treatment Integrity Tools
• Treatment Monitoring Interview
– Semi-structured interview meant to increase the
strength of a plan by providing direct support and
performance feedback.
• Treatment Integrity Planning Protocol
– Standardized process for planning and creating a
treatment integrity assessment
34

35. True or False?
• Better treatment integrity results in better
outcomes.
35

36. True or False?
• Better treatment integrity results in better
outcomes.
– It depends!
– There are other factors involved in determining
outcomes other than treatment integrity.
– How much treatment integrity is needed depends
on the intervention itself – some interventions
require more stringent treatment integrity than
others.
36

37. True or False?
• Treatment integrity must remain high at all
times for an intervention to be successful.
37

38. True or False?
• Treatment integrity must remain high at all
times for an intervention to be successful.
– This would be nice, BUT not necessarily true.
• Treatment drift happens.
– Don’t think of treatment integrity as an all or
nothing.
• We can view it as a percentage (25%, 50%, 100%)
38

39. Plan from the start
• When you design the plan/intervention,
consider how you can easily adapt materials to
check for treatment integrity
• Determine what aspects of integrity you want
to check
• Determine a schedule of assessment
39

40. Determine barriers to treatment
• What are the barriers to treatment
• What can be done to address the barrier ahead of time
– Staff says “I don’t have time to do prize box at end of day”
» Can an assistant do it? Is there another time/way for it to
be done? Can the student go to another person for prize
box (think of level 2/secondary level interventions in
PBS)?
• Consider best practices and work from there
– This is what best practice would say…now how can we make
it feasible
– Start from a positon of strength and then negotiate
– CONSIDER THE FUNCTION!
40

41. Training
• Know your audience
• Direct methods
– Modeling, role –playing, rehearsal and feedback
• Indirect methods
– Didactic instruction
– Written instruction
41

42. Video clips for training OR
feedback
• Online
• Youtube
• Made by you
– Example- easily add “captions” to video and share with staff on secure
drive
• Used a combination of short-video clips (3-5 min) and feedback
– Incorporated video-self monitoring, Teacher Performance Rate and Accuracy Scale or
Form (TPRA), and mentoring/feedback from “expert”
– Slim, Lina, "Exploring A Staff Training Model For Enhancing Post-Training Procedural Integrity And
Staff Performance Outcomes When Working With Children Diagnosed With ASD" (2015). Seton Hall
University Dissertations and Theses (ETDs). Paper 2058.
42

43. Staff Training
• Important to distinguish staff’s ability to perform skill versus
verbally describing skill or enhancing knowledge
• EBP consists of performance-based and competency based
strategies
– Performance refers to trainer and trainee actions during training
– Competency refers to continuing training until mastery criterion is met
• Training is data-based
• Essentially being referred to in literature as behavioral skills
training (we can’t cover more specifics, but online handouts
include more slides at end of presentation). BST should also
be covered in supervision training for BCBAs
43
Parson et al 2012

44. Initial training is just the start
• We cannot train and hope
– Several studies showed that post initial training-
high treatment integrity lasted 0-10 days.
• How often has a plan worked great initially
and then you hear about problems 2 months
later?
• Treatment drift
44

45. Performance Feedback
• Consider:
– What?
• Specify exactly what feedback will be about.
– Where?
– When?
• Immediately? Daily? Weekly?
– How?
• Verbal, written, graphic
• Try an incorporate preferences when possible
45

46. School Challenges
• Using negative reinforcement to increase
treatment integrity (teachers avoided
consultant meeting by achieving integrity)
• Meeting summaries
– Write up any decisions/changes that were agreed
upon at the meeting.
– Include action steps ((who will do what by when
(and if need be how))
– Delineate roles and responsibilities
– Role/Job analysis 46

47. School Challenges
• Buy-in and accountability
– Engage student, various staff members, and
parents as much as possible
• REINFORCE!!!!
• By engaging others, individuals more hesitant may get
“swept” along
• Have the student help with elements of intervention
(e.g. graphing own data = math practice)
• Start with one routine or teacher then expand
– Prevent-Teach Reinforce example
47

48. School Challenges
• Permanent product methods to assist/augment
direct observation.
• Write treatment integrity checks into plan.
Prepare and include sample forms with your
assessment/BIP.
• Using Google Docs to assess other elements
48

49. School Challenges
• Connect with other resources that can support
intervention
– Autism Internet Modules
– Evidence Based Practice Briefs from National
Professional Development Center on Autism
Spectrum Disorders
– PBIS
• Internal coaches
– Pyramidal approach to training
• Train the Trainer model 49

50. Scheduling
• Mini-training series (start of day or end of day
required time by contract)
• Augment with video-taping and sharing
securely
– Could even be used by implementer for self-
assessment
– Could be used by consultant to model intervention,
including important elements (captioning)
50

51. Scheduling
• Secure conferencing
• Utilize auxiliary resources
– Mentor Teacher
– School Psychologist
– Case Manager
– Administration
51

52. Home Challenges
• Why do we face treatment integrity issues in
the home?
– Parents may not agree on what is the best approach
OR
– Who has more energy to tough it out?
– Disconnect between parents and grandparents
52
Gulick (2014)
From Improving Treatment Integrity in Home-based Settings

53. How do we get better treatment
integrity in the home:
– We have to show that our approach will ultimately
be more EFFICIENT
– We address the various factors of parental
resistance
• Faulty rules – self-generated or found in media
– Consider teaching parent behavior that is
necessary to implementing a treatment plan
BEFORE having the parents do it.
• Shaping
53
Gulick (2014)
From Improving Treatment Integrity in the Home-based Setting

54. Specifically…
• We can collaborate with school to see how
they are doing things
– Videos
– Observation sessions
– If there is a home program aspect, have them come
in and demonstrate skills to parents/families
54

55. Beautiful Minds of Princeton
“Teach, Reach, & Expand Potential”
For more information:
Call: 1-800-675-2709
Email:
kyosmanovich@ beautifulmindsofprinceton.com
or Visit us:
www.beautifulmindsofprinceton.com
55© BMOP 2010 www.beautifulmindsofprinceton.com

56. References
Allen, K.D., & Warzak, W.J.(2000). The problem of parental nonadherence in clinical behavior analysis:
Effective treatment is not enough. Journal of Applied Behavior Analysis, 33(3), 373-391.
Collier-Meek, M. (2013). Increasing parents’ treatment integrity to a homework intervention through
conjoint behavioral consultation and performance feedback (Doctoral Dissertation). Retrieved from
http://digitalcommons.ucon.edu/dissertations
Courtemanche, A., Sheldon, J., Sherman, J., Schroeder, S., Bell, A., & House, R. (2014). Assessing the
effects of a staff training package on the treatment integrity of an intervention for self-injurious
behavior. Journal of Developmental and Physical Disabilities, 26(4), 371-389.
doi: 10.1007/s10882-014-9372-6.
Dart, E. H., Cook, C.R., Collins, T.A., Gresham, F.M., & Chenier, J.S. (2012). Test driving interventions
to increase treatment integrity and student outcomes. School Psychology Review, 41(4), 467-481.
De Fazio, C.M., Fain, A.C., & Duchaine, E.L. (2011). Using treatment integrity in the classroom to bring
research and practice together. Beyond Behavior, 20(1), 45-49.
DiGennaro-Reed, F.D., Reed, D.D., Baez, C.N., & Maguire, H. (2011). A parametric analysis of errors of
commission during discrete-trial training. Journal of Applied Behavior Analysis, 44(3), 611-615.
Fiske, K.E. (2008). Treatment integrity of school-based behavior analytic interventions: A review of the
research. Behavior Analysis in Practice, 1(2), 19-25.
Fryling, M.J., Wallace, M.D., & Yassine, J.N. (2012). Impact of treatment integrity on intervention
effectiveness. Journal of Applied Behavior Analysis, 45(2), 449-453.
Gresham, F.M., Gansle, K.A., & Noell, G.H. (1993). Treatment integrity in applied behavior analysis
with children. Journal of Applied Behavior Analysis, 26(2), 257-263. 56

57. References
Gulick, R.F (2014). Improving treatment integrity in home-based services [PowerPoint Slides].
Retrieved from http://www.achievemntctr.org/files/abai-presentation-2014-gulick.pdf
Hagermoser Sanetti, L.M., Fallon, L.M., & Collier-Meek, M.A. (2013). Increasing teacher treatment
integrity through performance feedback provided by school personnel. Psychology in the Schools,
50(2), 134-150.
doi: 10.1002/pits.21664.
Hagermoser Sanettin, L.M., & Kratochwill, T. R. (2008). Treatment integrity in behavioral consultation:
Measurement, promotion, and outcomes. International Journal of Behavioral Consultation and
Therapy, 4(1), 95-114.
Hagermoser Sanetti, L.M., Kratochwill, T. R. (2009). Toward developing a science of treatment integrity:
Introduction to the special series. School Psychology Review, 38(4), 445-459.
Hagermoser Sanetti, L. M., & Kratochwill, T. R. (2011). An evaluation of the Treatment Integrity
Planning Protocol and two schedules of treatment integrity: Impact on implementation and report
accuracy. Journal of Educational and Psychological Consultation, 21(4), 284-308.
doi: 10.1080/10474412.2011.620927.
Kuhn, S.A.C., Lerman, D.C., & Vorndan, C.M. (2003). Pyramidal training fro families of children with
problem behavior. Journal of Applied Behavior Analysis, 36(1), 77-88.
McIntyre, L.L., Gresham, F.M., DiGennaro, F.D., & Derek, D.R. (2007). Treatment integrity of
school-based interventions with children in the Journal of Applied Behavior Analysis. Journal of
Applied Behavior Analysis, 40(4). 659-672. 57

58. References
Page, T.J., Iwata, B.A., & Reid, D.H. (1982). Pyramidal training: A large-scale application with
instructional staff. Journal of Applied Behavior Analysis, 15(3), 335-351.
Plavnick, J.B., Ferreri, S.J., & Maupin, A.N. (2010). The effects of self- monitoring on the procedural
integrity of a behavioral intervention for young children with developmental disabilities. Journal of
Applied Behavior Analysis, 43(2), 315-320.
Sanetti, L.M.H. (2012). Assessing and promoting high levels of treatment integrity of school-based
Interventions [PowerPoint Slides]. Retrieved from:
http://www.cber.uconn.edu/up-content/uploads/2012/02/NE-PBIS-2011-10-03-11.pdf
Sanetti, L.M.H., Collier-Meek, M.A., Byrne, D.C., Byron. J.R., Wisniewsk, A., & Dobey, L. (2014).
Research-based strategies to assess and promote educators’ intervention implementation
[PowerPoint Slides]. Retrieved from
http://www.primeimplementation.com/wp-content/uploads/Promoting-TI-Miniskills_NASP2014.pdf
Sanetti, L., Williamson, K., Byron, J., Long, A., Kim, J., & Kratochwill, T. (2014). Defining and
quantifying treatment integrity: Conceptual and methodological barriers to assessment and
interpretation [PowerPoint Slides]. Retrieved from
http://www.primeimplementation.com/wp-content/uploads/Measuring-TI_NASP2014.pdf
Schulte, A.C., Easton, J.E., & Parker, J. (2009). Advances in treatment integrity research:
Multidisciplinary perspectives on the conceptualization, measurement, and enhancement of
treatment integrity. School Psychology Review, 38(4), 460-475.
58

59. References
Solomon, B.G., Klein, S.A., & Politylo, B.C. (2012). The effect of performance feedback on
teachers' treatment integrity: A meta-analysis of the single-case literature. School Psychology
Review, 41(2), 160-175.
St. Peter Pipkin, C., Vollmer, T. R., & Sloman, K. N. (2010). Effects of treatment integrity failures
during differential reinforcement of alternative behavior: A translational model. Journal of
Applied Behavior Analysis, 43(1), 47-70.
Vollmer, T.R., Sloman, K.N., St. Peter Pipkin, C. (2008). Practical implications of data reliability and
treatment integrity monitoring. Behavior Analysis in Practice, 1(2), 4-11.
59

60. Staff Training
• Step 1: Describe target skill (instruction)
– Include rationale as to why important
– Behaviorally define target skill
• Include performance checklist of necessary staff actions
• Step 2: Succinct Written Description
– This is not the same as the formal behavior plan
– This is essentially just the basics and a cheat sheet
– Can refer back to background documents like FBA
and full BIP
– Keep it simple and watch the language used 60
Parson et al 2012

61. Staff Training
• Step 3 Demonstrate the skill (model)
– Can use role-play or video model
– One person is staff, One is targeted individual
– Make sure well-scripted/planned
• Can use cue cards to prompt people to perform certain
actions (e.g. make an error, no response, touch only the
left side, etc)
– Stop and freeze at certain points to describe what
is being done and why
61
Parson et al 2012

62. Staff Training
• Step 4 Trainee practices skill (rehearsal)
– Role-play
– Practice is critical
• Step 5 Provide performance feedback during
practice
– Supportive
• Describe EXACTLY what was done correctly
– Corrective
• Describe EXACTLY what was incorrect
• Providing instruction on how to correct error
62
Parson et al 2012

63. Staff Training
• Consider giving feedback at the end rather
than in the middle. Can be a matter of
preference though
• Step 6 Repeat steps 4 and 5 until mastery
• On the job training
– Remember initial acquisition does not mean that
skills will be maintained or generalized
– Some “drift” has been noted in as little as 10 days
• Consider effectiveness, efficiency, and
acceptability of staff training 63
Parson et al 2012