SlideShare a Scribd company logo
1 of 95
Download to read offline
Cycle 2
Evaluation
Report
March
2015
ABA Services and Supports
for Children and Youth
in the Central West Region
ABA Cycle Two Evaluation Report
2 | P a g e
Evaluation Steering Committee
Kim Rodrigues (KPAS – Halton, Dufferin, Wellington, Waterloo)
Pam Tibbetts (BCBA – ROCK Halton)
Cathy White (Peel District School board – ret.; Autism Ontario)
Gail Jones (KPAS – Peel)
Angela Davis (Peel Behavioural Services)
Georgia Zadow (Parent, advocate)
Brett Friesen (CMHA – Waterloo Wellington Dufferin)
Debbie Dyment (Community Living North Halton)
CCBR Research Team
Rich Janzen
Kimia Ghomeshi
Kyla English
Jane Garant
Tangul Bilgehan
Centre for Community Based Research (CCBR)
73 King Street West, Suite 300
Kitchener, Ontario N2G 1A7
Phone: (519) 741-1318 Fax: (519) 741-8262
E-mail: rich@communitybasedresearch.ca
Website: www.communitybasedresearch.ca
ABA Cycle Two Evaluation Report
3 | P a g e
Contents
Executive Summary.......................................................................................................................................4
Introduction ..................................................................................................................................................8
Background ...............................................................................................................................................8
Evaluation Purpose ...................................................................................................................................9
Main Research Questions .........................................................................................................................9
Evaluation Design........................................................................................................................................10
Evaluation Approach...............................................................................................................................10
Program Theory of Change.....................................................................................................................11
Cycle Two Methods.................................................................................................................................14
Evaluation Findings .....................................................................................................................................16
Program Implementation .......................................................................................................................16
Program Outcomes.................................................................................................................................35
Future Directions ....................................................................................................................................46
Recommendations......................................................................................................................................51
Recommendations for the ABA Program................................................................................................51
Recommendations for Cycle Three Evaluation.......................................................................................53
References..................................................................................................................................................54
Appendices..................................................................................................................................................55
ABA Cycle Two Evaluation Report
4 | P a g e
EXECUTIVE SUMMARY
This report summarizes the results of the second evaluation cycle (April 2014-March 2015) for the
Applied Behaviour Analysis (ABA) program delivered to children and youth (ages 9-17) in the
Central West Region of Ontario. The lead partner for this program is Kerry’s Place Autism Services
(KPAS), in partnership with other key agencies in the region. KPAS works collaboratively with
Community Living North Halton (CLNH), Peel Behavioural Services (PBS), Reach Out Centre for Kids
(ROCK), Sunbeam Residential Development Centre (DSRC), and the Canadian Mental Health
Association Waterloo Wellington Dufferin (CMHA).
As part of the Cycle 1 evaluation, an ABA program logic model was created to explain the
continuum of service. The process starts with intake/referral, followed by service planning, ABA
service delivery, and discharge and interfacing. A child/youth and their family may go through the
process multiple times before they reach the age of 18. System-level activities are also undertaken
with KPAS partnership and external agencies.
Both quantitative and qualitative methods were used for this evaluation. The quantitative
methods included the ABA roll-up (including the Goal Attainment Scale (GAS)), the Consumer
Feedback Survey, the Central West Region (CWR) ABA Services Intake Management), Time
Utilization sheets, and the Secondary Outcomes tool. Qualitative methods included focus groups
with families, focus groups with staff, key informant interviews, and in-depth case studies.
Evaluation findings were categorized under the three main headings: program implementation,
outcomes, and future directions.
Implementation: In general, evaluation participants expressed satisfaction with the program and an
increasing number of families are interested in accessing ABA services. As a result, the waitlist
continues to grow in length. KPAS (which has centres in all the regions) provides service to the
highest number of children/youth overall, followed by PBS. The majority of families receiving or
waiting for services live in the Peel region. Families have the choice of participating in individual or
group sessions; most families choose individual sessions – often in the home setting – to meet
their goals.
Data from Cycle Two revealed five key strengths of program implementation. These included:
1. Consistency in program planning (e.g., the matching of consultants with past clients);
2. Variation in programming options (e.g., individual vs. groups sessions, at-home vs. agency
settings);
3. Opportunities to work with highly-skilled consultants;
4. The ability to focus families on manageable change; and
5. Effective partnerships among ASD agencies.
ABA Cycle Two Evaluation Report
5 | P a g e
Despite these strengths, evaluation participants also identified a number of challenges, many of which
were highlighted in Cycle 1. The six most significant challenges that emerged from Cycle 2 are as
follows:
1. Unsustainable targets, leading to staff burnout);
2. Limited program dosage, due to long wait lists and short program lengths;
3. Lack of other supports for parents during ABA and while on the waitlist;
4. Underdeveloped partnerships with schools and mental health agencies, leading to difficulties in
achieving and maintaining goals;
5. Limitations with social skills groups, with respect to matching clients and parental involvement.
Outcomes: The evaluation found that many of the outcomes anticipated by the program were in fact
being achieved. Often, families are seeing significant positive changes in their children’s behaviours and
skills. Most children either reached or exceeded their goals as indicated by both evaluation participants
and the GAS scores. Moreover, new data from Cycle Two revealed the presence of secondary outcomes
within ABA programming as well. Over 70% of children who were tracked using this tool experienced at
least one secondary outcome, such as getting along better with siblings.
Parents and caregivers themselves are also experiencing positive outcomes including heightened levels
of confidence and an increased capacity to cope with challenging situations. Nevertheless,
parents/caregivers reported some difficulty in maintaining goals over the long-term.
At the system level, there has been strong coordination between ASD agencies as well as an increase in
collaboration with external agencies (e.g., schools, mental health agencies, etc.) in some regions.
However, participants also indicated that significant challenges remain regarding working collaboratively
with schools, as well as providing coordinated services for clients with a dual-diagnosis.
Future Directions: Participants provided numerous suggestions to enhance the effectiveness of ABA
programming. For example, parents and some key informants identified a need for a reduction in the
waitlist and for more resources and support. Below are the key recommendations that emerged from
the Cycle Two report, based on discussions with the evaluation Steering Committee as well as
suggestions from evaluation participants. Recommendations related to changes in Cycle Three of the
evaluation are provided at the end of the report.
Recommendation #1: Increase program resources to sustainably meet service targets and achieve long-
term outcomes for the diversity of clients and their families. Prioritize the following program
improvements:
 Hire additional ABA consultants to reduce caseloads, shorten wait lists, and to improve the
diversity of staff.
 Provide flexibility in program length, with the option of longer program cycles for families who
wish to work on goals that would require more treatment time.
 Provide semi-regular capacity-building opportunities for staff, including workshops, conference
participation, and annual training sessions.
ABA Cycle Two Evaluation Report
6 | P a g e
 Add client-oriented supplies (e.g. toys, activities, etc.) to the ABA family resource centres.
Recommendation #2: Work with community partners to provide a range of additional resources in
multiple formats and languages (as needed) for parents/caregivers on the wait list. Prioritize the
following:
 Information and orientation sessions on the ABA program, its family-centered approach, and
goal setting process. Orientations (online or in-person) could be mandatory for new families.
 Workshops or resources for ABA clients and parents/caregivers on strategies to support youth
with ASD during their transition to high school or adulthood, or their transition from ABA
agencies serving up to eight year-olds to agencies serving 9-17 year-olds.
 An online forum for families that can be used to correspond with other families.
 Resources such as video tutorials on behavioural management of children/youth with ASD.
 Group social activities and related resources for children/youth with ASD.
Recommendation #3: Continue to adjust the first-come-first-serve policy for clients in the social skills
domain so that clients in each group are selected according to similar social capacities and needs. As
well, organize visits (in-person or by phone) with clients being considered for social skills groups to
ensure that clients are well matched.
Recommendation #4: Explore any possible options to start the service planning process while families
are on the wait list (e.g. provide examples of possible ABA goals or illustrations of how the program
works).
Recommendation #5: Explore or further implement group models of services, such as summer groups or
dyads.
Recommendation #6: Improve the goal setting process in the following ways:
 Suggest that parents select goals that address clients’ more significant behavioural challenges.
 Provide examples of possible ABA goals and program illustrations.
 Set goals for parent/caregivers to boost their involvement in the program.
 Further seek to involve other ASD-related professionals when appropriate to collaboratively
create the treatment plans.
 Continue to attempt to consult with the client’s school when the goal that would require school
involvement.
Recommendation #7: Continue having discussions with the Ministry of Child and Youth Services
regarding the suitability of the Child and Adolescent Needs and Strengths (CANS) to the ABA program.
Recommendation #8: Continue to explore options for including more natural settings in ABA sessions,
particularly for clients in the social skill and daily living domains to support them in navigating real life
situations.
ABA Cycle Two Evaluation Report
7 | P a g e
Recommendation #9: Further improve reporting mechanisms and increase parental involvement in
social skill interventions to strengthen long-term outcomes for participating children/youth.
Recommendation #10: Work with community partners to provide a treatment model for older clients
that is more age appropriate and engaging.
Recommendation #11: In the absence of a short wait list, institute bi-monthly follow-ups (via telephone
or email) following the booster sessions (when requested by parents/caregivers) to ensure maintenance
of goals and to support parents/caregivers with challenges as they arise before the next program cycle.
Recommendation #12: As part of the re-referral process, continue to identify whether families would
want to work with the same consultant in the next program cycle when possible and appropriate.
Recommendation #13: Prepare more detailed informational materials on the ABA program to distribute
to other ASD agencies, mental health and health care services, and schools. These resources should
clearly outline what the program is (and is not), and for schools should highlight how the program
complements the ASD services and supports they already offer. In some instances, organize meetings
between school representatives and ABA managers, including testimonials from principals, families, and
teachers who have had a positive experience with the program.
Recommendation #14: ASD agencies to prepare a transition plan for children/youth moving from
agencies serving clients up to eight year-olds to agencies serving 9 to 17 year-olds.
ABA Cycle Two Evaluation Report
8 | P a g e
INTRODUCTION
This report summarizes the Cycle Two evaluation findings of the Applied Behaviour Analysis (ABA)
services for children and youth (aged 9 to 17 years) in the Central West Region of Ontario (Waterloo,
Wellington, Dufferin, Halton, and Peel). The report covers the evaluation period beginning April 1, 2014
and ending March 31, 2015, and is the second of three evaluation cycles ending in MaAs lead agency for
the Central West Region, Kerry’s Place Autism Services (KPAS) contracted the Centre for Community
Based Research (CCBR) to lead the evaluation under the guidance of a cross-stakeholder evaluation
steering committee.
Background
ABA services in the Central West Region are delivered to children and youth ages 9 to 17 years with
Autism Spectrum Disorder (ASD) through a network of service agencies known as the ABA Central West
Region. This network delivers services that are focused on establishing skills and/or addressing key
issues that will improve the quality of life for children and youth with ASD. These services are oriented
around the following four domains:
 Communication
 Behaviour/emotional regulation
 Social skills
 Daily living skills
The partnering agencies also provide support to the parents/caregivers of children with ASD, aiming to
improve their quality of life, enhance their behaviour management skills, and instill a sense of
competence. To address these goals, a flexible support model is developed collaboratively with
participating children or youth and their parents/caregivers, and can include group, individual, family,
and parental capacity supports. Training is provided to parents and caregivers while their child or youth
is participating in ABA programming.
Partnering service delivery agencies for children/youth ages 9 to 17 years include Kerry’s Place Autism
Services (KPAS; lead partner), Community Living North Halton (CLNH), Peel Behavioural Services (PBS),
Reach Out Centre for Kids (ROCK), Sunbeam Residential Development Centre (DSRC), and the Canadian
Mental Health Association Waterloo Wellington Dufferin (CMHA). Another partnership exists for ABA
services delivered to children aged zero to eight years which is led by ErinoakKids (EOK) and includes
Dufferin Child and Family Services (DCAFS), KidsAbility Children’s Centre (KidsAbility), and Woodview
Children’s Centre (WCC). A joint waitlist for both partnerships is managed by KPAS.
The ABA Central West Region is committed to the development, delivery, evaluation, and continuous
improvement of its ABA-based services and supports. To this end, they requested that an evaluation of
the ABA services be conducted to enable the program to make evidence-based decisions for future
program delivery. The evaluation was to be grounded in the perspectives of children and youth, their
families, and the systems of service in their communities. The scope of the evaluation was to be focused
on the service delivery partnership for children ages 9 to 17 (i.e., the KPAS-led partnership).
ABA Cycle Two Evaluation Report
9 | P a g e
Evaluation Purpose
The evaluation was designed to understand how existing ABA programming is being implemented at
each of the Central West agencies for children and youth ages 9 to 17. In particular, the evaluation was
designed to assess what is working well and what is not working well within the context of ABA
programming. The evaluation should therefore clarify what helps and hinders ABA Central West from
creating the desired change that they wish to see in children and youth, their families, and the networks
of ASD-related services in their communities. In other words, the evaluation seeks to understand the
processes of ABA programming, and how these processes link to intended outcomes.
This evaluation was also designed to explore the impact of ABA programming in the Central West
Region. Specifically, the evaluation explores the outcomes of ABA on: 1) children and youth (aged 9 to
17); 2) families/caregivers of children and youth clients; and 3) the networks of ASD-related services.
Implementing a multi-cycle evaluation allows the program to consider outcomes over time.
Finally, the evaluation was designed to be forward looking. Data from the evaluation provides concrete
recommendations on how the Central West Region can improve their ABA services in the future for
children/youth, their families, and within the local networks of ASD-related services.
Overall, the evaluation has three inter-related objectives:
 To assess the implementation processes of ABA programming in the Central West Region
 To assess the outcomes of ABA programming at the child-, family-, and systems-levels
 To identify future directions for improving ABA services in the Central West Region
Main Research Questions
Three main research questions guided the evaluation, consistent with the three objectives mentioned
above. These questions (and corresponding sub-questions) are listed below, and form the basis of how
this report is organized.
1. How is the ABA program being implemented at agencies in the Central West Region? (process)
a) What is the underlying program theory, including the main activities and expected
outcomes?
b) What are the child and family demographics in ABA Central West Region communities?
c) What resources (e.g., human, financial, partnership) support the various aspects of program
functioning (i.e. inputs)?
d) What aspects of the program seem to be working well? Not working well?
e) What is facilitating and what is hindering effective program implementation?
f) How are the guiding principles evident in service delivery?
2. How and to what extent has ABA programming impacted participating children/youth and their
families? (outcomes)
a) What are the main accomplishments and products of the ABA program? (i.e. outputs)
ABA Cycle Two Evaluation Report
10 | P a g e
b) What outcomes has the ABA program had for participating children and youth in the
domains of communication, behaviour/emotional regulation, social skills, and daily living
skills?
c) What outcomes has the ABA program had for the families of participating children and
youth in terms of their quality of life, enhanced behaviour management skills, and sense of
competence?
d) What systems level outcomes has ABA programming achieved?
e) How have outputs and outcomes changed over time?
3. What recommendations can be made to improve the outcomes of ABA services for children, families,
and the networks of ASD-related services in their communities? (future directions)
a) What should be done to enhance the effectiveness of the ABA programs for children and
youth? For families? For the network of ASD-related services in their communities?
b) How should future ABA programming be evaluated?
EVALUATION DESIGN
Evaluation Approach
This evaluation is carried out using an approach that is consistent with the three hallmarks of
community-based research: community-situated, collaborative and action-oriented (Ochocka & Janzen,
2014).
Community-situated. Community-situated means that research is of practical relevance to community
members and is carried out in community settings, rather than being driven simply by researcher
interests. This evaluation therefore seeks to understand the program context across the Central West
Region, paying close attention to the program’s guiding principles. It also means that the evaluation
draws on information already collected by the program (i.e., secondary data), while involving ABA staff
in helping to collect primary data (e.g., helping to organize interviews and focus groups).
Collaborative. Collaborative means that community members and researchers share control of the
research agenda through reciprocal involvement in the research design, implementation and
dissemination. The ABA services evaluation therefore makes sure to involve different stakeholders
(including children/youth, parent/caregivers, various levels of staff, and community partners)
throughout the evaluation process. Concretely, most stakeholders are involved through the evaluation
Steering Committee that guides each step of the evaluation. This Committee creates momentum and
increases the likelihood that findings will be acted upon. In addition, stakeholders are also involved as
research participants.
Action-oriented. Action-oriented means that the process and results of research are useful to
community members in making positive change. This evaluation is therefore utilization-focused,
meaning it is designed to be as useful as possible. Action-oriented also means that there is ongoing
ABA Cycle Two Evaluation Report
11 | P a g e
feedback of the evaluation findings so that action can be taken following each evaluation cycle. To this
end, the evaluation report includes concrete recommendations for future action. Action-oriented also
means that the evaluation is open to improvement from one cycle to the next; recommendations to
improve Cycle Three of the evaluation are included at the end of this report.
Program Theory of Change
An understanding the ABA Central West Region’s program theory helped to guide the evaluation design.
A program theory is an explanation of how a program is expected to be an intervention for positive
change (see Chen, 2005). Such a theory is useful for evaluations as it brings clarification among
stakeholders about what the program is intended to do, and it informs the development of evaluation
data collection tools.
The program logic model is presented below. The program logic model was developed collaboratively
by reviewing project documents, holding a site visit, and through Steering Committee discussion.
Overall, the logic model emphasizes the individual and systems-level outcomes specific to ABA
programming, while acknowledging that these services are one element within the entire continuum of
care for children and youth ages 9 to 17 years with ASD.
Horizontally, the program logic model activities and outcomes are arranged according to where they
would occur within the service delivery flow. These categories include: Intake/Referral, Service Planning,
ABA Services Delivery (divided according to Child/Youth and Parent/Caregiver), Discharge and
Interfacing; and System (divided according to ‘With Partner’ and ‘With External Agencies’).
The program logic model shows the service delivery flow in a time-ordered fashion from first contact to
discharge/interfacing. Note that at the end of discharge/interfacing, a dashed arrow indicates that
program re-entry may occur. Notice also that the system-level category is not connected to the service
delivery flow with an arrow as it is assumed that the system operates externally and concurrently with
ABA programming. Waitlist resources are visually shown in the shape of a circle following intake (where
clients first access them), but in fact these resources cut across the ASD continuum of care.
Vertically, the program logic model is arranged according to Activities, Outcomes (both individual and
system), and Program Mission. Activities detail the parts of the service delivery. The set of activities for
each step in the service delivery flow unpack the work that is undertaken by the network partners.
System-level activities are also noted (both with partners and with external agencies).
Individual Outcomes are those that benefit children/youth and their parents/caregivers, while System
Outcomes are those that benefit ABA Central West partners and external agencies/institutions. Notice
that the Service Planning and the Discharge & Interfacing categories lead to the primary ABA program
outcomes at the individual and family level. It should also be noted that the Parent/Caregiver individual
outcomes fold into the Child/Youth outcomes, as ABA programming is designed to improve the capacity
of parents/caregivers to support improvements to their child’s behaviour. Similarly, system-level
outcomes at the individual level (which improve participants’ ability to navigate the ASD continuum of
care) fold into Parent/Caregiver outcomes. Finally, System Outcomes are shown to cradle the entire
service delivery flow, as their impact is reflected throughout the ABA program and continuum of care.
ABA Cycle Two Evaluation Report
12 | P a g e
The Program Mission is noted at the bottom of the logic model. This mission captures the guiding
principles of service delivery outlined by the Ministry of Children and Youth Services. This Ministry
document identifies the purpose of ABA services as they pertain to improving the services and supports
for children with ASD.
The ABA Central West Region theory of change has been evolving and will likely continue to evolve over
time. A deepening understanding of program theory is expected to occur as the program matures and
responds to changing circumstances and evaluation learnings. This means that the evaluation will need
to be open to exploring and capturing the unexpected, as well as the anticipated outcomes and
processes. It also means that the evaluation itself can be an intervention in further clarifying and honing
the program theory (Janzen, Seskar-Hencic, Dildar & McFadden, 2012).
ABA Cycle Two Evaluation Report
13 | P a g e
2014 ABA Program Logic Model
ABA Cycle Two Evaluation Report
14 | P a g e
Cycle Two Methods
This section outlines the methods of data gathering for Cycle Two of the ABA services evaluation. The
number of methods and the volume of data collected for Cycle Two was an expansion upon Cycle One
which was a shorter evaluation cycle.
The evaluation methods were determined through a systemic process of building an evaluation
framework. The evaluation framework process clarified the main research questions, program theory (i.e.,
program logic model), and corresponding measurement matrix (see Appendix 2). The measurement matrix
links expected program outcomes (impact measurement) and processes (performance measurement) with
indicators and corresponding data collection tools. The matrix also lists questions about future directions
(strategy measurement) linking them with corresponding data collection tools.
Based on this measurement matrix, data collection tools were identified or developed. The individual
tools include a mix of existing measures used by the ABA Central West Region and those developed by the
evaluation team. Some of the quantitative tools introduced in Cycle Two were developed to address gaps
in secondary data according to Cycle One findings. The qualitative tools that were introduced in Cycle One
were also revised and expanded to further explore major themes that emerged in the first cycle, and to
examine the relationship between the clients’ domain area or diagnosis and their experience in the
program. Combined, the tools include a mix of quantitative and qualitative measures to provide breadth
and depth of insight. Appendix 3 includes primary data collection tools with corresponding consent forms.
Cycle Two methods are listed below, organized into quantitative methods (representing secondary data
already gathered by the program) and qualitative methods (representing primary data that was collected
specifically for the evaluation).
Quantitative Methods (Secondary Data)
ABA Roll-up. The ABA Roll-up is an Excel spreadsheet that is regularly updated and maintained by clinical
supervisors across the Central West Region. Each region independently completes their own spreadsheet,
which includes quantitative information about program participants and program delivery. More
specifically, the spreadsheet tracks program outputs by client (e.g., number of hours of service delivery,
type of service, and region). Included within the ABA Roll-up are client results from the Goal Attainment
Scale (GAS).
GAS quantifies the achievement of goals set by parents and ABA consultants based on a 5-point rating
scale from -2 to +2. If a chosen goal (for example, reduction in aggression) is recorded as -2 in the Roll-up,
then the outcome level was much less than expected (e.g., there was a regression in the goal area). If the
goal is recorded at +2, then the outcome was much greater than expected (e.g., the child greatly improved
and the aggression was significantly reduced). A score of zero means that the expected level of goal
attainment was reached but not surpassed (i.e., the child met but did not exceed their goal).
This report includes data collected in the ABA Roll-up for most of Cycle Two, from April 1, 2014 until
December 31, 2015. In total, outcomes from 658 clients aged 9 to 17 years (across all agencies) were
tracked in the Roll-up during this time, and were included in the analysis.
Consumer Feedback Survey. The Consumer Feedback Survey (CFS) is self-administered by
parents/caregivers of children and youth participating in ABA programming. Upon completing a round of
ABA Cycle Two Evaluation Report
15 | P a g e
service, parents/caregivers are given a hardcopy of the CFS, and are asked to fill it out and return it in an
envelope (in-person or by mail) to the program manager at their agency. Central intake staff at the various
ASD agencies will then insert the individual CFS data into the online version (i.e., on Survey Monkey),
where all CFS data from agencies in the ABA Central West Region is stored.
The CFS quantitatively evaluates program outcomes in the domains of quality of life, behaviour
management skills, and feelings of competence at the family level. The survey also contributes
quantitative insight into program processes (i.e., what is working well and not working well), and provides
suggestions for future directions via qualitative components (i.e., boxes for parents/caregivers to provide
suggestions or comments). The CFS was revised in the fall of 2014, partially to include questions on the
generalization and maintenance of ABA goals.
In this evaluation cycle, survey responses of 423 parents/caregivers were analyzed, approximately 69%
from Peel Region, 20% from Halton Region, and 11% from the Waterloo, Wellington, and Dufferin Regions.
Central West Region (CWR) Applied Behavioural Analysis (ABA) Services- Intake Management. The CWR
ABA Services Intake Management database (referred to as ‘intake database’ here) provides demographic
information on all children/youth and their families that are currently enrolled in ABA services within the
Central West Region. The information in the intake database provides a demographic snapshot of clients
undergoing the program, in addition to total number of clients ever served and interventions delivered.
The database provides details about how many clients are on the central waitlist vs. in service, and
identifies the domain area(s) that families are working on during their sessions. Individual results of the
Child and Adolescent Needs and Strengths (CANS) tool are also available within the intake database.
Aggregate data from the intake database (for the period of April 1, 2014 to December 31, 2015) was
analyzed during Cycle Two. Due to privacy concerns, ABA staff de-identified and collated all data prior to
sending it to the external evaluating team. Data from the CANS portion of the intake was analyzed in Cycle
One and will be re-visited and re-analyzed in Cycle Three.
Time Utilization Sheets. The time utilization sheets were a tool developed out of recommendations from
Cycle One. These time sheets help ABA consultants to track the number of hours per day that they spend
on various tasks (e.g., direct client contact, client documentation, etc.). According to ABA consultants and
other program staff, a number of these tasks were not previously tracked.
In this report, data was analyzed over a three month period (October 2014 to December 2014), as these
were the only months consistently tracked across all agencies. In total, data from 25 consultants was
analyzed for this report: one from CLNH, seven from PBS, two from CMHA, two from ROCK, five from
DSRC, and 14 from KPAS.
Secondary Outcome Tool. The Secondary Outcome tool was developed out of recommendations from
Cycle One, and was implemented starting in November 2014. This tool helps to determine: 1) the
percentage of clients who exhibit positive outcomes other than the achievement of their main goal; 2) the
extent to which these secondary outcomes are exhibited; and 3) the degree to which these outcomes
occur in different environments (i.e., at home, school, or in public). Consistent with the ABA program
goals, secondary outcomes are tracked by the four main domain areas.
For the purposes of this evaluation report, 50 secondary outcome sheets were collected and analyzed
from various agencies in the Central West Region.
ABA Cycle Two Evaluation Report
16 | P a g e
Qualitative Methods (Primary Data)
Focus groups with parents/caregivers. A total of nine focus groups were conducted in Cycle Two with
parents/caregivers (n=32) of ABA clients. Five were held via teleconference, and four were held in-person.
Participants were selected and recruited in cooperation with program managers on the evaluation Steering
Committee, and according to purposive sampling criteria (i.e., diversity in geographic representation
across the Central West Region, length of involvement in ABA program, gender, ethnic diversity, range of
diagnosis, range of child/youth ages, and range of models of services). All main research questions were
addressed in these focus groups.
Focus groups with staff. Three focus groups were held with ABA staff during Cycle Two, including one with
clinical supervisors (n=5) and two with ABA consultants (n=19). Participants were selected and recruited in
cooperation with program managers and according to purposive sampling criteria developed by the
evaluation Steering Committee (i.e., representation across Central West Region, diversity in clinical and
frontline staff, and diversity in length of involvement with program). All three main research questions
were addressed to gain front-line insight on program processes and outcomes for child/youth and
parent/caregivers, as well as recommendations for improvement (see Appendix 3).
Key informant interviews. Twelve key informant interviews were held in Cycle Two via telephone, to gain
insight into the three main research questions related to systems-level processes, outcomes, and
recommendations for improvements. Participants were selected and recruited in cooperation with the
Steering Committee and according to purposive sampling criteria. Participants were located across the
Central West Region and included senior staff at ABA agencies, a Ministry of Child and Youth Services
representative, school representatives, and senior staff at ABA support agencies.
Case studies. Case studies were introduced as a new qualitative tool in Cycle Two to give further insight
into some of the common strengths and challenges of the program, while showcasing the diversity in
clients, families, and program experiences. Three clients were selected according to purposive sampling
criteria (i.e., diversity in geographic representation across the Central West Region, length of involvement
in the ABA program, gender and ethnic diversity, range of child/youth ages and diagnoses, and range of
models of service). For each case study, the child’s parent or caregiver was interviewed, as well as their
ABA consultant. When possible, the child/youth was also interviewed, or a family friend if the child was
uninterested or unable. In total, eight interviews were conducted across the three case studies.
EVALUATION FINDINGS
The findings of the evaluation are organized according to the three main research questions related to
program implementation, program outcomes, and future directions. Data was analyzed across all research
methods with main themes highlighted below.
Program Implementation
Overall, evaluation participants expressed an appreciation for the ABA program, and were mostly satisfied
with the service they delivered or received. While there remain significant challenges with program
implementation, data revealed that families are “wanting the service, needing the service, [and] liking the
ABA Cycle Two Evaluation Report
17 | P a g e
service” (Key Informant). This section begins by describing the program overall (i.e., number of clients
served, percentage of children in each domain), followed by a list of the top strengths and challenges
revealed in Cycle 2. Similarities and differences to Cycle 1 are highlighted throughout.
Description of ABA Participants
The ABA program is available to all children/youth under the age of 18 years who have a confirmed ASD
diagnosis (Autism/Autism Disorder, Pervasive Developmental Disorder, or Asperger’s Disorder).
Confirmation of diagnosis must come from a certified and qualified professional (i.e., a family physician,
pediatrician, psychiatrist, psychologist, or psychological associate) (Autism Ontario, 2015).
Chart 1 shows a snapshot of the total number of clients (aged 9 to 17 years) in service on March 6, 2015,
within all areas of the Central West Region. The majority of children who were in service were between
the ages of 11 to 14 years. In all areas, the 15-17 year age bracket had the fewest number of clients in
service. Chart 1 also demonstrates that Peel served the greatest number of clients during this time (135),
accounting for 55% of all children/youth who were in service across the Region. This is due to the fact that
Peel has a higher population than the other regions, and is thus allocated more of the funding and
resources.
Chart 1: Number of children/youth in service by age group and region (N=356)
Source: CWR ABA Service- Intake Management (Snapshot of those in service on March 6, 2015)
Chart 2 shows the total number of eligible children/youth in 2014 (ages 9-17 years) by service status and
by service provider cumulative for April 1, 2014 to December 31, 2014. In this report, ‘eligible’ is defined as
all children who: (a) are on the waitlist; (b) are in service; or (c) completed a block of service in the 2014
fiscal year. The total number of children served is a cumulative figure for the period of April 1 2014 to
December 31, 2014. “Other Eligible Children” refers to any clients who are on the waitlist, or have
completed ABA service. The majority of eligible children were clients in KPAS Peel (26%) and PBS (24%).
40
12
5 6 7
66
17
28
15
2
29
7 7
3
0
0
10
20
30
40
50
60
70
Peel Halton Waterloo Wellington Dufferin
TotalNumerofChildren/Youthin
Service
Ages 9-10
Ages 11-14
Ages 15-17
ABA Cycle Two Evaluation Report
18 | P a g e
Chart 2: Total number of eligible clients (aged 9-17 years) by service status (served vs. other eligible
children) and service provider (N=2264)
Source: CWR ABA Service- Intake Management (cumulative for April 1, 2014 to December 31, 2014)
Discussions with evaluation participants revealed that families represent a range of income levels. Given
that ABA services are offered at no cost, low-income families can participate in the program. Discussions
also revealed that ABA families are ethnically diverse. This is particularly true in the large urban areas such
as Peel Region. Some families have recently immigrated and are still in the process of learning English.
Some ABA consultants can speak multiple languages. Interpreters are also available, if needed, but at a
cost to the program. As was noted in the Cycle One report, some key documents are not always translated
(e.g. consent forms and individual plans).
Description of Program
As explained in the Cycle 1 report, children or youth can work on goals related to one of four domains:
communication, social skills, daily living skills, and behaviour/emotional regulation skills. Chart 3 depicts
the percentage of clients within each domain, from April 1, 2014 to January 21, 2015. Children/youth can
only work within one domain per session.
Chart 3 reveals that the majority of clients focused on either social skills (40%) or behavioural/emotional
regulation skills (31%). Clearly, the majority of families feel that social skills and behavioural/emotional
regulation are the most important domains to work on during their ABA program.
ABA Cycle Two Evaluation Report
19 | P a g e
Communication
12%
Social Skills
40%
Daily Living
Skills
17%
Behavioural /
Emotional
Regulation
31%
79
125
32 32
48 48
21 30 26
81
48
19
4 6 3 7
35
10
0
20
40
60
80
100
120
140
TotalNumberofChildren/Youth
Service Providers
Individual
Group
Chart 3: Skill domains in ABA services for children/youth aged 9-17 years (N=256)
Source: CWR ABA Services- Intake Management (CANS April 1, 2014-January 21, 2015)
ABA is offered as an individual, one-on-one program or as a group program with other children/youth of a
similar age, level of functioning, and specified goal area. Chart 4 reveals the total number of clients per
ASD agency who participated in each type of service during Cycle Two. Clearly, most agencies served a
greater number of clients through one-on-one sessions, with the exception of PBS and DSRC who served
slightly more clients through their group sessions.
Chart 4: Form of service delivery by service provider for children/youth aged 9-17 (N=662)
Source: ABA Roll-up (April 1, 2014 – December 31, 2014)
Chart 5 demonstrates the total number of hours spent on client intervention (i.e., directly with the
child/youth) versus parent training from April 1, 2014 until December 31, 2014. Definitions of client
intervention hours and parent training hours are provided in Appendix 4. All agencies spent more time
working with clients as opposed to training parents or caregivers. The average amount of time that
consultants spent with each client was between 9 and 18 hours per round. The average amount of time
that consultants spent training parents or caregivers was between four and 15 hours per round. In
ABA Cycle Two Evaluation Report
20 | P a g e
comparison to all other service providers, KPAS Peel spent the most amount of time, on average, with
clients (approximately 18 hours/round), while CLNH spent the most amount of time, on average, training
parents or caregivers (approximately 15 hours/round). CLNH spent an almost equivalent amount of time
on both, as opposed to agencies such as KPAS Peel, DSRC and CMHA who spent much more time on client
intervention. It must also be noted that some clients, especially those with more complex cases (e.g., a
dual diagnosis) may be in service for up to six months, if a supervisor’s clinical judgment dictates that the
client cannot be released. This, according to many key informants, can contribute to the challenges
outlined later in this report (e.g., growing waitlist, staff burnout, etc.).
Chart 5: Amount of time spent on client intervention and parent/caregiver training by service provider for
children/youth aged 9-17 years (N=621)
Source: ABA Roll-Up (April 1, 2014 – December 31, 2014)
Chart 6 gives a more fulsome picture of how ABA consultants spend their time, not only with clients and
their parents/caregivers, but on paperwork, training and other duties and responsibilities. The tool to
collect this data (‘Time Utilization Sheet’) was implemented for the first time in Cycle Two; it is filled out
monthly by ABA consultants. Chart 6 demonstrates the average time spent on each task type, from
October 2014 to December 2014. The chart clearly shows that consultants spend the greatest percentage
of their time on documentation (~26%) followed by direct client contact (~23%) and administration (~12%).
Definitions of each type of task are provided in Appendix 5.
15.4
17.7
14.5
9.4
15.0
16.9
15.9
13.9
12.7
12.9
4.9
8.7
5.0
11.3
12.9
15.0
4.1
4.8
0.0
2.0
4.0
6.0
8.0
10.0
12.0
14.0
16.0
18.0
20.0
Averageamountoftimespentoninterventionand
training(perclient)
Service Providers
Client Intervention Hours
Parent Training Hours
ABA Cycle Two Evaluation Report
21 | P a g e
Chart 6: Total Time Spent by Task Type for October 2014 – December 2014
Source: Time Utilization Sheets, October 2014 – December 2014
Strengths of Program Implementation
Evaluation participants highlighted several strengths in regards to ABA program implementation, many of
which were similar to those revealed in Cycle One. Overall, parents and caregivers were grateful for the
program and appreciated the various components involved in service planning and delivery.
“This ABA program is like gold. It is an untapped hidden resource that is extremely effective.”
(Parent)
“The tracking was good. The feedback was good. The tools were excellent. Knowledge of the child
and an understanding of the child’s needs was excellent. It worked.” (Parent)
Chart 7 further depicts parents’/caregivers’ satisfaction through their response to the statement “Overall, I
found the program helpful.” 93% of all responders agreed or strongly agreed.
Direct Client
Contact
23.0%
Indirect Client
Contact
5.6%
Client
Documentation
25.9%
Administration
12.3%
Case Review
4.8%
Supervision
1.9%
Committee
Meeting
& Relations
1.3%
Staff Education
2.9%
Student Education
0.3%
Community
Education
0.5%
Other
3.2%
Travel
8.9%
Sick
1.9%
Vacation
7.6%
ABA Cycle Two Evaluation Report
22 | P a g e
Chart 7: Overall, I found the program helpful (N=414)
Source: Consumer Feedback Survey (April 1, 2014 to January 21, 2015)
Data from Cycle Two revealed five key strengths of program implementation. These included: (1) the
consistency in program planning (e.g., the matching of consultants with past clients); (2) the choice of
various programming options (e.g., individual vs. groups sessions, at-home vs. agency settings); (3) the
opportunity to work with highly-skilled consultants; (4) the ability to focus families on manageable change;
and (5) the partnerships among ASD agencies. These strengths are further explored below.
Consistency in program planning
Several parents/caregivers and frontline consultants appreciated the program’s consistency in terms of
client-consultant matching as well as grouping for children in social skills groups. Such consistencies in
service led to greater efficiencies in programming and higher rates of success. For example, when clients
are matched with the same consultant as in previous sessions, service delivery is strengthened due to
existing relationships and trust. In addition, the consultant may be better able to tailor the program to
meet the needs of the client, if they are already aware of their strengths and needs from previous sessions.
Consultants can also help their client to build upon skills that they had worked on previously.
“Having the same person be with your child on an ongoing, more fluid basis… makes all the
difference.” (Parent)
For similar reasons, parents and caregivers appreciated when their children were matched with the same
children from previous social skills groups. Bringing the same children together (when the children are well
matched) can improve the quality of their service and help the children to develop stronger relationships
with each other.
According to several evaluation participants, having a consistent program also means that parents and
children/youth can expect a similar structure each time they come off the waitlist. Children are more
aware of the expectations of the program, and understand what will happen during the sessions. Children
can also build on skills from previous sessions, and continue learning from where they left off.
Strongly Agree
70%
Agree
23%
Neutral
6%
Disagree
1%
Strongly Disagree
0%
ABA Cycle Two Evaluation Report
23 | P a g e
“We see more compliance [in subsequent sessions]… The kids are understanding what the
expectations coming back in are.” (Key Informant)
“I also feel that the families who are coming back in for 2nd
and 3rd
and 4th
round of service are
taking the option to build on the skills that have already been learned… We have some clients that
come through and only work on communication, and that’s because they’re building on what they
have previously learned.” (Key Informant)
Choice in programming options
According to many evaluation participants, another strength of program implementation is the choice in
programming options for families. Parents/caregivers appreciated the option of one-on-one or group
sessions, as well as the location of service (in the home, ABA agency, or community). For many children,
depending on their goal, individual sessions work best; they can work directly with a consultant to focus on
a goal specific to their needs. Some of these sessions occur in the family home, which many parents and
caregivers considered an effective approach for working towards change. Skills can often be more
effectively taught, learned and applied in the natural home setting, where many behaviours occur. In
addition, home-based sessions are sometimes more convenient for parents who are balancing multiple
responsibilities and may have other children to care for. Some clients also prefer to keep their diagnosis to
themselves, especially those in their teenage years, and thus prefer at-home sessions where they won’t be
seen with their consultant.
The option of having ABA sessions in an agency setting is also appreciated by parents and caregivers.
Children are sometimes less distracted in a more formal setting, and can work in a room where they won’t
be distracted by the goings-on in a home. Group sessions also happen most often in the agency setting; for
some clients and some goals, group sessions are the most effective as they provide ample opportunity for
children to interact with others and to develop social skills in a safe and supportive environment.
Evaluation participants appreciated the large and growing number of agency settings; having a greater
choice in where the program can occur makes it more accessible for families in the region.
“It’s really good that families are given a choice between in-home and group supports.” (Key
Informant)
“Agencies continue to open up centres to make the services more accessible…in an effort to make
the services and spaces more accessible to children with ASD.” (Key Informant)
Another option for ABA programming emerged last summer, as several agencies began offering ABA
summer groups that typically lasted one week (approximately 30 hours). Families who came off the waitlist
during these months could choose to attend a summer group, rather than have the standard eight-week
session. These groups were often situated in or involved field trips to natural environments (malls, bowling
alleys, etc.). In general, parents/caregivers found that their children benefitted from having sessions in
natural environments, whether in the context of the summer group or during regular social skills groups.
ABA consultants generally agreed that despite the challenges (e.g., logistics, finance, safety), using natural
environments contributed to the generalization and maintenance of goals.
ABA Cycle Two Evaluation Report
24 | P a g e
0
50
100
150
200
250
300
350
Strongly
Agree
Agree
Neutral
Disagree
Strongly
Disagree
325
76
12
0 1
NumberofRespondents
Skilled consultants
As was found in Cycle 1, the majority of key informants, clinical supervisors, and parents/caregivers in this
evaluation expressed gratitude for their highly skilled and extremely dedicated consultants. The
consultants showed creativity in how they tailored their tools and their approach to meet the needs of
individual clients. Consultants were a valued resource for families who felt they lacked the knowledge and
tools to help their children/youth on their own. One family, for example, mentioned how they only came
to understand the root of their child’s behaviour with the help of their consultant. A number of families
felt that their consultants went above and beyond to support them, and provided guidance on non-ABA
related matters as well (e.g., referrals to other services).
“The caliber of therapists is outstanding.” (Parent)
“I think our consultants… they are very skilled in the services they deliver, and I don’t think they shy
away from difficult situations.” (Key Informant)
Chart 8 illustrates one aspect of parents’ and caregivers’ satisfaction with their ABA consultant: how well
they understood the information presented to them. The chart shows that almost all parents and
caregivers believed that the information presented by the consultant was done so in a clear, easy-to-
understand way.
Chart 8: The information presented by the therapist was presented in a clear, easy to understand manner
(N=417)
Source: Consumer Feedback Survey (April 1, 2014 to January 21, 2015)
As with Cycle 1, the ABA consultants reported a strong commitment to the program both professionally
and personally, and desired to see it succeed. In general, they appreciate opportunities to advance their
skills (training workshops, conferences, etc.), although they have limited time to pursue capacity-building
opportunities.
ABA Cycle Two Evaluation Report
25 | P a g e
Focusing families on manageable change
Another strength of the program is that it helps to focus families on manageable change, despite the often
overwhelming challenges they face in caring for their children/youth with ASD. To begin, most key
informants and program staff as well as a few parents/caregivers discussed the benefits of focusing on one
goal during each round of ABA. As they explained, concentrating on one simple goal in the allotted two to
six month session can increase the probability of success for clients and parents/caregivers. This program
strength was seen to be a key factor in leading to many of the client and parent/caregiver outcomes, as
outlined in the outcomes section below.
“It’s one thing at a time… Just having that one goal takes away that overwhelming piece for
parents and it… it helps them kind of work on one thing.” (ABA Consultant)
“The beauty of this program is that it gets down to the small goals that are really important to
identify and work on and master.” (Key Informant)
“When [families] focus on one specific thing, it does build resiliency.” (Key Informant)
Despite the benefits of focusing on one goal, many parents/caregivers sometimes felt limited by this
approach. As they explained, their children face numerous interconnected challenges, many of which
cannot be resolved by focusing on one small goal. A few parents talked about the need for a more well-
rounded program which considers small goals while focusing on a bigger picture of change. Choosing a
specific goal was described as difficult by many parents/caregivers, especially those who are new to the
program and had not narrowed down their child’s needs before. In these cases, many parents felt
overwhelmed by the process and said they lacked direction in helping to choose their child’s goal.
Yet overall, the focus on manageable change (that is, doable given the program timeframe) seemed to be
working. Another facilitating factor was how consultants worked with parents and caregivers to teach
them skills they will need to maintain or generalize the client’s goal. By learning skills relevant to their
child’s behaviour, they were better able to work towards manageable change. As demonstrated in Chart 9,
the majority of parents/caregivers found that the training in skills/behavioural techniques helped them
with their child’s goals (87% agreed or strongly agreed). Only a small number of parents/caregivers did not
find the training useful (2%).
ABA Cycle Two Evaluation Report
26 | P a g e
Chart 9: The training in skills/behaviour techniques helped me with my goals (N=416)
Source: Consumer Feedback Survey (April 1, 2014 to January 21, 2015)
Furthermore, as demonstrated in Chart 10, 87% of parents/caregivers found that they were able to carry
out the strategies/techniques of the behaviour support plan (54% strongly agreed and 33% agreed).
Chart 10: I was able to carry out the procedures/strategies of the plan (N=414)
Source: Consumer Feedback Survey (April 1, 2014 to January 21, 2015)
Partnerships among ASD agencies
Finally, partnerships among ASD agencies emerged as a key strength in Cycle 2. According to ABA
consultants, clinical supervisors and several key informants, the agencies involved in the delivery of ASD
services work well together. Each Central West region formed committees of ABA and/or ASD-related
agencies and institutions that meet semi-regularly to share insights and best practices, to address service
gaps for ASD children/youth, and to strengthen regional coordination. In a few regions, the ABA program
Strongly Agree
57%
Agree
30%
Neutral
11%
Disagree
2%
Strongly Agree
54%
Agree
33%
Neutral
12% Disagree
1%
ABA Cycle Two Evaluation Report
27 | P a g e
was said to foster stronger relationships between agencies due to their increased interactions. Agencies
had a clearer sense of each other’s services, thereby making accurate referrals to each other’s programs
with greater regularity. To a lesser extent, there was also some evidence of effective coordination with
mental health agencies, and some examples of strong partnerships with schools and school boards.
“We can overall serve people better because we know more about these agencies.” (Key Informant)
“Working as a community is so important… We would never be able to deliver this program as one
agency.” (Key Informant)
Clinical supervisors from various agencies also meet on a regular basis, as do managers; these meetings
were considered a valuable space for knowledge exchange. Given that the program is implemented
differently across the Central West Region, clinical supervisors appreciated the opportunity to learn from
best practices employed in other agencies, and to brainstorm solutions to unique program challenges in
their jurisdiction.
“I found the clinical advisory committee meetings really wonderful… I feel like I still have a team. I
think we work nicely together.” (Clinical Supervisor)
Challenges of Program Implementation
Despite the large number of strengths associated with ABA program implementation, evaluation
participants also identified a number of challenges, many of which were highlighted in Cycle 1. The six
most significant challenges that emerged from Cycle 2 are as follows: (1) unsustainable targets (leading to
staff burnout); (2) limited program dosage (due to long wait lists and short program lengths); (3) lack of
other supports for parents (during ABA and while on the waitlist); (4) underdeveloped partnerships with
schools and mental health agencies (leading to difficulties in achieving and maintaining goals); and (5)
limitations with social skills groups (in terms of matching clients and parental involvement).
Unsustainable targets
The number one challenge with program implementation is undoubtedly the high number of children (i.e.,
targets) that consultants are expected to support each ABA cycle. As various consultants explained, the
amount of work required to provide high quality service to their assigned number of clients is
unsustainable and leads to burnout among staff. Due to their heavy workload, consultants are also unable
to provide flexibility in service that might benefit some clients. For example, even if children/youth would
benefit from having additional sessions with their consultant, consultants cannot often provide this
support due to time constraints. Similarly, if ABA sessions are missed due to illness or vacation, ABA
consultants are unable to make these up. This inflexibility has led to frustration among several evaluation
participants. For clients who need more time in service due to complex needs and/or a complex diagnosis,
consultants will extend the program. However, this adds additional pressure to their workload as there is
little flexibility in service targets to account for these circumstances. In other words, consultants are still
expected to reach their targets, even if some clients present particularly challenging cases and take more
time, energy and resources.
ABA Cycle Two Evaluation Report
28 | P a g e
“I have great staff who are dedicated to this program and who are very determined to provide top
quality clinical service, but they are killing themselves to do it.” (Key Informant)
“In this region, we are implementing the most clinically sound program we can. However we are
stretching our staff extremely thin in order to do so.” (Clinical Supervisor)
Part of the issue regarding consultants’ schedules is the amount of time spent on documentation. As
shown in Chart 6, ABA consultants spend more time on documentation than they do on direct client
contact. Moreover, some of this documentation is considered not very useful, by both ABA consultants
and parents/caregivers. The CANS tool, for example, was described by consultants as overly detailed (i.e.,
takes a long time to complete), yet not very helpful as the ratings are overly subjective and do not
contribute to ABA service delivery.
“The CANS has got to go… It’s not helpful for our type of service delivery… It’s very subjective and
not relevant at all, and it’s extra work; a formality. It just gets filled out and stuck in their folder.”
(ABA Consultant)
Another component of the program that takes a significant amount of time is travel. Time spent traveling
to meet clients was substantial, at an average of 8.9% of their total working hours. These and other
aspects are often not reported to the Ministry, due to standardized reporting mechanisms. According to
ABA consultants, this partial reporting gives an erroneous perception of the work required to deliver ABA
services, and could perpetuate unrealistic ministry service targets.
In sum, ABA consultants reported being overworked and seemed doubtful that they could manage their
caseloads long-term. Seeing as the program’s success has been found to partially hinge upon consistency
in consultants, there was widespread concern among evaluation participants that the ambitiousness of
ministry targets would jeopardize the quality of service delivery for ABA clients.
Limited program dosage
As found in Cycle One, another significant challenge with program implementation is the length of the
waitlist, both at initial intake and between ABA sessions. A growing number of families are participating in
the ABA program, which is a positive indication of community demand. However, because program
resources have not increased to meet the growing demand, many families are waiting up to a year and a
half for service. Chart 11 shows the number of clients on the waitlist vs. in service by agency. Clearly, there
are significantly more children on the waitlist in all agencies across the Central West Region.
ABA Cycle Two Evaluation Report
29 | P a g e
Chart 11: Number of clients (aged 9-17 years) on waitlist and in service by service provider on December
31, 2014 (N=1612)
Source: CWR ABA Services- Intake Management (Q4 April 1, 2014-January 21, 2015)
The waitlist was considered discouraging by various evaluation participants, especially because the
program itself is often only two to three months, and very few resources are available to families when on
the waitlist.
“The wait times…are discouraging. We need treatment for our kids so badly.” (Parent)
“The wait times are our challenge, every day.” (Key Informant)
With such long wait times, parents/caregivers reported having high expectations for the program, yet felt
that only limited gains were possible within such a short period of time. Since the first few weeks of the
program are dedicated to service planning and relationship building between the child and their
consultant, actual treatment time was seen as insufficient to have any significant impact on families or to
maintain and generalize goals. Parents also felt limited in the types of goals they could choose, as they
were not able to choose goals that take more time.
“I feel like we don’t spend enough time in getting them to generalize the skills across
environments.” (Key Informant)
Parents and caregivers of high-functioning children were especially frustrated with the short length of the
ABA program, as they are ineligible for most other ASD services and rely solely on ABA for support.
Similarly, parents and caregivers of children aging out of the system expect more from the program while
158
94 87
65
315
144
113 107
373
15 14 17 9
66
5 7 4
19
0
50
100
150
200
250
300
350
400
TotalNumberofChildren/Youth
Service Provider
On Waitlist
In Service
ABA Cycle Two Evaluation Report
30 | P a g e
the service is still available to them. Longer gaps between program cycles can also impede upon program
outcomes for re-referred clients, who struggle to maintain ABA goals over the long term without
professional support.
“The only problem is, my son doesn’t qualify for anything else. He never got IBI or anything, so this
is all he gets… So you get the eight weeks, and then wait almost a year until it happens again. And
he needs something on an ongoing basis, and there is nothing for him.” (Parent)
One type of support available to families after completing an ABA cycle is booster sessions. About a month
after completing service, consultants will attempt to contact their families to ask about goal maintenance.
If parents/caregivers feel that their child needs a booster session, the consultant will attempt to arrange it.
While some parents/caregivers reported receiving such a service, many others reported never being
contacted after the completion of a cycle. As they explained, in-person booster sessions rarely occurred,
and were in many cases ineffective as parents required more regular follow-up and support to maintain
goals. They generally felt positive about the idea of booster sessions, but said that in reality, they are few
and far between.
From the perspective of ABA consultants, many reported difficulties with the follow-up process, saying
that families are difficult to reach or had limited availability for in-person booster sessions. It was also
stressful for consultants to schedule these sessions whilst starting programs with new clients
“Lots of times it’s a struggle to get that [booster sessions] worked in, for a variety of reasons. I
mean the consultants have already moved on to their next group of children and youth that they’re
working with.” (Key Informant)
For consultants that did conduct follow-up over the phone, most had a positive response from families and
reported that goals were successfully maintained. Even still, consultants questioned whether families
were being honest or just appeasing them with positive feedback.
Limitations in social skills groups
Similar to Cycle 1, a final key challenge of program implementation was the difficulty around developing
relevant and meaningful social skills groups. While ABA staff considered the first-come-first-serve nature of
the wait list to be fair in many respects, it also creates challenges in forming social skill groups with well-
matched clients (i.e., similar diagnosis, gender, behavioural challenges, and program goals). In some areas
of the Central West Region, the policy was recently changed so that clients could be selected up to six
months down the wait list for social skills groups. With the wave of new clients entering the program,
however, some consultants reported that they were not familiar enough with clients to match them
effectively.
ABA Cycle Two Evaluation Report
31 | P a g e
Case-in-Point: Rishi’s Case Study
Unfortunately, Aarav has little understanding of what
occurred during his son Rishi’s first ABA session, as the
program did not facilitate parent observation. Rather, he
waited in the agency’s lobby until each session was complete.
While parents are now encouraged to observe some of their
children’s groups, Aarav still feels as though “when you [the
parent] are left out, all of the added value that you bring to
the session is gone”. In the end, Aarav explained:
“I’m not sure how successful that was... I was literally shut
off, so I didn’t have much idea of what was going on.... Did
I gain anything? Did [Rishi] gain anything? No idea,
whatsoever”.
Even though Aarav was provided the reports at the end of
ABA, he still found them difficult to read. As he said, “I
couldn’t tell if there was any perceptible gain or even if he
achieved the goal”.
 To read Rishi’s full case study, see Appendix 1.
“This [creating social skills
groups] has been challenging
lately because newer families are
coming onto the list. So you
don’t know that client and you
don’t know that family and so
it’s very hard to pick them out
and pair them appropriately.”
(ABA Consultant)
Forming well matched girls-only groups
was reported as especially difficult, as
there are significantly fewer female
clients to select from. As well, due to the
large geographic area in places like
Halton, it was difficult to form groups
with children living in the same area.
According to several evaluation
participants, social skills groups provided
few opportunities for parents to observe
and participate. This limited parents’
understanding of the program’s process
and outcomes, as well as their ability to
help maintain and generalize skills.
“I find because we are heavy in our group supports, the parents’ participation is limited, which
impacts generalization and even just their capacity to understand the program and see that, you
know, my child is actually making gains based on the goal that was set.” (Key Informant)
Finally, a small number of families were dissatisfied with the lack of confidentiality in how feedback
regarding their child/youth was provided at agencies during social group sessions, as other families and
staff were often present in the room and could overhear the conversation.
Lack of Other Supports
Another challenge reported by most parents/caregivers as well as some key informants was the lack of
supports for families outside of the short program sessions. While a select few received resources while
on the ABA waitlist, the majority of parents and caregivers felt as though they were “off the radar”
(parent) during this time. Throughout the focus groups, they consistently expressed frustration regarding
their feelings of isolation and lack of support. Program managers and other key informants agreed that
there’s not much available for families while they wait a year or more for ABA.
Parent training and workshops have become increasingly available since the start of this evaluation, yet
are not offered in all areas of the Region, nor are they actively attended. Many parents and caregivers
cannot take the time away from their families to attend a workshop in person.
ABA Cycle Two Evaluation Report
32 | P a g e
Case-in-Point: Emily’s Case Study
As a single dad, John relies on services like these to
help support him with his daughter Emily’s needs.
He stated, “the waitlist is horrible. I applied when
[Emily] was 11, and she is now 14 and only getting
her second [cycle].” Both John and Lisa [the ABA
consultant] acknowledged there is a need to have
supports for parents during this time and John
recalls being able to contact Lisa outside of the
program when he needed her help or advice.
“I did have a couple of challenges after [Emily]
was discharged. And it was kind of, ‘okay, now
what do I do?’ Luckily, I had [Lisa’s] phone
number, so I called her up and said ‘hey’. And she
was very helpful, but it wasn’t something that
was covered by the program. It was something
she did on her own time. And I didn’t think that
was fair to her, and I almost felt guilty doing that
to her. But she was extremely helpful.” (John)
 To read Emily’s full case study, see
Appendix 1.
“I know that there were some services that were tried…
to try and have them [parents] prepared for when they’re
coming into service, but I’m not sure. My recollection is
that they weren’t well attended, and they weren’t
accessed by families as they were waiting.” (Key
Informant)
“Unfortunately when you have a kid with, you know,
special needs and stuff… an hour is gold. You know? Like
your time is really precious, I find anyway.” (Parent)
Two examples of types of supports that parents and
caregivers desired are: (1) opportunities to talk to other
parents and share insight and support with others facing
similar challenges; and (2) opportunities to talk to
professional ASD consultants outside of the ABA setting.
Both of these supports would help parents to feel more
capable of handling day-to-day challenges, and less
overwhelmed by the ABA wait times.
One resource available to parents is Service
Coordination. However, while most families have access
to a Service Coordinator in their region, many do not
access the service and others find it unhelpful. As some
parents explained, Service Coordinators will answer specific questions related to service, but cannot
always spend time explaining the different options available, if the family is not clear in what they’re
looking for. This, in part, is due to the tremendous case load that each Service Coordinator is given. As one
key informant explained, some Service Coordinators have over 200 clients that they serve. As a result,
Service Coordinators end up largely helping families with complex cases or crisis situations, rather than
those who need help with navigation or understanding services. Consequently, ABA consultants often take
on a service coordination role for their clients.
“They [Service Coordinators] ask you, what do you need? I don’t know what I need… [So] I gave up
on them.” (Parent)
Finally, a number of parents/caregivers reported that they could not afford to purchase resources that
were recommended during service delivery (e.g. toys, books, puzzles). While some supplies were made
available at agency libraries, these were mostly adult resources. Libraries with client-friendly resources
were desired by parents and caregivers.
“The financial aspects of autism are equally as horrible as the problem with autism. Seeing people
put third mortgages on their house or cashing their RRSPs or working three jobs or whatever it
takes to get the money, it’s just a terrible thing.” (Parent)
ABA Cycle Two Evaluation Report
33 | P a g e
Underdeveloped partnerships with schools and mental health workers
Another challenge with program implementation is underdeveloped partnerships in many regions, both
with schools and school boards as well as various mental health agencies. This is an important challenge
as many goals cannot be attained or generalized without the cooperation of these agencies or institutions.
According to evaluation participants, schools across the Central West Region vary in the degree to which
they work with ABA consultants. For example, communication between school boards and ABA
consultants has been fairly effective in regions such as Wellington and areas of Peel. Some successful
stories were shared by evaluation participants, in which schools supported the attainment of a goal. In
cases where a child’s goal was related to behaviours at school, and schools were receptive to the ABA
program, there was often greater success in goal attainment, maintenance, and generalization. In regions
where these partnerships are less developed, however, consultants are finding that they must work with
schools on a case-by-case basis. A reluctance and even resistance to the ABA program was noted by some
evaluation participants at the school, school board and even union level.
“I stopped getting the school involved. Why bother? They will just say no.” (Parent)
“Hit and miss, I think. Some schools are willing to have conversations and some are not. Some are
willing to let you come in and observe, and some are like ‘we’re fine’.” (ABA Consultant)
According to evaluation participants, some principals, ASD itinerants, and special education teachers see
the value in connecting with ABA consultants, whereas others see it as an entrenchment on their own
treatment models and service. For example, a key informant from the school board explained that schools
in his region are generally willing to entertain conversations about collaborating with ABA consultants, and
are receptive to sharing knowledge and learning from each other.
“There has to be a respect for our [schools’] expertise and our knowledge. We do have behavioural
services, and I don’t need somebody to tell me how to do behavioural programs.” (Key Informant)
“I would say they’re receptive to it as a whole, in terms of supporting the need for ABA. In terms of
whether a worker can go in and do some of that implementation of ABA in the school or in
daycares or those settings, I’ve heard mixed messages.” (Key Informant)
ABA consultants and a few key informants noted that schools seemed to be more open to working
together when the ABA goal was behavioural in nature, especially in cases when the school faced difficulty
managing the behaviour of the child/youth. Schools were also more receptive when families were actively
working to bring the school representatives and ABA staff together.
ABA consultants and key informants from some regions also discussed the issue of underdeveloped
partnerships with mental health agencies and workers. Oftentimes, children/youth with both ASD and a
mental health issue face many unique challenges that cannot be resolved through the treatment approach
of the ABA program. ABA consultants do not have the abilities or skills to tackle a mental health concern,
which often takes precedence over the one goal of ABA. Yet in many regions, mental health workers and
ABA consultants have not been able to work collaboratively to address these challenges. A few consultants
reported that mental health workers sometimes overlook the mental health diagnosis and see it as a side
effect of ASD, and thereby assume that ABA treatment is sufficient for dual diagnosis clients. Without
ABA Cycle Two Evaluation Report
34 | P a g e
proper treatment, the client may ultimately end up in a crisis situation, possibly putting themselves and/or
their ABA consultant in danger. Thus, better partnerships with mental health agencies need to be
developed in many regions, so they can work together to best serve the child or youth in need.
Limited response to age transitions
Another key challenge of ABA program implementation revolved around age transitions, for children
switching agencies (i.e., at the age of 9 years), for youth entering high school, and for youth aging out of
the system (i.e., at the age of 17 years). Overall, there is an immediate need to make these transitions
more seamless, and to provide more support for families at these different stages of life.
A number of key informants reported inefficiencies in the transition for clients moving from ABA delivery
agencies serving 0-8 year olds to those serving clients between the ages of 9 to 17. For the most part,
minimal or no information is passed between agencies, which leads to confusion for families who felt that
the receiving agencies had no sense of their child/youth’s story. Staff and key informants also reported
frustration with this process, stating that consistent reports should be passed between agencies in order to
better serve clients. In response to these challenges, ABA delivery agencies have begun to discuss how this
transition could be improved.
“We don’t want parents to have to retell their stories every time… And they have to.” (Key
Informant)
Secondly, youth who are transitioning into high school also face unique challenges, and parents/caregivers
talked about the need for support from ABA during this time. For example, some parents said that their
children did really well in structured elementary school settings, but struggled upon entering high school
as they did not receive the same degree of direction when presented with recreational or free time. As
well, some parents wanted to talk to their children about issues such as puberty and romantic/sexual
relationships, but their children were uninterested or unable to speak about these topics with them.
Parents felt that the ABA program should be more proactive in guiding youth with ASD diagnoses through
these issues, as they may have different or unanticipated challenges than other children/youth.
“Going into the teenage years, it’s hard enough for a ‘normal’ kid to go through. But then [ASD] kids
who are struggling with the social – and social becomes uber huge – then it becomes even more
important to have these supports as they get older. It’s like [helping them] deal with hormones, or
opposite gender, or same gender if they’re struggling with that.” (Parent)
Thirdly, youth aging out of the ASD system struggle with this transition. ABA is only offered to youth up
until their 18th
birthday, after which there are few or no ASD services to support them. Various evaluation
participants expressed the limitations of this design, saying that many youth over the age of 17 still
desperately need services.
ABA Cycle Two Evaluation Report
35 | P a g e
Program Outcomes
As was found in Cycle 1, many of the anticipated outcomes that are outlined in the program logic model
(page 10) were in fact achieved. Most of the children and youth within the program reached their short-
term goals, and in this year’s evaluation we also found evidence that many experienced secondary
outcomes as a result of participating in the program. Family outcomes were not quite as positive; while
parents gained skills relevant to their child’s goal and often felt more confident in working with their child,
many still felt relatively unsupported outside of ABA. Thirdly, while some system-level outcomes were
positively reported, most evaluation participants still reported this to be the weakest outcome area in
terms of partnering with other agencies/institutions (schools, etc.), providing a stronger continuum of
support, and creating greater awareness of the ABA program itself.
This outcome section is structured similarly to the Cycle 1 report, with outcomes outlined according to the
three major outcome areas as shown in the program logic model: client (individual child/youth) outcomes,
individual family outcomes, and system-level outcomes.
Client Outcomes
Almost all children in the ABA program reached or exceeded their goal. Chart 12 shows the percentage of
clients (aged 9-17 years) across all agencies that attained each of the five levels of outcomes from the Goal
Attainment Scale (GAS). The two most common GAS outcomes were +2 (outcome much more than
expected; 37%) and +1 (outcome more than expected; 37%). In total, 97% of all clients (aged 9-17 years)
who completed the program between April 1, 2014 and December 31, 2014 reached or exceeded their
expected outcome.
Chart 12: Goal Attainment Scale (GAS) scores for children/youth aged 9-17 years (N=619)
Source: ABA Roll-Up (April 1, 2014 – December 31, 2014)
Chart 13 shows GAS scores by service provider. All regions had higher numbers of clients meeting or
exceeding goals than clients with unsatisfactory results. The most common GAS score across all regions
was +1 (six out of nine agencies). While an average of +1 might contribute to the perception that ABA
ABA Cycle Two Evaluation Report
36 | P a g e
programming is very successful, a few clinical supervisors explained that such a high score should not be
the norm. If the average score is +1, with a large number of children achieving +2, then programs have the
potential to be a lot more challenging and to push children to achieve more difficult goals. As they
explained, the average score should be zero (expected outcome), with +1 and +2 being less common.
Chart 13: Goal Attainment Scale score by service provider (N=619)
Source: ABA Roll-Up (April 1, 2014 – December 31, 2014)
In speaking with evaluation participants, most agreed that children and youth were achieving or exceeding
their goal(s). This was reported to be consistent across domain areas and most diagnoses.
“There’s an overwhelming amount of children that are meeting goals and making progress in their
skill development and their reduction of the maladapted behaviours.” (Key informant)
“The behaviours have lessened significantly. It’s incredible.” (Parent)
Chart 14 provides additional insight into parents’ and caregivers’ perspectives on client outcomes, as
reported in the Consumer Feedback Survey. As shown in this chart, the majority of parents (72%) either
agreed or strongly agreed that ABA goals were met.
0
10
20
30
40
50
60
70
80
90
100
110
TotalNumberofChildren(age9-17)
-2 Outcome
much less
than expected
-1 Outcome
less than
expected
0 Expected
outcome
reached
+1 Outcome
more than
expected
+2 Outcome
much more
than expected
ABA Cycle Two Evaluation Report
37 | P a g e
Chart 14: In my opinion, my son/daughter met his/her goal (N=415)
Source: Consumer Feedback Survey (April 1, 2014 to January 21, 2015)
Interestingly, 21% of parents/caregivers who completed the Consumer Feedback Survey felt “neutral” and
7% of parents/caregivers either disagreed or strongly disagreed that their child’s goal was met. These
numbers are higher than the 3% of clients who did not meet their goal according to the GAS. This
discrepancy demonstrates that parental or caregiver perceptions of ‘success’ (whether or not a goal was
achieved) appears to sometimes differ from the results of the GAS completed by staff. While the questions
were worded somewhat differently in these two quantitative measures, this apparent (and occasional)
discrepancy between GAS scores and parent/caregiver perspectives was further mirrored in several focus
groups. While most parents acknowledged that their child “met their goal” according to the GAS, some still
felt their child had a long way to go in order to maintain or generalize it. Some ABA consultants had similar
views, stating that the GAS scores do not always reflect what they would define as success for the client.
Chart 15 provides additional information on parent/caregiver perspectives. According to this chart, 83% of
parents agreed or strongly agreed that there was a positive change in the skills and/or behaviours
addressed in the ABA program. Thus, in comparison to data from Chart 14, more parents/caregivers (83%)
acknowledged a positive change as opposed to an achieved goal (72%).
ABA Cycle Two Evaluation Report
38 | P a g e
Chart 15: There has been a positive change in skills/behaviours addressed (N= 414)
Source: Consumer Feedback Survey (April 1, 2014 to January 21, 2015)
Some parents directly commented on their child’s ability to maintain or generalize the goal, post ABA.
However, because follow-up phone calls and in-person booster sessions are difficult to schedule, much of
this data is not tracked within program documentation.
“For the first ABA, he met his goals at home. But it had a ripple effect because he was less
anxious…He was also able to carry that forward into other areas.” (Parent)
Some parents/caregivers appreciated that after the program, their children exhibited fewer negative
behaviours (hitting, scratching, etc.) in relation to the goal or skill addressed. If the child, for example,
learned how to better communicate within the ABA program, than they might feel less frustrated when
interacting with people and exhibit fewer negative behaviours as a result. Parents and other evaluation
participants appreciated the preventative rather than punitive approach of ABA; ABA helps families to
reduce the incidence of negative behaviours, rather than react to these behaviours after they happen.
“I think it’s the one program that actually does hands-on work with families on how to approach a
problem… I think a lot of times families [receive] services that are consultative in nature, and I think
this is a hand-on, step-by-step approach to why this is happening, and what can be done to change
it.” (Key Informant)
Several evaluation participants reported heightened levels of self-confidence and independence in their
children following the ABA program. Some parents/caregivers reported that their child felt less stigmatized
and more “normal” when they learned to do things independently and to interact with others. According
to parents/caregivers and staff, it was very rewarding for the clients to see themselves achieve goals, and
to recognize their strengths.
“For him, it will be confidence for life.” (Parent)
ABA Cycle Two Evaluation Report
39 | P a g e
Case-in-Point: Eric’s Case Study
By the end of the program, Eric had achieved his first goal. He
learned how to follow a recipe and became confident in his
abilities to prepare different foods and to work with another
person – Rachel [his ABA consultant]. Eric appreciated that
Rachel “…was very enthusiastic”; his favourite part was “that I
had someone to help me with working on some areas that I
wasn’t necessarily good at”. Rachel equally enjoyed working
with Eric. As she explained:
“[Eric] warmed up quite quickly, just doing something
that he was enjoying. For [Eric], by the end, he was feeling
very confident and very proud of himself that he could do
these things he wasn’t able to do before.” (Rachel)
 To read Eric’s full case study, see Appendix 1.
“To interact normally with others in society, this is a huge skill.” (Parent)
Parents and caregivers consistently expressed positive outcomes when their child had home-based
sessions. On the other hand, perceptions of outcomes from the social skills groups varied. As some
evaluation participants explained, this may be partly due to the limited opportunities for parents to
observe or participate in these groups. Those who reported a positive change in their child explained that
social groups helped their son/daughter to build friendships and socialize with others. Clients were also
generally able to apply what they learned in their social skills groups to other environments with their
peers.
“My child relates to other kids better
and uses some of the strategies to
interact with peers.” (Parent)
“My child has more confidence and is
willing to try new things.” (Parent)
“With my son, it’s reduced his anxiety.
Because if he has some relationships
with some people, then he’s less
stressed when he goes places.”
(Parent)
Secondary outcomes
One recommendation from Cycle One was to
augment GAS scores with secondary
outcomes, recognizing the limitation of GAS in
capturing the full range of client outcomes. In response, secondary outcomes were tracked in Cycle Two.
“Parents will always ask you questions [about secondary outcomes] …So, yeah, the GAS is there but we’re
doing a lot more that we’re not able to show.” (ABA Consultant)
Of the 50 clients who were reported as having secondary outcomes, five were seen to have one secondary
outcome, and 31 had two or more. In contrast, 14 clients were seen to have no secondary outcomes. In
total, there were 94 secondary outcomes reported across the 50 clients. Some examples of the most
common secondary outcomes achieved by clients during Cycle Two are listed below. For the complete list
of secondary outcomes reported, see Appendix 6.
Social/Interpersonal
 Getting along better with siblings
 Increased self-confidence
 Initiating positive play with same-age peers
 Increased eye contact
ABA Cycle Two Evaluation Report
40 | P a g e
Communication
 Requesting items using sentences
 Increased depth and complexity of conversations
 Communicating effectively with new people
 Using greetings consistently
Behavioural/Emotional Regulation
 Decreased challenging behaviours
 Increased compliance and ability to listen/compromise
 Increased flexibility with new tasks
Daily Living
 Increased initiative with chore completion and ability to complete independently
 Better able to navigate purchasing items in stores
 Able to engage in larger variety of activities
Secondary outcomes were reported for 72% of clients tracked. On average, clients acheived two secondary
outcomes. As shown in Chart 16, secondary outcomes were fairly evenly distributed across domains.
Secondary outcomes were found most commonly in the Social/Interpersonal domain (at 29% of the time)
and communication the least (21% of the time).
Chart 16: Frequency of Secondary Outcomes by Secondary Outcome Domain Area (N=94)
Source: ABA Secondary Outcomes from November 2014 – February 2015
Chart 17 shows that most of the secondary outcomes occur in the home, especially in the case of Daily
Living, in line with the nature of ABA programming. On average, clients are making considerable positive
change in all four secondary outcome domain areas but the degree to which these outcomes are exhibited
are always most considerable in the home. The school environment is where occurrences of secondary
ABA Cycle Two Evaluation Report
41 | P a g e
outcomes were found the least. It is important to note, that though there were only 94 secondary
outcomes reported, some of these would occur in multiple environments simultaneously and to different
degrees of success.
Chart 17: Occurrence of Secondary Outcomes by Environment (N= 94)
Source: ABA Secondary Outcomes from November 2014 – February 2015
Chart 18 shows the relationship between one’s primary domain and related secondary outcomes. There
seems to be a close relationship between the social/interpersonal and communication domain areas on
the one hand, and behavioural and daily living goals on the other. That is to say, for clients whose primary
goal was social/interpersonal, they were most likely to have secondary outcomes within the
communications domain (and vice versa). Similarly, for clients whose primary goal area was behavioural,
they were most likely to experience secondary outcomes in the daily living domain (and vice versa). Within
this sample of 50 clients 100% of those with daily living as their primary goal reported seeing secondary
outcomes. In contrast, those with behaviour as their primary goal only reported 62% of clients having
achieved any secondary outcomes.
ABA Cycle Two Evaluation Report
42 | P a g e
Chart 18: Frequency of Secondary Outcomes According to Primary Goal Domain:
Source: ABA Secondary Outcomes from November 2014 – February 2015
Parent/Caregiver Outcomes
Many parents/caregivers reported some positive changes in their own lives as a result of participating in
the ABA program. Several parents/caregivers expressed increased confidence in their own ability to
support their child. They had a better understanding of how to engage with their child as a result of the
program, and felt more confident that change was possible after seeing their child achieve goals. As well,
families felt more self-assured in their ability to support their children, and felt that they gained a better
understanding of how to relate to their child. These perceptions were often mirrored by ABA consultants
and other program staff.
“It is such a big help to understand my child.” (Parent)
“[I feel] less stress, less frustration. The behaviour’s reduced. But once in a while when it pops up,
I‘m more able to understand why.” (Parent)
“I really like the parent involvement piece [of the ABA program]. I think, as a parent, if you can
learn different ways of relating to your child, whatever it may be, that’s always helpful.” (Key
Informant)
With increased confidence, parents reported feeling better able to cope with their child’s behaviour. Chart
19 reveals that 74% of parents agreed that after ABA, they feel more capable of helping their child with
their new skill, or in dealing with their behaviour.
ABA Cycle Two Evaluation Report
43 | P a g e
Chart 19: I feel better because I am better able to cope with this skill/behaviour (N=410)
Source: Consumer Feedback Survey (April 1, 2014 to January 21, 2015)
Parent training is an important part of the ABA program and provides parents/caregivers with techniques
and ideas to help them address their child’s skills and behaviours. According to Chart 20, the majority of
parents (79%) were able to apply the techniques they were taught by ABA consultants. Moreover, several
parents who participated in this evaluation reported that they had leveraged what they had learned in ABA
to address other maladaptive behaviours and to teach their children new skills.
ABA Cycle Two Evaluation Report
44 | P a g e
Chart 20: I am able to use these techniques to address the skills/behaviours on my own (N=413)
Source: Consumer Feedback Survey (April 1, 2014 to January, 21 2015)
The success of the program was seen to be family-dependent. Many evaluation participants noted that
better outcomes were achieved for clients when parents were actively involved. According to program
staff, there was less likelihood that ABA goals would be maintained if the parents were unengaged in the
sessions. Yet, even parents/caregivers who actively participated in the program reported difficulty
maintaining the goals over the long-term without some additional professional support.
System-Level Outcomes
There was comparatively less data regarding system-level outcomes compared to other program
outcomes. Program stakeholders seemed to have difficulty commenting on system-level outcomes
achieved as a result of the ABA program to date. Not surprisingly, it was key informants who spoke most
about system-level outcomes, identifying continued strength in some ASD interagency coordination and
collaboration, however also acknowledging limitations in terms of partnering with external
agencies/institutions (schools, etc.), providing a stronger continuum of support, and creating greater
awareness of the ABA program itself.
According to key informants there continues to be stronger coordination evident among ASD agencies
across the Central West Region. Agencies that did not previously collaborate were now more aware of
each other’s services.
“ABA providers certainly get together to improve the program and discuss issues about the
program.” (Key Informant)
“I think our partnership is definitely working well. As I said, we meet with the clinical partners on a
monthly basis and then bi-monthly usually there is the ABA service provider meeting...[We are]
always trying to make it more seamless and transparent with families and how can we go about
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report
ABA Program and Services - Cycle 2 Evaluation Report

More Related Content

What's hot

Nancy Warren Mini-U Presentation March 2014
Nancy Warren Mini-U Presentation March 2014Nancy Warren Mini-U Presentation March 2014
Nancy Warren Mini-U Presentation March 2014
Curamericas
 
Sharon K Kulesz July 2015 Resume (r)
Sharon K Kulesz July 2015 Resume (r)Sharon K Kulesz July 2015 Resume (r)
Sharon K Kulesz July 2015 Resume (r)
Sharon Kulesz
 
Bringing Operations Research to Life_Akogbeto_Riese_5.2.12
Bringing Operations Research to Life_Akogbeto_Riese_5.2.12Bringing Operations Research to Life_Akogbeto_Riese_5.2.12
Bringing Operations Research to Life_Akogbeto_Riese_5.2.12
CORE Group
 
Strengthening the Supply Chain Workforce through Mentorship and On the Job Tr...
Strengthening the Supply Chain Workforce through Mentorship and On the Job Tr...Strengthening the Supply Chain Workforce through Mentorship and On the Job Tr...
Strengthening the Supply Chain Workforce through Mentorship and On the Job Tr...
JSI
 

What's hot (20)

Nancy Warren Mini-U Presentation March 2014
Nancy Warren Mini-U Presentation March 2014Nancy Warren Mini-U Presentation March 2014
Nancy Warren Mini-U Presentation March 2014
 
Social Accountability for Improved Community Health Winch
Social Accountability for Improved Community Health WinchSocial Accountability for Improved Community Health Winch
Social Accountability for Improved Community Health Winch
 
Colorado Beacon Annual Report 2011
Colorado Beacon Annual Report 2011Colorado Beacon Annual Report 2011
Colorado Beacon Annual Report 2011
 
Using the government health system to deliver nutrition interventions in Bang...
Using the government health system to deliver nutrition interventions in Bang...Using the government health system to deliver nutrition interventions in Bang...
Using the government health system to deliver nutrition interventions in Bang...
 
Social Accountability for Improved Community Health Otchere
Social Accountability for Improved Community Health OtchereSocial Accountability for Improved Community Health Otchere
Social Accountability for Improved Community Health Otchere
 
Social Accountability for Improved Community Health Shanklin
Social Accountability for Improved Community Health ShanklinSocial Accountability for Improved Community Health Shanklin
Social Accountability for Improved Community Health Shanklin
 
Sharon K Kulesz July 2015 Resume (r)
Sharon K Kulesz July 2015 Resume (r)Sharon K Kulesz July 2015 Resume (r)
Sharon K Kulesz July 2015 Resume (r)
 
California Community Care Coordination Collaborative - September 2014
California Community Care Coordination Collaborative - September 2014California Community Care Coordination Collaborative - September 2014
California Community Care Coordination Collaborative - September 2014
 
Moving From Community Assessment to Sustainable Strategic Plan
Moving From Community Assessment to Sustainable Strategic PlanMoving From Community Assessment to Sustainable Strategic Plan
Moving From Community Assessment to Sustainable Strategic Plan
 
Saurav Tiwary
Saurav Tiwary Saurav Tiwary
Saurav Tiwary
 
Bringing Operations Research to Life_Akogbeto_Riese_5.2.12
Bringing Operations Research to Life_Akogbeto_Riese_5.2.12Bringing Operations Research to Life_Akogbeto_Riese_5.2.12
Bringing Operations Research to Life_Akogbeto_Riese_5.2.12
 
Faith Chambers Resume - Director of Health Services - Managed Care Specialis...
Faith Chambers Resume  - Director of Health Services - Managed Care Specialis...Faith Chambers Resume  - Director of Health Services - Managed Care Specialis...
Faith Chambers Resume - Director of Health Services - Managed Care Specialis...
 
Day 2 panel 4 quality improvement for mnh tz 108020
Day 2 panel 4 quality improvement for mnh tz 108020Day 2 panel 4 quality improvement for mnh tz 108020
Day 2 panel 4 quality improvement for mnh tz 108020
 
Sustaining quality approaches for locally embedded community health services ...
Sustaining quality approaches for locally embedded community health services ...Sustaining quality approaches for locally embedded community health services ...
Sustaining quality approaches for locally embedded community health services ...
 
Alignment slides
Alignment slidesAlignment slides
Alignment slides
 
Creating Your Accountability Blueprint
Creating Your Accountability BlueprintCreating Your Accountability Blueprint
Creating Your Accountability Blueprint
 
Hygiene promotion at scale and working with the private sector
Hygiene promotion at scale and working with the private sectorHygiene promotion at scale and working with the private sector
Hygiene promotion at scale and working with the private sector
 
Strengthening the Supply Chain Workforce through Mentorship and On the Job Tr...
Strengthening the Supply Chain Workforce through Mentorship and On the Job Tr...Strengthening the Supply Chain Workforce through Mentorship and On the Job Tr...
Strengthening the Supply Chain Workforce through Mentorship and On the Job Tr...
 
Quality improvement programme launch event slides
Quality improvement programme   launch event slidesQuality improvement programme   launch event slides
Quality improvement programme launch event slides
 
Falls Prevention Program Implementation
Falls Prevention Program ImplementationFalls Prevention Program Implementation
Falls Prevention Program Implementation
 

Similar to ABA Program and Services - Cycle 2 Evaluation Report

Post Project Sustainability Study: Health, Wat/San and Agricultural Intervent...
Post Project Sustainability Study: Health, Wat/San and Agricultural Intervent...Post Project Sustainability Study: Health, Wat/San and Agricultural Intervent...
Post Project Sustainability Study: Health, Wat/San and Agricultural Intervent...
CORE Group
 
Families and alcohol
Families and alcohol Families and alcohol
Families and alcohol
localinsight
 
Task Force Reviewand%20 Charge[1]
Task Force Reviewand%20 Charge[1]Task Force Reviewand%20 Charge[1]
Task Force Reviewand%20 Charge[1]
Timothy M. Ervolina
 
Assessing a Healthcare Program Policy Evaluation.docx
Assessing a Healthcare Program Policy Evaluation.docxAssessing a Healthcare Program Policy Evaluation.docx
Assessing a Healthcare Program Policy Evaluation.docx
write22
 
Introduction to Program Evaluation for Public Health.docx
Introduction to Program Evaluation for Public Health.docxIntroduction to Program Evaluation for Public Health.docx
Introduction to Program Evaluation for Public Health.docx
mariuse18nolet
 
Introduction to Program Evaluation for Public Health.docx
Introduction to Program Evaluation for Public Health.docxIntroduction to Program Evaluation for Public Health.docx
Introduction to Program Evaluation for Public Health.docx
bagotjesusa
 
Running head LOGIC MODELLOGIC MODEL 2Logic modelStu.docx
Running head LOGIC MODELLOGIC MODEL 2Logic modelStu.docxRunning head LOGIC MODELLOGIC MODEL 2Logic modelStu.docx
Running head LOGIC MODELLOGIC MODEL 2Logic modelStu.docx
wlynn1
 
Project PARIS: Parents And Residents In Session
Project PARIS: Parents And Residents In SessionProject PARIS: Parents And Residents In Session
Project PARIS: Parents And Residents In Session
Picker Institute, Inc.
 
Program evaluation
Program evaluationProgram evaluation
Program evaluation
Nongre Arphon
 
Running head DIVERSITY AND MULTICULTURALISM IN HEALTH CARE .docx
Running head DIVERSITY AND MULTICULTURALISM IN HEALTH CARE     .docxRunning head DIVERSITY AND MULTICULTURALISM IN HEALTH CARE     .docx
Running head DIVERSITY AND MULTICULTURALISM IN HEALTH CARE .docx
susanschei
 
香港六合彩
香港六合彩香港六合彩
香港六合彩
taoyan
 

Similar to ABA Program and Services - Cycle 2 Evaluation Report (20)

Post Project Sustainability Study: Health, Wat/San and Agricultural Intervent...
Post Project Sustainability Study: Health, Wat/San and Agricultural Intervent...Post Project Sustainability Study: Health, Wat/San and Agricultural Intervent...
Post Project Sustainability Study: Health, Wat/San and Agricultural Intervent...
 
Families and alcohol
Families and alcohol Families and alcohol
Families and alcohol
 
Improving quality in the early years report
Improving quality in the early years reportImproving quality in the early years report
Improving quality in the early years report
 
Celebrating Learning Through Evaluation Report
Celebrating Learning Through Evaluation ReportCelebrating Learning Through Evaluation Report
Celebrating Learning Through Evaluation Report
 
Task Force Reviewand%20 Charge[1]
Task Force Reviewand%20 Charge[1]Task Force Reviewand%20 Charge[1]
Task Force Reviewand%20 Charge[1]
 
Presentation of the Parent Aid Program
Presentation of the Parent Aid ProgramPresentation of the Parent Aid Program
Presentation of the Parent Aid Program
 
Assessing a Healthcare Program Policy Evaluation.docx
Assessing a Healthcare Program Policy Evaluation.docxAssessing a Healthcare Program Policy Evaluation.docx
Assessing a Healthcare Program Policy Evaluation.docx
 
Introduction to Program Evaluation for Public Health.docx
Introduction to Program Evaluation for Public Health.docxIntroduction to Program Evaluation for Public Health.docx
Introduction to Program Evaluation for Public Health.docx
 
Introduction to Program Evaluation for Public Health.docx
Introduction to Program Evaluation for Public Health.docxIntroduction to Program Evaluation for Public Health.docx
Introduction to Program Evaluation for Public Health.docx
 
BFDI Legal Advocacy: Evaluation Strategy Paper
BFDI Legal Advocacy: Evaluation Strategy PaperBFDI Legal Advocacy: Evaluation Strategy Paper
BFDI Legal Advocacy: Evaluation Strategy Paper
 
Monitoring and Evaluating the Transition of Large-Scale Programs in Global He...
Monitoring and Evaluating the Transition of Large-Scale Programs in Global He...Monitoring and Evaluating the Transition of Large-Scale Programs in Global He...
Monitoring and Evaluating the Transition of Large-Scale Programs in Global He...
 
Evaluating Impact of OVC Programs: Standardizing our methods
Evaluating Impact of OVC Programs: Standardizing our methodsEvaluating Impact of OVC Programs: Standardizing our methods
Evaluating Impact of OVC Programs: Standardizing our methods
 
Running head LOGIC MODELLOGIC MODEL 2Logic modelStu.docx
Running head LOGIC MODELLOGIC MODEL 2Logic modelStu.docxRunning head LOGIC MODELLOGIC MODEL 2Logic modelStu.docx
Running head LOGIC MODELLOGIC MODEL 2Logic modelStu.docx
 
Publishing in Public Health
Publishing in Public HealthPublishing in Public Health
Publishing in Public Health
 
Evaluating and Developing the Early Education Pilot for Two Year Olds
Evaluating and Developing the Early Education Pilot for Two Year OldsEvaluating and Developing the Early Education Pilot for Two Year Olds
Evaluating and Developing the Early Education Pilot for Two Year Olds
 
Project PARIS: Parents And Residents In Session
Project PARIS: Parents And Residents In SessionProject PARIS: Parents And Residents In Session
Project PARIS: Parents And Residents In Session
 
Program evaluation
Program evaluationProgram evaluation
Program evaluation
 
The Money Mentors Programme
The Money Mentors ProgrammeThe Money Mentors Programme
The Money Mentors Programme
 
Running head DIVERSITY AND MULTICULTURALISM IN HEALTH CARE .docx
Running head DIVERSITY AND MULTICULTURALISM IN HEALTH CARE     .docxRunning head DIVERSITY AND MULTICULTURALISM IN HEALTH CARE     .docx
Running head DIVERSITY AND MULTICULTURALISM IN HEALTH CARE .docx
 
香港六合彩
香港六合彩香港六合彩
香港六合彩
 

ABA Program and Services - Cycle 2 Evaluation Report

  • 1. Cycle 2 Evaluation Report March 2015 ABA Services and Supports for Children and Youth in the Central West Region
  • 2. ABA Cycle Two Evaluation Report 2 | P a g e Evaluation Steering Committee Kim Rodrigues (KPAS – Halton, Dufferin, Wellington, Waterloo) Pam Tibbetts (BCBA – ROCK Halton) Cathy White (Peel District School board – ret.; Autism Ontario) Gail Jones (KPAS – Peel) Angela Davis (Peel Behavioural Services) Georgia Zadow (Parent, advocate) Brett Friesen (CMHA – Waterloo Wellington Dufferin) Debbie Dyment (Community Living North Halton) CCBR Research Team Rich Janzen Kimia Ghomeshi Kyla English Jane Garant Tangul Bilgehan Centre for Community Based Research (CCBR) 73 King Street West, Suite 300 Kitchener, Ontario N2G 1A7 Phone: (519) 741-1318 Fax: (519) 741-8262 E-mail: rich@communitybasedresearch.ca Website: www.communitybasedresearch.ca
  • 3. ABA Cycle Two Evaluation Report 3 | P a g e Contents Executive Summary.......................................................................................................................................4 Introduction ..................................................................................................................................................8 Background ...............................................................................................................................................8 Evaluation Purpose ...................................................................................................................................9 Main Research Questions .........................................................................................................................9 Evaluation Design........................................................................................................................................10 Evaluation Approach...............................................................................................................................10 Program Theory of Change.....................................................................................................................11 Cycle Two Methods.................................................................................................................................14 Evaluation Findings .....................................................................................................................................16 Program Implementation .......................................................................................................................16 Program Outcomes.................................................................................................................................35 Future Directions ....................................................................................................................................46 Recommendations......................................................................................................................................51 Recommendations for the ABA Program................................................................................................51 Recommendations for Cycle Three Evaluation.......................................................................................53 References..................................................................................................................................................54 Appendices..................................................................................................................................................55
  • 4. ABA Cycle Two Evaluation Report 4 | P a g e EXECUTIVE SUMMARY This report summarizes the results of the second evaluation cycle (April 2014-March 2015) for the Applied Behaviour Analysis (ABA) program delivered to children and youth (ages 9-17) in the Central West Region of Ontario. The lead partner for this program is Kerry’s Place Autism Services (KPAS), in partnership with other key agencies in the region. KPAS works collaboratively with Community Living North Halton (CLNH), Peel Behavioural Services (PBS), Reach Out Centre for Kids (ROCK), Sunbeam Residential Development Centre (DSRC), and the Canadian Mental Health Association Waterloo Wellington Dufferin (CMHA). As part of the Cycle 1 evaluation, an ABA program logic model was created to explain the continuum of service. The process starts with intake/referral, followed by service planning, ABA service delivery, and discharge and interfacing. A child/youth and their family may go through the process multiple times before they reach the age of 18. System-level activities are also undertaken with KPAS partnership and external agencies. Both quantitative and qualitative methods were used for this evaluation. The quantitative methods included the ABA roll-up (including the Goal Attainment Scale (GAS)), the Consumer Feedback Survey, the Central West Region (CWR) ABA Services Intake Management), Time Utilization sheets, and the Secondary Outcomes tool. Qualitative methods included focus groups with families, focus groups with staff, key informant interviews, and in-depth case studies. Evaluation findings were categorized under the three main headings: program implementation, outcomes, and future directions. Implementation: In general, evaluation participants expressed satisfaction with the program and an increasing number of families are interested in accessing ABA services. As a result, the waitlist continues to grow in length. KPAS (which has centres in all the regions) provides service to the highest number of children/youth overall, followed by PBS. The majority of families receiving or waiting for services live in the Peel region. Families have the choice of participating in individual or group sessions; most families choose individual sessions – often in the home setting – to meet their goals. Data from Cycle Two revealed five key strengths of program implementation. These included: 1. Consistency in program planning (e.g., the matching of consultants with past clients); 2. Variation in programming options (e.g., individual vs. groups sessions, at-home vs. agency settings); 3. Opportunities to work with highly-skilled consultants; 4. The ability to focus families on manageable change; and 5. Effective partnerships among ASD agencies.
  • 5. ABA Cycle Two Evaluation Report 5 | P a g e Despite these strengths, evaluation participants also identified a number of challenges, many of which were highlighted in Cycle 1. The six most significant challenges that emerged from Cycle 2 are as follows: 1. Unsustainable targets, leading to staff burnout); 2. Limited program dosage, due to long wait lists and short program lengths; 3. Lack of other supports for parents during ABA and while on the waitlist; 4. Underdeveloped partnerships with schools and mental health agencies, leading to difficulties in achieving and maintaining goals; 5. Limitations with social skills groups, with respect to matching clients and parental involvement. Outcomes: The evaluation found that many of the outcomes anticipated by the program were in fact being achieved. Often, families are seeing significant positive changes in their children’s behaviours and skills. Most children either reached or exceeded their goals as indicated by both evaluation participants and the GAS scores. Moreover, new data from Cycle Two revealed the presence of secondary outcomes within ABA programming as well. Over 70% of children who were tracked using this tool experienced at least one secondary outcome, such as getting along better with siblings. Parents and caregivers themselves are also experiencing positive outcomes including heightened levels of confidence and an increased capacity to cope with challenging situations. Nevertheless, parents/caregivers reported some difficulty in maintaining goals over the long-term. At the system level, there has been strong coordination between ASD agencies as well as an increase in collaboration with external agencies (e.g., schools, mental health agencies, etc.) in some regions. However, participants also indicated that significant challenges remain regarding working collaboratively with schools, as well as providing coordinated services for clients with a dual-diagnosis. Future Directions: Participants provided numerous suggestions to enhance the effectiveness of ABA programming. For example, parents and some key informants identified a need for a reduction in the waitlist and for more resources and support. Below are the key recommendations that emerged from the Cycle Two report, based on discussions with the evaluation Steering Committee as well as suggestions from evaluation participants. Recommendations related to changes in Cycle Three of the evaluation are provided at the end of the report. Recommendation #1: Increase program resources to sustainably meet service targets and achieve long- term outcomes for the diversity of clients and their families. Prioritize the following program improvements:  Hire additional ABA consultants to reduce caseloads, shorten wait lists, and to improve the diversity of staff.  Provide flexibility in program length, with the option of longer program cycles for families who wish to work on goals that would require more treatment time.  Provide semi-regular capacity-building opportunities for staff, including workshops, conference participation, and annual training sessions.
  • 6. ABA Cycle Two Evaluation Report 6 | P a g e  Add client-oriented supplies (e.g. toys, activities, etc.) to the ABA family resource centres. Recommendation #2: Work with community partners to provide a range of additional resources in multiple formats and languages (as needed) for parents/caregivers on the wait list. Prioritize the following:  Information and orientation sessions on the ABA program, its family-centered approach, and goal setting process. Orientations (online or in-person) could be mandatory for new families.  Workshops or resources for ABA clients and parents/caregivers on strategies to support youth with ASD during their transition to high school or adulthood, or their transition from ABA agencies serving up to eight year-olds to agencies serving 9-17 year-olds.  An online forum for families that can be used to correspond with other families.  Resources such as video tutorials on behavioural management of children/youth with ASD.  Group social activities and related resources for children/youth with ASD. Recommendation #3: Continue to adjust the first-come-first-serve policy for clients in the social skills domain so that clients in each group are selected according to similar social capacities and needs. As well, organize visits (in-person or by phone) with clients being considered for social skills groups to ensure that clients are well matched. Recommendation #4: Explore any possible options to start the service planning process while families are on the wait list (e.g. provide examples of possible ABA goals or illustrations of how the program works). Recommendation #5: Explore or further implement group models of services, such as summer groups or dyads. Recommendation #6: Improve the goal setting process in the following ways:  Suggest that parents select goals that address clients’ more significant behavioural challenges.  Provide examples of possible ABA goals and program illustrations.  Set goals for parent/caregivers to boost their involvement in the program.  Further seek to involve other ASD-related professionals when appropriate to collaboratively create the treatment plans.  Continue to attempt to consult with the client’s school when the goal that would require school involvement. Recommendation #7: Continue having discussions with the Ministry of Child and Youth Services regarding the suitability of the Child and Adolescent Needs and Strengths (CANS) to the ABA program. Recommendation #8: Continue to explore options for including more natural settings in ABA sessions, particularly for clients in the social skill and daily living domains to support them in navigating real life situations.
  • 7. ABA Cycle Two Evaluation Report 7 | P a g e Recommendation #9: Further improve reporting mechanisms and increase parental involvement in social skill interventions to strengthen long-term outcomes for participating children/youth. Recommendation #10: Work with community partners to provide a treatment model for older clients that is more age appropriate and engaging. Recommendation #11: In the absence of a short wait list, institute bi-monthly follow-ups (via telephone or email) following the booster sessions (when requested by parents/caregivers) to ensure maintenance of goals and to support parents/caregivers with challenges as they arise before the next program cycle. Recommendation #12: As part of the re-referral process, continue to identify whether families would want to work with the same consultant in the next program cycle when possible and appropriate. Recommendation #13: Prepare more detailed informational materials on the ABA program to distribute to other ASD agencies, mental health and health care services, and schools. These resources should clearly outline what the program is (and is not), and for schools should highlight how the program complements the ASD services and supports they already offer. In some instances, organize meetings between school representatives and ABA managers, including testimonials from principals, families, and teachers who have had a positive experience with the program. Recommendation #14: ASD agencies to prepare a transition plan for children/youth moving from agencies serving clients up to eight year-olds to agencies serving 9 to 17 year-olds.
  • 8. ABA Cycle Two Evaluation Report 8 | P a g e INTRODUCTION This report summarizes the Cycle Two evaluation findings of the Applied Behaviour Analysis (ABA) services for children and youth (aged 9 to 17 years) in the Central West Region of Ontario (Waterloo, Wellington, Dufferin, Halton, and Peel). The report covers the evaluation period beginning April 1, 2014 and ending March 31, 2015, and is the second of three evaluation cycles ending in MaAs lead agency for the Central West Region, Kerry’s Place Autism Services (KPAS) contracted the Centre for Community Based Research (CCBR) to lead the evaluation under the guidance of a cross-stakeholder evaluation steering committee. Background ABA services in the Central West Region are delivered to children and youth ages 9 to 17 years with Autism Spectrum Disorder (ASD) through a network of service agencies known as the ABA Central West Region. This network delivers services that are focused on establishing skills and/or addressing key issues that will improve the quality of life for children and youth with ASD. These services are oriented around the following four domains:  Communication  Behaviour/emotional regulation  Social skills  Daily living skills The partnering agencies also provide support to the parents/caregivers of children with ASD, aiming to improve their quality of life, enhance their behaviour management skills, and instill a sense of competence. To address these goals, a flexible support model is developed collaboratively with participating children or youth and their parents/caregivers, and can include group, individual, family, and parental capacity supports. Training is provided to parents and caregivers while their child or youth is participating in ABA programming. Partnering service delivery agencies for children/youth ages 9 to 17 years include Kerry’s Place Autism Services (KPAS; lead partner), Community Living North Halton (CLNH), Peel Behavioural Services (PBS), Reach Out Centre for Kids (ROCK), Sunbeam Residential Development Centre (DSRC), and the Canadian Mental Health Association Waterloo Wellington Dufferin (CMHA). Another partnership exists for ABA services delivered to children aged zero to eight years which is led by ErinoakKids (EOK) and includes Dufferin Child and Family Services (DCAFS), KidsAbility Children’s Centre (KidsAbility), and Woodview Children’s Centre (WCC). A joint waitlist for both partnerships is managed by KPAS. The ABA Central West Region is committed to the development, delivery, evaluation, and continuous improvement of its ABA-based services and supports. To this end, they requested that an evaluation of the ABA services be conducted to enable the program to make evidence-based decisions for future program delivery. The evaluation was to be grounded in the perspectives of children and youth, their families, and the systems of service in their communities. The scope of the evaluation was to be focused on the service delivery partnership for children ages 9 to 17 (i.e., the KPAS-led partnership).
  • 9. ABA Cycle Two Evaluation Report 9 | P a g e Evaluation Purpose The evaluation was designed to understand how existing ABA programming is being implemented at each of the Central West agencies for children and youth ages 9 to 17. In particular, the evaluation was designed to assess what is working well and what is not working well within the context of ABA programming. The evaluation should therefore clarify what helps and hinders ABA Central West from creating the desired change that they wish to see in children and youth, their families, and the networks of ASD-related services in their communities. In other words, the evaluation seeks to understand the processes of ABA programming, and how these processes link to intended outcomes. This evaluation was also designed to explore the impact of ABA programming in the Central West Region. Specifically, the evaluation explores the outcomes of ABA on: 1) children and youth (aged 9 to 17); 2) families/caregivers of children and youth clients; and 3) the networks of ASD-related services. Implementing a multi-cycle evaluation allows the program to consider outcomes over time. Finally, the evaluation was designed to be forward looking. Data from the evaluation provides concrete recommendations on how the Central West Region can improve their ABA services in the future for children/youth, their families, and within the local networks of ASD-related services. Overall, the evaluation has three inter-related objectives:  To assess the implementation processes of ABA programming in the Central West Region  To assess the outcomes of ABA programming at the child-, family-, and systems-levels  To identify future directions for improving ABA services in the Central West Region Main Research Questions Three main research questions guided the evaluation, consistent with the three objectives mentioned above. These questions (and corresponding sub-questions) are listed below, and form the basis of how this report is organized. 1. How is the ABA program being implemented at agencies in the Central West Region? (process) a) What is the underlying program theory, including the main activities and expected outcomes? b) What are the child and family demographics in ABA Central West Region communities? c) What resources (e.g., human, financial, partnership) support the various aspects of program functioning (i.e. inputs)? d) What aspects of the program seem to be working well? Not working well? e) What is facilitating and what is hindering effective program implementation? f) How are the guiding principles evident in service delivery? 2. How and to what extent has ABA programming impacted participating children/youth and their families? (outcomes) a) What are the main accomplishments and products of the ABA program? (i.e. outputs)
  • 10. ABA Cycle Two Evaluation Report 10 | P a g e b) What outcomes has the ABA program had for participating children and youth in the domains of communication, behaviour/emotional regulation, social skills, and daily living skills? c) What outcomes has the ABA program had for the families of participating children and youth in terms of their quality of life, enhanced behaviour management skills, and sense of competence? d) What systems level outcomes has ABA programming achieved? e) How have outputs and outcomes changed over time? 3. What recommendations can be made to improve the outcomes of ABA services for children, families, and the networks of ASD-related services in their communities? (future directions) a) What should be done to enhance the effectiveness of the ABA programs for children and youth? For families? For the network of ASD-related services in their communities? b) How should future ABA programming be evaluated? EVALUATION DESIGN Evaluation Approach This evaluation is carried out using an approach that is consistent with the three hallmarks of community-based research: community-situated, collaborative and action-oriented (Ochocka & Janzen, 2014). Community-situated. Community-situated means that research is of practical relevance to community members and is carried out in community settings, rather than being driven simply by researcher interests. This evaluation therefore seeks to understand the program context across the Central West Region, paying close attention to the program’s guiding principles. It also means that the evaluation draws on information already collected by the program (i.e., secondary data), while involving ABA staff in helping to collect primary data (e.g., helping to organize interviews and focus groups). Collaborative. Collaborative means that community members and researchers share control of the research agenda through reciprocal involvement in the research design, implementation and dissemination. The ABA services evaluation therefore makes sure to involve different stakeholders (including children/youth, parent/caregivers, various levels of staff, and community partners) throughout the evaluation process. Concretely, most stakeholders are involved through the evaluation Steering Committee that guides each step of the evaluation. This Committee creates momentum and increases the likelihood that findings will be acted upon. In addition, stakeholders are also involved as research participants. Action-oriented. Action-oriented means that the process and results of research are useful to community members in making positive change. This evaluation is therefore utilization-focused, meaning it is designed to be as useful as possible. Action-oriented also means that there is ongoing
  • 11. ABA Cycle Two Evaluation Report 11 | P a g e feedback of the evaluation findings so that action can be taken following each evaluation cycle. To this end, the evaluation report includes concrete recommendations for future action. Action-oriented also means that the evaluation is open to improvement from one cycle to the next; recommendations to improve Cycle Three of the evaluation are included at the end of this report. Program Theory of Change An understanding the ABA Central West Region’s program theory helped to guide the evaluation design. A program theory is an explanation of how a program is expected to be an intervention for positive change (see Chen, 2005). Such a theory is useful for evaluations as it brings clarification among stakeholders about what the program is intended to do, and it informs the development of evaluation data collection tools. The program logic model is presented below. The program logic model was developed collaboratively by reviewing project documents, holding a site visit, and through Steering Committee discussion. Overall, the logic model emphasizes the individual and systems-level outcomes specific to ABA programming, while acknowledging that these services are one element within the entire continuum of care for children and youth ages 9 to 17 years with ASD. Horizontally, the program logic model activities and outcomes are arranged according to where they would occur within the service delivery flow. These categories include: Intake/Referral, Service Planning, ABA Services Delivery (divided according to Child/Youth and Parent/Caregiver), Discharge and Interfacing; and System (divided according to ‘With Partner’ and ‘With External Agencies’). The program logic model shows the service delivery flow in a time-ordered fashion from first contact to discharge/interfacing. Note that at the end of discharge/interfacing, a dashed arrow indicates that program re-entry may occur. Notice also that the system-level category is not connected to the service delivery flow with an arrow as it is assumed that the system operates externally and concurrently with ABA programming. Waitlist resources are visually shown in the shape of a circle following intake (where clients first access them), but in fact these resources cut across the ASD continuum of care. Vertically, the program logic model is arranged according to Activities, Outcomes (both individual and system), and Program Mission. Activities detail the parts of the service delivery. The set of activities for each step in the service delivery flow unpack the work that is undertaken by the network partners. System-level activities are also noted (both with partners and with external agencies). Individual Outcomes are those that benefit children/youth and their parents/caregivers, while System Outcomes are those that benefit ABA Central West partners and external agencies/institutions. Notice that the Service Planning and the Discharge & Interfacing categories lead to the primary ABA program outcomes at the individual and family level. It should also be noted that the Parent/Caregiver individual outcomes fold into the Child/Youth outcomes, as ABA programming is designed to improve the capacity of parents/caregivers to support improvements to their child’s behaviour. Similarly, system-level outcomes at the individual level (which improve participants’ ability to navigate the ASD continuum of care) fold into Parent/Caregiver outcomes. Finally, System Outcomes are shown to cradle the entire service delivery flow, as their impact is reflected throughout the ABA program and continuum of care.
  • 12. ABA Cycle Two Evaluation Report 12 | P a g e The Program Mission is noted at the bottom of the logic model. This mission captures the guiding principles of service delivery outlined by the Ministry of Children and Youth Services. This Ministry document identifies the purpose of ABA services as they pertain to improving the services and supports for children with ASD. The ABA Central West Region theory of change has been evolving and will likely continue to evolve over time. A deepening understanding of program theory is expected to occur as the program matures and responds to changing circumstances and evaluation learnings. This means that the evaluation will need to be open to exploring and capturing the unexpected, as well as the anticipated outcomes and processes. It also means that the evaluation itself can be an intervention in further clarifying and honing the program theory (Janzen, Seskar-Hencic, Dildar & McFadden, 2012).
  • 13. ABA Cycle Two Evaluation Report 13 | P a g e 2014 ABA Program Logic Model
  • 14. ABA Cycle Two Evaluation Report 14 | P a g e Cycle Two Methods This section outlines the methods of data gathering for Cycle Two of the ABA services evaluation. The number of methods and the volume of data collected for Cycle Two was an expansion upon Cycle One which was a shorter evaluation cycle. The evaluation methods were determined through a systemic process of building an evaluation framework. The evaluation framework process clarified the main research questions, program theory (i.e., program logic model), and corresponding measurement matrix (see Appendix 2). The measurement matrix links expected program outcomes (impact measurement) and processes (performance measurement) with indicators and corresponding data collection tools. The matrix also lists questions about future directions (strategy measurement) linking them with corresponding data collection tools. Based on this measurement matrix, data collection tools were identified or developed. The individual tools include a mix of existing measures used by the ABA Central West Region and those developed by the evaluation team. Some of the quantitative tools introduced in Cycle Two were developed to address gaps in secondary data according to Cycle One findings. The qualitative tools that were introduced in Cycle One were also revised and expanded to further explore major themes that emerged in the first cycle, and to examine the relationship between the clients’ domain area or diagnosis and their experience in the program. Combined, the tools include a mix of quantitative and qualitative measures to provide breadth and depth of insight. Appendix 3 includes primary data collection tools with corresponding consent forms. Cycle Two methods are listed below, organized into quantitative methods (representing secondary data already gathered by the program) and qualitative methods (representing primary data that was collected specifically for the evaluation). Quantitative Methods (Secondary Data) ABA Roll-up. The ABA Roll-up is an Excel spreadsheet that is regularly updated and maintained by clinical supervisors across the Central West Region. Each region independently completes their own spreadsheet, which includes quantitative information about program participants and program delivery. More specifically, the spreadsheet tracks program outputs by client (e.g., number of hours of service delivery, type of service, and region). Included within the ABA Roll-up are client results from the Goal Attainment Scale (GAS). GAS quantifies the achievement of goals set by parents and ABA consultants based on a 5-point rating scale from -2 to +2. If a chosen goal (for example, reduction in aggression) is recorded as -2 in the Roll-up, then the outcome level was much less than expected (e.g., there was a regression in the goal area). If the goal is recorded at +2, then the outcome was much greater than expected (e.g., the child greatly improved and the aggression was significantly reduced). A score of zero means that the expected level of goal attainment was reached but not surpassed (i.e., the child met but did not exceed their goal). This report includes data collected in the ABA Roll-up for most of Cycle Two, from April 1, 2014 until December 31, 2015. In total, outcomes from 658 clients aged 9 to 17 years (across all agencies) were tracked in the Roll-up during this time, and were included in the analysis. Consumer Feedback Survey. The Consumer Feedback Survey (CFS) is self-administered by parents/caregivers of children and youth participating in ABA programming. Upon completing a round of
  • 15. ABA Cycle Two Evaluation Report 15 | P a g e service, parents/caregivers are given a hardcopy of the CFS, and are asked to fill it out and return it in an envelope (in-person or by mail) to the program manager at their agency. Central intake staff at the various ASD agencies will then insert the individual CFS data into the online version (i.e., on Survey Monkey), where all CFS data from agencies in the ABA Central West Region is stored. The CFS quantitatively evaluates program outcomes in the domains of quality of life, behaviour management skills, and feelings of competence at the family level. The survey also contributes quantitative insight into program processes (i.e., what is working well and not working well), and provides suggestions for future directions via qualitative components (i.e., boxes for parents/caregivers to provide suggestions or comments). The CFS was revised in the fall of 2014, partially to include questions on the generalization and maintenance of ABA goals. In this evaluation cycle, survey responses of 423 parents/caregivers were analyzed, approximately 69% from Peel Region, 20% from Halton Region, and 11% from the Waterloo, Wellington, and Dufferin Regions. Central West Region (CWR) Applied Behavioural Analysis (ABA) Services- Intake Management. The CWR ABA Services Intake Management database (referred to as ‘intake database’ here) provides demographic information on all children/youth and their families that are currently enrolled in ABA services within the Central West Region. The information in the intake database provides a demographic snapshot of clients undergoing the program, in addition to total number of clients ever served and interventions delivered. The database provides details about how many clients are on the central waitlist vs. in service, and identifies the domain area(s) that families are working on during their sessions. Individual results of the Child and Adolescent Needs and Strengths (CANS) tool are also available within the intake database. Aggregate data from the intake database (for the period of April 1, 2014 to December 31, 2015) was analyzed during Cycle Two. Due to privacy concerns, ABA staff de-identified and collated all data prior to sending it to the external evaluating team. Data from the CANS portion of the intake was analyzed in Cycle One and will be re-visited and re-analyzed in Cycle Three. Time Utilization Sheets. The time utilization sheets were a tool developed out of recommendations from Cycle One. These time sheets help ABA consultants to track the number of hours per day that they spend on various tasks (e.g., direct client contact, client documentation, etc.). According to ABA consultants and other program staff, a number of these tasks were not previously tracked. In this report, data was analyzed over a three month period (October 2014 to December 2014), as these were the only months consistently tracked across all agencies. In total, data from 25 consultants was analyzed for this report: one from CLNH, seven from PBS, two from CMHA, two from ROCK, five from DSRC, and 14 from KPAS. Secondary Outcome Tool. The Secondary Outcome tool was developed out of recommendations from Cycle One, and was implemented starting in November 2014. This tool helps to determine: 1) the percentage of clients who exhibit positive outcomes other than the achievement of their main goal; 2) the extent to which these secondary outcomes are exhibited; and 3) the degree to which these outcomes occur in different environments (i.e., at home, school, or in public). Consistent with the ABA program goals, secondary outcomes are tracked by the four main domain areas. For the purposes of this evaluation report, 50 secondary outcome sheets were collected and analyzed from various agencies in the Central West Region.
  • 16. ABA Cycle Two Evaluation Report 16 | P a g e Qualitative Methods (Primary Data) Focus groups with parents/caregivers. A total of nine focus groups were conducted in Cycle Two with parents/caregivers (n=32) of ABA clients. Five were held via teleconference, and four were held in-person. Participants were selected and recruited in cooperation with program managers on the evaluation Steering Committee, and according to purposive sampling criteria (i.e., diversity in geographic representation across the Central West Region, length of involvement in ABA program, gender, ethnic diversity, range of diagnosis, range of child/youth ages, and range of models of services). All main research questions were addressed in these focus groups. Focus groups with staff. Three focus groups were held with ABA staff during Cycle Two, including one with clinical supervisors (n=5) and two with ABA consultants (n=19). Participants were selected and recruited in cooperation with program managers and according to purposive sampling criteria developed by the evaluation Steering Committee (i.e., representation across Central West Region, diversity in clinical and frontline staff, and diversity in length of involvement with program). All three main research questions were addressed to gain front-line insight on program processes and outcomes for child/youth and parent/caregivers, as well as recommendations for improvement (see Appendix 3). Key informant interviews. Twelve key informant interviews were held in Cycle Two via telephone, to gain insight into the three main research questions related to systems-level processes, outcomes, and recommendations for improvements. Participants were selected and recruited in cooperation with the Steering Committee and according to purposive sampling criteria. Participants were located across the Central West Region and included senior staff at ABA agencies, a Ministry of Child and Youth Services representative, school representatives, and senior staff at ABA support agencies. Case studies. Case studies were introduced as a new qualitative tool in Cycle Two to give further insight into some of the common strengths and challenges of the program, while showcasing the diversity in clients, families, and program experiences. Three clients were selected according to purposive sampling criteria (i.e., diversity in geographic representation across the Central West Region, length of involvement in the ABA program, gender and ethnic diversity, range of child/youth ages and diagnoses, and range of models of service). For each case study, the child’s parent or caregiver was interviewed, as well as their ABA consultant. When possible, the child/youth was also interviewed, or a family friend if the child was uninterested or unable. In total, eight interviews were conducted across the three case studies. EVALUATION FINDINGS The findings of the evaluation are organized according to the three main research questions related to program implementation, program outcomes, and future directions. Data was analyzed across all research methods with main themes highlighted below. Program Implementation Overall, evaluation participants expressed an appreciation for the ABA program, and were mostly satisfied with the service they delivered or received. While there remain significant challenges with program implementation, data revealed that families are “wanting the service, needing the service, [and] liking the
  • 17. ABA Cycle Two Evaluation Report 17 | P a g e service” (Key Informant). This section begins by describing the program overall (i.e., number of clients served, percentage of children in each domain), followed by a list of the top strengths and challenges revealed in Cycle 2. Similarities and differences to Cycle 1 are highlighted throughout. Description of ABA Participants The ABA program is available to all children/youth under the age of 18 years who have a confirmed ASD diagnosis (Autism/Autism Disorder, Pervasive Developmental Disorder, or Asperger’s Disorder). Confirmation of diagnosis must come from a certified and qualified professional (i.e., a family physician, pediatrician, psychiatrist, psychologist, or psychological associate) (Autism Ontario, 2015). Chart 1 shows a snapshot of the total number of clients (aged 9 to 17 years) in service on March 6, 2015, within all areas of the Central West Region. The majority of children who were in service were between the ages of 11 to 14 years. In all areas, the 15-17 year age bracket had the fewest number of clients in service. Chart 1 also demonstrates that Peel served the greatest number of clients during this time (135), accounting for 55% of all children/youth who were in service across the Region. This is due to the fact that Peel has a higher population than the other regions, and is thus allocated more of the funding and resources. Chart 1: Number of children/youth in service by age group and region (N=356) Source: CWR ABA Service- Intake Management (Snapshot of those in service on March 6, 2015) Chart 2 shows the total number of eligible children/youth in 2014 (ages 9-17 years) by service status and by service provider cumulative for April 1, 2014 to December 31, 2014. In this report, ‘eligible’ is defined as all children who: (a) are on the waitlist; (b) are in service; or (c) completed a block of service in the 2014 fiscal year. The total number of children served is a cumulative figure for the period of April 1 2014 to December 31, 2014. “Other Eligible Children” refers to any clients who are on the waitlist, or have completed ABA service. The majority of eligible children were clients in KPAS Peel (26%) and PBS (24%). 40 12 5 6 7 66 17 28 15 2 29 7 7 3 0 0 10 20 30 40 50 60 70 Peel Halton Waterloo Wellington Dufferin TotalNumerofChildren/Youthin Service Ages 9-10 Ages 11-14 Ages 15-17
  • 18. ABA Cycle Two Evaluation Report 18 | P a g e Chart 2: Total number of eligible clients (aged 9-17 years) by service status (served vs. other eligible children) and service provider (N=2264) Source: CWR ABA Service- Intake Management (cumulative for April 1, 2014 to December 31, 2014) Discussions with evaluation participants revealed that families represent a range of income levels. Given that ABA services are offered at no cost, low-income families can participate in the program. Discussions also revealed that ABA families are ethnically diverse. This is particularly true in the large urban areas such as Peel Region. Some families have recently immigrated and are still in the process of learning English. Some ABA consultants can speak multiple languages. Interpreters are also available, if needed, but at a cost to the program. As was noted in the Cycle One report, some key documents are not always translated (e.g. consent forms and individual plans). Description of Program As explained in the Cycle 1 report, children or youth can work on goals related to one of four domains: communication, social skills, daily living skills, and behaviour/emotional regulation skills. Chart 3 depicts the percentage of clients within each domain, from April 1, 2014 to January 21, 2015. Children/youth can only work within one domain per session. Chart 3 reveals that the majority of clients focused on either social skills (40%) or behavioural/emotional regulation skills (31%). Clearly, the majority of families feel that social skills and behavioural/emotional regulation are the most important domains to work on during their ABA program.
  • 19. ABA Cycle Two Evaluation Report 19 | P a g e Communication 12% Social Skills 40% Daily Living Skills 17% Behavioural / Emotional Regulation 31% 79 125 32 32 48 48 21 30 26 81 48 19 4 6 3 7 35 10 0 20 40 60 80 100 120 140 TotalNumberofChildren/Youth Service Providers Individual Group Chart 3: Skill domains in ABA services for children/youth aged 9-17 years (N=256) Source: CWR ABA Services- Intake Management (CANS April 1, 2014-January 21, 2015) ABA is offered as an individual, one-on-one program or as a group program with other children/youth of a similar age, level of functioning, and specified goal area. Chart 4 reveals the total number of clients per ASD agency who participated in each type of service during Cycle Two. Clearly, most agencies served a greater number of clients through one-on-one sessions, with the exception of PBS and DSRC who served slightly more clients through their group sessions. Chart 4: Form of service delivery by service provider for children/youth aged 9-17 (N=662) Source: ABA Roll-up (April 1, 2014 – December 31, 2014) Chart 5 demonstrates the total number of hours spent on client intervention (i.e., directly with the child/youth) versus parent training from April 1, 2014 until December 31, 2014. Definitions of client intervention hours and parent training hours are provided in Appendix 4. All agencies spent more time working with clients as opposed to training parents or caregivers. The average amount of time that consultants spent with each client was between 9 and 18 hours per round. The average amount of time that consultants spent training parents or caregivers was between four and 15 hours per round. In
  • 20. ABA Cycle Two Evaluation Report 20 | P a g e comparison to all other service providers, KPAS Peel spent the most amount of time, on average, with clients (approximately 18 hours/round), while CLNH spent the most amount of time, on average, training parents or caregivers (approximately 15 hours/round). CLNH spent an almost equivalent amount of time on both, as opposed to agencies such as KPAS Peel, DSRC and CMHA who spent much more time on client intervention. It must also be noted that some clients, especially those with more complex cases (e.g., a dual diagnosis) may be in service for up to six months, if a supervisor’s clinical judgment dictates that the client cannot be released. This, according to many key informants, can contribute to the challenges outlined later in this report (e.g., growing waitlist, staff burnout, etc.). Chart 5: Amount of time spent on client intervention and parent/caregiver training by service provider for children/youth aged 9-17 years (N=621) Source: ABA Roll-Up (April 1, 2014 – December 31, 2014) Chart 6 gives a more fulsome picture of how ABA consultants spend their time, not only with clients and their parents/caregivers, but on paperwork, training and other duties and responsibilities. The tool to collect this data (‘Time Utilization Sheet’) was implemented for the first time in Cycle Two; it is filled out monthly by ABA consultants. Chart 6 demonstrates the average time spent on each task type, from October 2014 to December 2014. The chart clearly shows that consultants spend the greatest percentage of their time on documentation (~26%) followed by direct client contact (~23%) and administration (~12%). Definitions of each type of task are provided in Appendix 5. 15.4 17.7 14.5 9.4 15.0 16.9 15.9 13.9 12.7 12.9 4.9 8.7 5.0 11.3 12.9 15.0 4.1 4.8 0.0 2.0 4.0 6.0 8.0 10.0 12.0 14.0 16.0 18.0 20.0 Averageamountoftimespentoninterventionand training(perclient) Service Providers Client Intervention Hours Parent Training Hours
  • 21. ABA Cycle Two Evaluation Report 21 | P a g e Chart 6: Total Time Spent by Task Type for October 2014 – December 2014 Source: Time Utilization Sheets, October 2014 – December 2014 Strengths of Program Implementation Evaluation participants highlighted several strengths in regards to ABA program implementation, many of which were similar to those revealed in Cycle One. Overall, parents and caregivers were grateful for the program and appreciated the various components involved in service planning and delivery. “This ABA program is like gold. It is an untapped hidden resource that is extremely effective.” (Parent) “The tracking was good. The feedback was good. The tools were excellent. Knowledge of the child and an understanding of the child’s needs was excellent. It worked.” (Parent) Chart 7 further depicts parents’/caregivers’ satisfaction through their response to the statement “Overall, I found the program helpful.” 93% of all responders agreed or strongly agreed. Direct Client Contact 23.0% Indirect Client Contact 5.6% Client Documentation 25.9% Administration 12.3% Case Review 4.8% Supervision 1.9% Committee Meeting & Relations 1.3% Staff Education 2.9% Student Education 0.3% Community Education 0.5% Other 3.2% Travel 8.9% Sick 1.9% Vacation 7.6%
  • 22. ABA Cycle Two Evaluation Report 22 | P a g e Chart 7: Overall, I found the program helpful (N=414) Source: Consumer Feedback Survey (April 1, 2014 to January 21, 2015) Data from Cycle Two revealed five key strengths of program implementation. These included: (1) the consistency in program planning (e.g., the matching of consultants with past clients); (2) the choice of various programming options (e.g., individual vs. groups sessions, at-home vs. agency settings); (3) the opportunity to work with highly-skilled consultants; (4) the ability to focus families on manageable change; and (5) the partnerships among ASD agencies. These strengths are further explored below. Consistency in program planning Several parents/caregivers and frontline consultants appreciated the program’s consistency in terms of client-consultant matching as well as grouping for children in social skills groups. Such consistencies in service led to greater efficiencies in programming and higher rates of success. For example, when clients are matched with the same consultant as in previous sessions, service delivery is strengthened due to existing relationships and trust. In addition, the consultant may be better able to tailor the program to meet the needs of the client, if they are already aware of their strengths and needs from previous sessions. Consultants can also help their client to build upon skills that they had worked on previously. “Having the same person be with your child on an ongoing, more fluid basis… makes all the difference.” (Parent) For similar reasons, parents and caregivers appreciated when their children were matched with the same children from previous social skills groups. Bringing the same children together (when the children are well matched) can improve the quality of their service and help the children to develop stronger relationships with each other. According to several evaluation participants, having a consistent program also means that parents and children/youth can expect a similar structure each time they come off the waitlist. Children are more aware of the expectations of the program, and understand what will happen during the sessions. Children can also build on skills from previous sessions, and continue learning from where they left off. Strongly Agree 70% Agree 23% Neutral 6% Disagree 1% Strongly Disagree 0%
  • 23. ABA Cycle Two Evaluation Report 23 | P a g e “We see more compliance [in subsequent sessions]… The kids are understanding what the expectations coming back in are.” (Key Informant) “I also feel that the families who are coming back in for 2nd and 3rd and 4th round of service are taking the option to build on the skills that have already been learned… We have some clients that come through and only work on communication, and that’s because they’re building on what they have previously learned.” (Key Informant) Choice in programming options According to many evaluation participants, another strength of program implementation is the choice in programming options for families. Parents/caregivers appreciated the option of one-on-one or group sessions, as well as the location of service (in the home, ABA agency, or community). For many children, depending on their goal, individual sessions work best; they can work directly with a consultant to focus on a goal specific to their needs. Some of these sessions occur in the family home, which many parents and caregivers considered an effective approach for working towards change. Skills can often be more effectively taught, learned and applied in the natural home setting, where many behaviours occur. In addition, home-based sessions are sometimes more convenient for parents who are balancing multiple responsibilities and may have other children to care for. Some clients also prefer to keep their diagnosis to themselves, especially those in their teenage years, and thus prefer at-home sessions where they won’t be seen with their consultant. The option of having ABA sessions in an agency setting is also appreciated by parents and caregivers. Children are sometimes less distracted in a more formal setting, and can work in a room where they won’t be distracted by the goings-on in a home. Group sessions also happen most often in the agency setting; for some clients and some goals, group sessions are the most effective as they provide ample opportunity for children to interact with others and to develop social skills in a safe and supportive environment. Evaluation participants appreciated the large and growing number of agency settings; having a greater choice in where the program can occur makes it more accessible for families in the region. “It’s really good that families are given a choice between in-home and group supports.” (Key Informant) “Agencies continue to open up centres to make the services more accessible…in an effort to make the services and spaces more accessible to children with ASD.” (Key Informant) Another option for ABA programming emerged last summer, as several agencies began offering ABA summer groups that typically lasted one week (approximately 30 hours). Families who came off the waitlist during these months could choose to attend a summer group, rather than have the standard eight-week session. These groups were often situated in or involved field trips to natural environments (malls, bowling alleys, etc.). In general, parents/caregivers found that their children benefitted from having sessions in natural environments, whether in the context of the summer group or during regular social skills groups. ABA consultants generally agreed that despite the challenges (e.g., logistics, finance, safety), using natural environments contributed to the generalization and maintenance of goals.
  • 24. ABA Cycle Two Evaluation Report 24 | P a g e 0 50 100 150 200 250 300 350 Strongly Agree Agree Neutral Disagree Strongly Disagree 325 76 12 0 1 NumberofRespondents Skilled consultants As was found in Cycle 1, the majority of key informants, clinical supervisors, and parents/caregivers in this evaluation expressed gratitude for their highly skilled and extremely dedicated consultants. The consultants showed creativity in how they tailored their tools and their approach to meet the needs of individual clients. Consultants were a valued resource for families who felt they lacked the knowledge and tools to help their children/youth on their own. One family, for example, mentioned how they only came to understand the root of their child’s behaviour with the help of their consultant. A number of families felt that their consultants went above and beyond to support them, and provided guidance on non-ABA related matters as well (e.g., referrals to other services). “The caliber of therapists is outstanding.” (Parent) “I think our consultants… they are very skilled in the services they deliver, and I don’t think they shy away from difficult situations.” (Key Informant) Chart 8 illustrates one aspect of parents’ and caregivers’ satisfaction with their ABA consultant: how well they understood the information presented to them. The chart shows that almost all parents and caregivers believed that the information presented by the consultant was done so in a clear, easy-to- understand way. Chart 8: The information presented by the therapist was presented in a clear, easy to understand manner (N=417) Source: Consumer Feedback Survey (April 1, 2014 to January 21, 2015) As with Cycle 1, the ABA consultants reported a strong commitment to the program both professionally and personally, and desired to see it succeed. In general, they appreciate opportunities to advance their skills (training workshops, conferences, etc.), although they have limited time to pursue capacity-building opportunities.
  • 25. ABA Cycle Two Evaluation Report 25 | P a g e Focusing families on manageable change Another strength of the program is that it helps to focus families on manageable change, despite the often overwhelming challenges they face in caring for their children/youth with ASD. To begin, most key informants and program staff as well as a few parents/caregivers discussed the benefits of focusing on one goal during each round of ABA. As they explained, concentrating on one simple goal in the allotted two to six month session can increase the probability of success for clients and parents/caregivers. This program strength was seen to be a key factor in leading to many of the client and parent/caregiver outcomes, as outlined in the outcomes section below. “It’s one thing at a time… Just having that one goal takes away that overwhelming piece for parents and it… it helps them kind of work on one thing.” (ABA Consultant) “The beauty of this program is that it gets down to the small goals that are really important to identify and work on and master.” (Key Informant) “When [families] focus on one specific thing, it does build resiliency.” (Key Informant) Despite the benefits of focusing on one goal, many parents/caregivers sometimes felt limited by this approach. As they explained, their children face numerous interconnected challenges, many of which cannot be resolved by focusing on one small goal. A few parents talked about the need for a more well- rounded program which considers small goals while focusing on a bigger picture of change. Choosing a specific goal was described as difficult by many parents/caregivers, especially those who are new to the program and had not narrowed down their child’s needs before. In these cases, many parents felt overwhelmed by the process and said they lacked direction in helping to choose their child’s goal. Yet overall, the focus on manageable change (that is, doable given the program timeframe) seemed to be working. Another facilitating factor was how consultants worked with parents and caregivers to teach them skills they will need to maintain or generalize the client’s goal. By learning skills relevant to their child’s behaviour, they were better able to work towards manageable change. As demonstrated in Chart 9, the majority of parents/caregivers found that the training in skills/behavioural techniques helped them with their child’s goals (87% agreed or strongly agreed). Only a small number of parents/caregivers did not find the training useful (2%).
  • 26. ABA Cycle Two Evaluation Report 26 | P a g e Chart 9: The training in skills/behaviour techniques helped me with my goals (N=416) Source: Consumer Feedback Survey (April 1, 2014 to January 21, 2015) Furthermore, as demonstrated in Chart 10, 87% of parents/caregivers found that they were able to carry out the strategies/techniques of the behaviour support plan (54% strongly agreed and 33% agreed). Chart 10: I was able to carry out the procedures/strategies of the plan (N=414) Source: Consumer Feedback Survey (April 1, 2014 to January 21, 2015) Partnerships among ASD agencies Finally, partnerships among ASD agencies emerged as a key strength in Cycle 2. According to ABA consultants, clinical supervisors and several key informants, the agencies involved in the delivery of ASD services work well together. Each Central West region formed committees of ABA and/or ASD-related agencies and institutions that meet semi-regularly to share insights and best practices, to address service gaps for ASD children/youth, and to strengthen regional coordination. In a few regions, the ABA program Strongly Agree 57% Agree 30% Neutral 11% Disagree 2% Strongly Agree 54% Agree 33% Neutral 12% Disagree 1%
  • 27. ABA Cycle Two Evaluation Report 27 | P a g e was said to foster stronger relationships between agencies due to their increased interactions. Agencies had a clearer sense of each other’s services, thereby making accurate referrals to each other’s programs with greater regularity. To a lesser extent, there was also some evidence of effective coordination with mental health agencies, and some examples of strong partnerships with schools and school boards. “We can overall serve people better because we know more about these agencies.” (Key Informant) “Working as a community is so important… We would never be able to deliver this program as one agency.” (Key Informant) Clinical supervisors from various agencies also meet on a regular basis, as do managers; these meetings were considered a valuable space for knowledge exchange. Given that the program is implemented differently across the Central West Region, clinical supervisors appreciated the opportunity to learn from best practices employed in other agencies, and to brainstorm solutions to unique program challenges in their jurisdiction. “I found the clinical advisory committee meetings really wonderful… I feel like I still have a team. I think we work nicely together.” (Clinical Supervisor) Challenges of Program Implementation Despite the large number of strengths associated with ABA program implementation, evaluation participants also identified a number of challenges, many of which were highlighted in Cycle 1. The six most significant challenges that emerged from Cycle 2 are as follows: (1) unsustainable targets (leading to staff burnout); (2) limited program dosage (due to long wait lists and short program lengths); (3) lack of other supports for parents (during ABA and while on the waitlist); (4) underdeveloped partnerships with schools and mental health agencies (leading to difficulties in achieving and maintaining goals); and (5) limitations with social skills groups (in terms of matching clients and parental involvement). Unsustainable targets The number one challenge with program implementation is undoubtedly the high number of children (i.e., targets) that consultants are expected to support each ABA cycle. As various consultants explained, the amount of work required to provide high quality service to their assigned number of clients is unsustainable and leads to burnout among staff. Due to their heavy workload, consultants are also unable to provide flexibility in service that might benefit some clients. For example, even if children/youth would benefit from having additional sessions with their consultant, consultants cannot often provide this support due to time constraints. Similarly, if ABA sessions are missed due to illness or vacation, ABA consultants are unable to make these up. This inflexibility has led to frustration among several evaluation participants. For clients who need more time in service due to complex needs and/or a complex diagnosis, consultants will extend the program. However, this adds additional pressure to their workload as there is little flexibility in service targets to account for these circumstances. In other words, consultants are still expected to reach their targets, even if some clients present particularly challenging cases and take more time, energy and resources.
  • 28. ABA Cycle Two Evaluation Report 28 | P a g e “I have great staff who are dedicated to this program and who are very determined to provide top quality clinical service, but they are killing themselves to do it.” (Key Informant) “In this region, we are implementing the most clinically sound program we can. However we are stretching our staff extremely thin in order to do so.” (Clinical Supervisor) Part of the issue regarding consultants’ schedules is the amount of time spent on documentation. As shown in Chart 6, ABA consultants spend more time on documentation than they do on direct client contact. Moreover, some of this documentation is considered not very useful, by both ABA consultants and parents/caregivers. The CANS tool, for example, was described by consultants as overly detailed (i.e., takes a long time to complete), yet not very helpful as the ratings are overly subjective and do not contribute to ABA service delivery. “The CANS has got to go… It’s not helpful for our type of service delivery… It’s very subjective and not relevant at all, and it’s extra work; a formality. It just gets filled out and stuck in their folder.” (ABA Consultant) Another component of the program that takes a significant amount of time is travel. Time spent traveling to meet clients was substantial, at an average of 8.9% of their total working hours. These and other aspects are often not reported to the Ministry, due to standardized reporting mechanisms. According to ABA consultants, this partial reporting gives an erroneous perception of the work required to deliver ABA services, and could perpetuate unrealistic ministry service targets. In sum, ABA consultants reported being overworked and seemed doubtful that they could manage their caseloads long-term. Seeing as the program’s success has been found to partially hinge upon consistency in consultants, there was widespread concern among evaluation participants that the ambitiousness of ministry targets would jeopardize the quality of service delivery for ABA clients. Limited program dosage As found in Cycle One, another significant challenge with program implementation is the length of the waitlist, both at initial intake and between ABA sessions. A growing number of families are participating in the ABA program, which is a positive indication of community demand. However, because program resources have not increased to meet the growing demand, many families are waiting up to a year and a half for service. Chart 11 shows the number of clients on the waitlist vs. in service by agency. Clearly, there are significantly more children on the waitlist in all agencies across the Central West Region.
  • 29. ABA Cycle Two Evaluation Report 29 | P a g e Chart 11: Number of clients (aged 9-17 years) on waitlist and in service by service provider on December 31, 2014 (N=1612) Source: CWR ABA Services- Intake Management (Q4 April 1, 2014-January 21, 2015) The waitlist was considered discouraging by various evaluation participants, especially because the program itself is often only two to three months, and very few resources are available to families when on the waitlist. “The wait times…are discouraging. We need treatment for our kids so badly.” (Parent) “The wait times are our challenge, every day.” (Key Informant) With such long wait times, parents/caregivers reported having high expectations for the program, yet felt that only limited gains were possible within such a short period of time. Since the first few weeks of the program are dedicated to service planning and relationship building between the child and their consultant, actual treatment time was seen as insufficient to have any significant impact on families or to maintain and generalize goals. Parents also felt limited in the types of goals they could choose, as they were not able to choose goals that take more time. “I feel like we don’t spend enough time in getting them to generalize the skills across environments.” (Key Informant) Parents and caregivers of high-functioning children were especially frustrated with the short length of the ABA program, as they are ineligible for most other ASD services and rely solely on ABA for support. Similarly, parents and caregivers of children aging out of the system expect more from the program while 158 94 87 65 315 144 113 107 373 15 14 17 9 66 5 7 4 19 0 50 100 150 200 250 300 350 400 TotalNumberofChildren/Youth Service Provider On Waitlist In Service
  • 30. ABA Cycle Two Evaluation Report 30 | P a g e the service is still available to them. Longer gaps between program cycles can also impede upon program outcomes for re-referred clients, who struggle to maintain ABA goals over the long term without professional support. “The only problem is, my son doesn’t qualify for anything else. He never got IBI or anything, so this is all he gets… So you get the eight weeks, and then wait almost a year until it happens again. And he needs something on an ongoing basis, and there is nothing for him.” (Parent) One type of support available to families after completing an ABA cycle is booster sessions. About a month after completing service, consultants will attempt to contact their families to ask about goal maintenance. If parents/caregivers feel that their child needs a booster session, the consultant will attempt to arrange it. While some parents/caregivers reported receiving such a service, many others reported never being contacted after the completion of a cycle. As they explained, in-person booster sessions rarely occurred, and were in many cases ineffective as parents required more regular follow-up and support to maintain goals. They generally felt positive about the idea of booster sessions, but said that in reality, they are few and far between. From the perspective of ABA consultants, many reported difficulties with the follow-up process, saying that families are difficult to reach or had limited availability for in-person booster sessions. It was also stressful for consultants to schedule these sessions whilst starting programs with new clients “Lots of times it’s a struggle to get that [booster sessions] worked in, for a variety of reasons. I mean the consultants have already moved on to their next group of children and youth that they’re working with.” (Key Informant) For consultants that did conduct follow-up over the phone, most had a positive response from families and reported that goals were successfully maintained. Even still, consultants questioned whether families were being honest or just appeasing them with positive feedback. Limitations in social skills groups Similar to Cycle 1, a final key challenge of program implementation was the difficulty around developing relevant and meaningful social skills groups. While ABA staff considered the first-come-first-serve nature of the wait list to be fair in many respects, it also creates challenges in forming social skill groups with well- matched clients (i.e., similar diagnosis, gender, behavioural challenges, and program goals). In some areas of the Central West Region, the policy was recently changed so that clients could be selected up to six months down the wait list for social skills groups. With the wave of new clients entering the program, however, some consultants reported that they were not familiar enough with clients to match them effectively.
  • 31. ABA Cycle Two Evaluation Report 31 | P a g e Case-in-Point: Rishi’s Case Study Unfortunately, Aarav has little understanding of what occurred during his son Rishi’s first ABA session, as the program did not facilitate parent observation. Rather, he waited in the agency’s lobby until each session was complete. While parents are now encouraged to observe some of their children’s groups, Aarav still feels as though “when you [the parent] are left out, all of the added value that you bring to the session is gone”. In the end, Aarav explained: “I’m not sure how successful that was... I was literally shut off, so I didn’t have much idea of what was going on.... Did I gain anything? Did [Rishi] gain anything? No idea, whatsoever”. Even though Aarav was provided the reports at the end of ABA, he still found them difficult to read. As he said, “I couldn’t tell if there was any perceptible gain or even if he achieved the goal”.  To read Rishi’s full case study, see Appendix 1. “This [creating social skills groups] has been challenging lately because newer families are coming onto the list. So you don’t know that client and you don’t know that family and so it’s very hard to pick them out and pair them appropriately.” (ABA Consultant) Forming well matched girls-only groups was reported as especially difficult, as there are significantly fewer female clients to select from. As well, due to the large geographic area in places like Halton, it was difficult to form groups with children living in the same area. According to several evaluation participants, social skills groups provided few opportunities for parents to observe and participate. This limited parents’ understanding of the program’s process and outcomes, as well as their ability to help maintain and generalize skills. “I find because we are heavy in our group supports, the parents’ participation is limited, which impacts generalization and even just their capacity to understand the program and see that, you know, my child is actually making gains based on the goal that was set.” (Key Informant) Finally, a small number of families were dissatisfied with the lack of confidentiality in how feedback regarding their child/youth was provided at agencies during social group sessions, as other families and staff were often present in the room and could overhear the conversation. Lack of Other Supports Another challenge reported by most parents/caregivers as well as some key informants was the lack of supports for families outside of the short program sessions. While a select few received resources while on the ABA waitlist, the majority of parents and caregivers felt as though they were “off the radar” (parent) during this time. Throughout the focus groups, they consistently expressed frustration regarding their feelings of isolation and lack of support. Program managers and other key informants agreed that there’s not much available for families while they wait a year or more for ABA. Parent training and workshops have become increasingly available since the start of this evaluation, yet are not offered in all areas of the Region, nor are they actively attended. Many parents and caregivers cannot take the time away from their families to attend a workshop in person.
  • 32. ABA Cycle Two Evaluation Report 32 | P a g e Case-in-Point: Emily’s Case Study As a single dad, John relies on services like these to help support him with his daughter Emily’s needs. He stated, “the waitlist is horrible. I applied when [Emily] was 11, and she is now 14 and only getting her second [cycle].” Both John and Lisa [the ABA consultant] acknowledged there is a need to have supports for parents during this time and John recalls being able to contact Lisa outside of the program when he needed her help or advice. “I did have a couple of challenges after [Emily] was discharged. And it was kind of, ‘okay, now what do I do?’ Luckily, I had [Lisa’s] phone number, so I called her up and said ‘hey’. And she was very helpful, but it wasn’t something that was covered by the program. It was something she did on her own time. And I didn’t think that was fair to her, and I almost felt guilty doing that to her. But she was extremely helpful.” (John)  To read Emily’s full case study, see Appendix 1. “I know that there were some services that were tried… to try and have them [parents] prepared for when they’re coming into service, but I’m not sure. My recollection is that they weren’t well attended, and they weren’t accessed by families as they were waiting.” (Key Informant) “Unfortunately when you have a kid with, you know, special needs and stuff… an hour is gold. You know? Like your time is really precious, I find anyway.” (Parent) Two examples of types of supports that parents and caregivers desired are: (1) opportunities to talk to other parents and share insight and support with others facing similar challenges; and (2) opportunities to talk to professional ASD consultants outside of the ABA setting. Both of these supports would help parents to feel more capable of handling day-to-day challenges, and less overwhelmed by the ABA wait times. One resource available to parents is Service Coordination. However, while most families have access to a Service Coordinator in their region, many do not access the service and others find it unhelpful. As some parents explained, Service Coordinators will answer specific questions related to service, but cannot always spend time explaining the different options available, if the family is not clear in what they’re looking for. This, in part, is due to the tremendous case load that each Service Coordinator is given. As one key informant explained, some Service Coordinators have over 200 clients that they serve. As a result, Service Coordinators end up largely helping families with complex cases or crisis situations, rather than those who need help with navigation or understanding services. Consequently, ABA consultants often take on a service coordination role for their clients. “They [Service Coordinators] ask you, what do you need? I don’t know what I need… [So] I gave up on them.” (Parent) Finally, a number of parents/caregivers reported that they could not afford to purchase resources that were recommended during service delivery (e.g. toys, books, puzzles). While some supplies were made available at agency libraries, these were mostly adult resources. Libraries with client-friendly resources were desired by parents and caregivers. “The financial aspects of autism are equally as horrible as the problem with autism. Seeing people put third mortgages on their house or cashing their RRSPs or working three jobs or whatever it takes to get the money, it’s just a terrible thing.” (Parent)
  • 33. ABA Cycle Two Evaluation Report 33 | P a g e Underdeveloped partnerships with schools and mental health workers Another challenge with program implementation is underdeveloped partnerships in many regions, both with schools and school boards as well as various mental health agencies. This is an important challenge as many goals cannot be attained or generalized without the cooperation of these agencies or institutions. According to evaluation participants, schools across the Central West Region vary in the degree to which they work with ABA consultants. For example, communication between school boards and ABA consultants has been fairly effective in regions such as Wellington and areas of Peel. Some successful stories were shared by evaluation participants, in which schools supported the attainment of a goal. In cases where a child’s goal was related to behaviours at school, and schools were receptive to the ABA program, there was often greater success in goal attainment, maintenance, and generalization. In regions where these partnerships are less developed, however, consultants are finding that they must work with schools on a case-by-case basis. A reluctance and even resistance to the ABA program was noted by some evaluation participants at the school, school board and even union level. “I stopped getting the school involved. Why bother? They will just say no.” (Parent) “Hit and miss, I think. Some schools are willing to have conversations and some are not. Some are willing to let you come in and observe, and some are like ‘we’re fine’.” (ABA Consultant) According to evaluation participants, some principals, ASD itinerants, and special education teachers see the value in connecting with ABA consultants, whereas others see it as an entrenchment on their own treatment models and service. For example, a key informant from the school board explained that schools in his region are generally willing to entertain conversations about collaborating with ABA consultants, and are receptive to sharing knowledge and learning from each other. “There has to be a respect for our [schools’] expertise and our knowledge. We do have behavioural services, and I don’t need somebody to tell me how to do behavioural programs.” (Key Informant) “I would say they’re receptive to it as a whole, in terms of supporting the need for ABA. In terms of whether a worker can go in and do some of that implementation of ABA in the school or in daycares or those settings, I’ve heard mixed messages.” (Key Informant) ABA consultants and a few key informants noted that schools seemed to be more open to working together when the ABA goal was behavioural in nature, especially in cases when the school faced difficulty managing the behaviour of the child/youth. Schools were also more receptive when families were actively working to bring the school representatives and ABA staff together. ABA consultants and key informants from some regions also discussed the issue of underdeveloped partnerships with mental health agencies and workers. Oftentimes, children/youth with both ASD and a mental health issue face many unique challenges that cannot be resolved through the treatment approach of the ABA program. ABA consultants do not have the abilities or skills to tackle a mental health concern, which often takes precedence over the one goal of ABA. Yet in many regions, mental health workers and ABA consultants have not been able to work collaboratively to address these challenges. A few consultants reported that mental health workers sometimes overlook the mental health diagnosis and see it as a side effect of ASD, and thereby assume that ABA treatment is sufficient for dual diagnosis clients. Without
  • 34. ABA Cycle Two Evaluation Report 34 | P a g e proper treatment, the client may ultimately end up in a crisis situation, possibly putting themselves and/or their ABA consultant in danger. Thus, better partnerships with mental health agencies need to be developed in many regions, so they can work together to best serve the child or youth in need. Limited response to age transitions Another key challenge of ABA program implementation revolved around age transitions, for children switching agencies (i.e., at the age of 9 years), for youth entering high school, and for youth aging out of the system (i.e., at the age of 17 years). Overall, there is an immediate need to make these transitions more seamless, and to provide more support for families at these different stages of life. A number of key informants reported inefficiencies in the transition for clients moving from ABA delivery agencies serving 0-8 year olds to those serving clients between the ages of 9 to 17. For the most part, minimal or no information is passed between agencies, which leads to confusion for families who felt that the receiving agencies had no sense of their child/youth’s story. Staff and key informants also reported frustration with this process, stating that consistent reports should be passed between agencies in order to better serve clients. In response to these challenges, ABA delivery agencies have begun to discuss how this transition could be improved. “We don’t want parents to have to retell their stories every time… And they have to.” (Key Informant) Secondly, youth who are transitioning into high school also face unique challenges, and parents/caregivers talked about the need for support from ABA during this time. For example, some parents said that their children did really well in structured elementary school settings, but struggled upon entering high school as they did not receive the same degree of direction when presented with recreational or free time. As well, some parents wanted to talk to their children about issues such as puberty and romantic/sexual relationships, but their children were uninterested or unable to speak about these topics with them. Parents felt that the ABA program should be more proactive in guiding youth with ASD diagnoses through these issues, as they may have different or unanticipated challenges than other children/youth. “Going into the teenage years, it’s hard enough for a ‘normal’ kid to go through. But then [ASD] kids who are struggling with the social – and social becomes uber huge – then it becomes even more important to have these supports as they get older. It’s like [helping them] deal with hormones, or opposite gender, or same gender if they’re struggling with that.” (Parent) Thirdly, youth aging out of the ASD system struggle with this transition. ABA is only offered to youth up until their 18th birthday, after which there are few or no ASD services to support them. Various evaluation participants expressed the limitations of this design, saying that many youth over the age of 17 still desperately need services.
  • 35. ABA Cycle Two Evaluation Report 35 | P a g e Program Outcomes As was found in Cycle 1, many of the anticipated outcomes that are outlined in the program logic model (page 10) were in fact achieved. Most of the children and youth within the program reached their short- term goals, and in this year’s evaluation we also found evidence that many experienced secondary outcomes as a result of participating in the program. Family outcomes were not quite as positive; while parents gained skills relevant to their child’s goal and often felt more confident in working with their child, many still felt relatively unsupported outside of ABA. Thirdly, while some system-level outcomes were positively reported, most evaluation participants still reported this to be the weakest outcome area in terms of partnering with other agencies/institutions (schools, etc.), providing a stronger continuum of support, and creating greater awareness of the ABA program itself. This outcome section is structured similarly to the Cycle 1 report, with outcomes outlined according to the three major outcome areas as shown in the program logic model: client (individual child/youth) outcomes, individual family outcomes, and system-level outcomes. Client Outcomes Almost all children in the ABA program reached or exceeded their goal. Chart 12 shows the percentage of clients (aged 9-17 years) across all agencies that attained each of the five levels of outcomes from the Goal Attainment Scale (GAS). The two most common GAS outcomes were +2 (outcome much more than expected; 37%) and +1 (outcome more than expected; 37%). In total, 97% of all clients (aged 9-17 years) who completed the program between April 1, 2014 and December 31, 2014 reached or exceeded their expected outcome. Chart 12: Goal Attainment Scale (GAS) scores for children/youth aged 9-17 years (N=619) Source: ABA Roll-Up (April 1, 2014 – December 31, 2014) Chart 13 shows GAS scores by service provider. All regions had higher numbers of clients meeting or exceeding goals than clients with unsatisfactory results. The most common GAS score across all regions was +1 (six out of nine agencies). While an average of +1 might contribute to the perception that ABA
  • 36. ABA Cycle Two Evaluation Report 36 | P a g e programming is very successful, a few clinical supervisors explained that such a high score should not be the norm. If the average score is +1, with a large number of children achieving +2, then programs have the potential to be a lot more challenging and to push children to achieve more difficult goals. As they explained, the average score should be zero (expected outcome), with +1 and +2 being less common. Chart 13: Goal Attainment Scale score by service provider (N=619) Source: ABA Roll-Up (April 1, 2014 – December 31, 2014) In speaking with evaluation participants, most agreed that children and youth were achieving or exceeding their goal(s). This was reported to be consistent across domain areas and most diagnoses. “There’s an overwhelming amount of children that are meeting goals and making progress in their skill development and their reduction of the maladapted behaviours.” (Key informant) “The behaviours have lessened significantly. It’s incredible.” (Parent) Chart 14 provides additional insight into parents’ and caregivers’ perspectives on client outcomes, as reported in the Consumer Feedback Survey. As shown in this chart, the majority of parents (72%) either agreed or strongly agreed that ABA goals were met. 0 10 20 30 40 50 60 70 80 90 100 110 TotalNumberofChildren(age9-17) -2 Outcome much less than expected -1 Outcome less than expected 0 Expected outcome reached +1 Outcome more than expected +2 Outcome much more than expected
  • 37. ABA Cycle Two Evaluation Report 37 | P a g e Chart 14: In my opinion, my son/daughter met his/her goal (N=415) Source: Consumer Feedback Survey (April 1, 2014 to January 21, 2015) Interestingly, 21% of parents/caregivers who completed the Consumer Feedback Survey felt “neutral” and 7% of parents/caregivers either disagreed or strongly disagreed that their child’s goal was met. These numbers are higher than the 3% of clients who did not meet their goal according to the GAS. This discrepancy demonstrates that parental or caregiver perceptions of ‘success’ (whether or not a goal was achieved) appears to sometimes differ from the results of the GAS completed by staff. While the questions were worded somewhat differently in these two quantitative measures, this apparent (and occasional) discrepancy between GAS scores and parent/caregiver perspectives was further mirrored in several focus groups. While most parents acknowledged that their child “met their goal” according to the GAS, some still felt their child had a long way to go in order to maintain or generalize it. Some ABA consultants had similar views, stating that the GAS scores do not always reflect what they would define as success for the client. Chart 15 provides additional information on parent/caregiver perspectives. According to this chart, 83% of parents agreed or strongly agreed that there was a positive change in the skills and/or behaviours addressed in the ABA program. Thus, in comparison to data from Chart 14, more parents/caregivers (83%) acknowledged a positive change as opposed to an achieved goal (72%).
  • 38. ABA Cycle Two Evaluation Report 38 | P a g e Chart 15: There has been a positive change in skills/behaviours addressed (N= 414) Source: Consumer Feedback Survey (April 1, 2014 to January 21, 2015) Some parents directly commented on their child’s ability to maintain or generalize the goal, post ABA. However, because follow-up phone calls and in-person booster sessions are difficult to schedule, much of this data is not tracked within program documentation. “For the first ABA, he met his goals at home. But it had a ripple effect because he was less anxious…He was also able to carry that forward into other areas.” (Parent) Some parents/caregivers appreciated that after the program, their children exhibited fewer negative behaviours (hitting, scratching, etc.) in relation to the goal or skill addressed. If the child, for example, learned how to better communicate within the ABA program, than they might feel less frustrated when interacting with people and exhibit fewer negative behaviours as a result. Parents and other evaluation participants appreciated the preventative rather than punitive approach of ABA; ABA helps families to reduce the incidence of negative behaviours, rather than react to these behaviours after they happen. “I think it’s the one program that actually does hands-on work with families on how to approach a problem… I think a lot of times families [receive] services that are consultative in nature, and I think this is a hand-on, step-by-step approach to why this is happening, and what can be done to change it.” (Key Informant) Several evaluation participants reported heightened levels of self-confidence and independence in their children following the ABA program. Some parents/caregivers reported that their child felt less stigmatized and more “normal” when they learned to do things independently and to interact with others. According to parents/caregivers and staff, it was very rewarding for the clients to see themselves achieve goals, and to recognize their strengths. “For him, it will be confidence for life.” (Parent)
  • 39. ABA Cycle Two Evaluation Report 39 | P a g e Case-in-Point: Eric’s Case Study By the end of the program, Eric had achieved his first goal. He learned how to follow a recipe and became confident in his abilities to prepare different foods and to work with another person – Rachel [his ABA consultant]. Eric appreciated that Rachel “…was very enthusiastic”; his favourite part was “that I had someone to help me with working on some areas that I wasn’t necessarily good at”. Rachel equally enjoyed working with Eric. As she explained: “[Eric] warmed up quite quickly, just doing something that he was enjoying. For [Eric], by the end, he was feeling very confident and very proud of himself that he could do these things he wasn’t able to do before.” (Rachel)  To read Eric’s full case study, see Appendix 1. “To interact normally with others in society, this is a huge skill.” (Parent) Parents and caregivers consistently expressed positive outcomes when their child had home-based sessions. On the other hand, perceptions of outcomes from the social skills groups varied. As some evaluation participants explained, this may be partly due to the limited opportunities for parents to observe or participate in these groups. Those who reported a positive change in their child explained that social groups helped their son/daughter to build friendships and socialize with others. Clients were also generally able to apply what they learned in their social skills groups to other environments with their peers. “My child relates to other kids better and uses some of the strategies to interact with peers.” (Parent) “My child has more confidence and is willing to try new things.” (Parent) “With my son, it’s reduced his anxiety. Because if he has some relationships with some people, then he’s less stressed when he goes places.” (Parent) Secondary outcomes One recommendation from Cycle One was to augment GAS scores with secondary outcomes, recognizing the limitation of GAS in capturing the full range of client outcomes. In response, secondary outcomes were tracked in Cycle Two. “Parents will always ask you questions [about secondary outcomes] …So, yeah, the GAS is there but we’re doing a lot more that we’re not able to show.” (ABA Consultant) Of the 50 clients who were reported as having secondary outcomes, five were seen to have one secondary outcome, and 31 had two or more. In contrast, 14 clients were seen to have no secondary outcomes. In total, there were 94 secondary outcomes reported across the 50 clients. Some examples of the most common secondary outcomes achieved by clients during Cycle Two are listed below. For the complete list of secondary outcomes reported, see Appendix 6. Social/Interpersonal  Getting along better with siblings  Increased self-confidence  Initiating positive play with same-age peers  Increased eye contact
  • 40. ABA Cycle Two Evaluation Report 40 | P a g e Communication  Requesting items using sentences  Increased depth and complexity of conversations  Communicating effectively with new people  Using greetings consistently Behavioural/Emotional Regulation  Decreased challenging behaviours  Increased compliance and ability to listen/compromise  Increased flexibility with new tasks Daily Living  Increased initiative with chore completion and ability to complete independently  Better able to navigate purchasing items in stores  Able to engage in larger variety of activities Secondary outcomes were reported for 72% of clients tracked. On average, clients acheived two secondary outcomes. As shown in Chart 16, secondary outcomes were fairly evenly distributed across domains. Secondary outcomes were found most commonly in the Social/Interpersonal domain (at 29% of the time) and communication the least (21% of the time). Chart 16: Frequency of Secondary Outcomes by Secondary Outcome Domain Area (N=94) Source: ABA Secondary Outcomes from November 2014 – February 2015 Chart 17 shows that most of the secondary outcomes occur in the home, especially in the case of Daily Living, in line with the nature of ABA programming. On average, clients are making considerable positive change in all four secondary outcome domain areas but the degree to which these outcomes are exhibited are always most considerable in the home. The school environment is where occurrences of secondary
  • 41. ABA Cycle Two Evaluation Report 41 | P a g e outcomes were found the least. It is important to note, that though there were only 94 secondary outcomes reported, some of these would occur in multiple environments simultaneously and to different degrees of success. Chart 17: Occurrence of Secondary Outcomes by Environment (N= 94) Source: ABA Secondary Outcomes from November 2014 – February 2015 Chart 18 shows the relationship between one’s primary domain and related secondary outcomes. There seems to be a close relationship between the social/interpersonal and communication domain areas on the one hand, and behavioural and daily living goals on the other. That is to say, for clients whose primary goal was social/interpersonal, they were most likely to have secondary outcomes within the communications domain (and vice versa). Similarly, for clients whose primary goal area was behavioural, they were most likely to experience secondary outcomes in the daily living domain (and vice versa). Within this sample of 50 clients 100% of those with daily living as their primary goal reported seeing secondary outcomes. In contrast, those with behaviour as their primary goal only reported 62% of clients having achieved any secondary outcomes.
  • 42. ABA Cycle Two Evaluation Report 42 | P a g e Chart 18: Frequency of Secondary Outcomes According to Primary Goal Domain: Source: ABA Secondary Outcomes from November 2014 – February 2015 Parent/Caregiver Outcomes Many parents/caregivers reported some positive changes in their own lives as a result of participating in the ABA program. Several parents/caregivers expressed increased confidence in their own ability to support their child. They had a better understanding of how to engage with their child as a result of the program, and felt more confident that change was possible after seeing their child achieve goals. As well, families felt more self-assured in their ability to support their children, and felt that they gained a better understanding of how to relate to their child. These perceptions were often mirrored by ABA consultants and other program staff. “It is such a big help to understand my child.” (Parent) “[I feel] less stress, less frustration. The behaviour’s reduced. But once in a while when it pops up, I‘m more able to understand why.” (Parent) “I really like the parent involvement piece [of the ABA program]. I think, as a parent, if you can learn different ways of relating to your child, whatever it may be, that’s always helpful.” (Key Informant) With increased confidence, parents reported feeling better able to cope with their child’s behaviour. Chart 19 reveals that 74% of parents agreed that after ABA, they feel more capable of helping their child with their new skill, or in dealing with their behaviour.
  • 43. ABA Cycle Two Evaluation Report 43 | P a g e Chart 19: I feel better because I am better able to cope with this skill/behaviour (N=410) Source: Consumer Feedback Survey (April 1, 2014 to January 21, 2015) Parent training is an important part of the ABA program and provides parents/caregivers with techniques and ideas to help them address their child’s skills and behaviours. According to Chart 20, the majority of parents (79%) were able to apply the techniques they were taught by ABA consultants. Moreover, several parents who participated in this evaluation reported that they had leveraged what they had learned in ABA to address other maladaptive behaviours and to teach their children new skills.
  • 44. ABA Cycle Two Evaluation Report 44 | P a g e Chart 20: I am able to use these techniques to address the skills/behaviours on my own (N=413) Source: Consumer Feedback Survey (April 1, 2014 to January, 21 2015) The success of the program was seen to be family-dependent. Many evaluation participants noted that better outcomes were achieved for clients when parents were actively involved. According to program staff, there was less likelihood that ABA goals would be maintained if the parents were unengaged in the sessions. Yet, even parents/caregivers who actively participated in the program reported difficulty maintaining the goals over the long-term without some additional professional support. System-Level Outcomes There was comparatively less data regarding system-level outcomes compared to other program outcomes. Program stakeholders seemed to have difficulty commenting on system-level outcomes achieved as a result of the ABA program to date. Not surprisingly, it was key informants who spoke most about system-level outcomes, identifying continued strength in some ASD interagency coordination and collaboration, however also acknowledging limitations in terms of partnering with external agencies/institutions (schools, etc.), providing a stronger continuum of support, and creating greater awareness of the ABA program itself. According to key informants there continues to be stronger coordination evident among ASD agencies across the Central West Region. Agencies that did not previously collaborate were now more aware of each other’s services. “ABA providers certainly get together to improve the program and discuss issues about the program.” (Key Informant) “I think our partnership is definitely working well. As I said, we meet with the clinical partners on a monthly basis and then bi-monthly usually there is the ABA service provider meeting...[We are] always trying to make it more seamless and transparent with families and how can we go about