2. ABA Cycle Two Evaluation Report
2 | P a g e
Evaluation Steering Committee
Kim Rodrigues (KPAS – Halton, Dufferin, Wellington, Waterloo)
Pam Tibbetts (BCBA – ROCK Halton)
Cathy White (Peel District School board – ret.; Autism Ontario)
Gail Jones (KPAS – Peel)
Angela Davis (Peel Behavioural Services)
Georgia Zadow (Parent, advocate)
Brett Friesen (CMHA – Waterloo Wellington Dufferin)
Debbie Dyment (Community Living North Halton)
CCBR Research Team
Rich Janzen
Kimia Ghomeshi
Kyla English
Jane Garant
Tangul Bilgehan
Centre for Community Based Research (CCBR)
73 King Street West, Suite 300
Kitchener, Ontario N2G 1A7
Phone: (519) 741-1318 Fax: (519) 741-8262
E-mail: rich@communitybasedresearch.ca
Website: www.communitybasedresearch.ca
3. ABA Cycle Two Evaluation Report
3 | P a g e
Contents
Executive Summary.......................................................................................................................................4
Introduction ..................................................................................................................................................8
Background ...............................................................................................................................................8
Evaluation Purpose ...................................................................................................................................9
Main Research Questions .........................................................................................................................9
Evaluation Design........................................................................................................................................10
Evaluation Approach...............................................................................................................................10
Program Theory of Change.....................................................................................................................11
Cycle Two Methods.................................................................................................................................14
Evaluation Findings .....................................................................................................................................16
Program Implementation .......................................................................................................................16
Program Outcomes.................................................................................................................................35
Future Directions ....................................................................................................................................46
Recommendations......................................................................................................................................51
Recommendations for the ABA Program................................................................................................51
Recommendations for Cycle Three Evaluation.......................................................................................53
References..................................................................................................................................................54
Appendices..................................................................................................................................................55
4. ABA Cycle Two Evaluation Report
4 | P a g e
EXECUTIVE SUMMARY
This report summarizes the results of the second evaluation cycle (April 2014-March 2015) for the
Applied Behaviour Analysis (ABA) program delivered to children and youth (ages 9-17) in the
Central West Region of Ontario. The lead partner for this program is Kerry’s Place Autism Services
(KPAS), in partnership with other key agencies in the region. KPAS works collaboratively with
Community Living North Halton (CLNH), Peel Behavioural Services (PBS), Reach Out Centre for Kids
(ROCK), Sunbeam Residential Development Centre (DSRC), and the Canadian Mental Health
Association Waterloo Wellington Dufferin (CMHA).
As part of the Cycle 1 evaluation, an ABA program logic model was created to explain the
continuum of service. The process starts with intake/referral, followed by service planning, ABA
service delivery, and discharge and interfacing. A child/youth and their family may go through the
process multiple times before they reach the age of 18. System-level activities are also undertaken
with KPAS partnership and external agencies.
Both quantitative and qualitative methods were used for this evaluation. The quantitative
methods included the ABA roll-up (including the Goal Attainment Scale (GAS)), the Consumer
Feedback Survey, the Central West Region (CWR) ABA Services Intake Management), Time
Utilization sheets, and the Secondary Outcomes tool. Qualitative methods included focus groups
with families, focus groups with staff, key informant interviews, and in-depth case studies.
Evaluation findings were categorized under the three main headings: program implementation,
outcomes, and future directions.
Implementation: In general, evaluation participants expressed satisfaction with the program and an
increasing number of families are interested in accessing ABA services. As a result, the waitlist
continues to grow in length. KPAS (which has centres in all the regions) provides service to the
highest number of children/youth overall, followed by PBS. The majority of families receiving or
waiting for services live in the Peel region. Families have the choice of participating in individual or
group sessions; most families choose individual sessions – often in the home setting – to meet
their goals.
Data from Cycle Two revealed five key strengths of program implementation. These included:
1. Consistency in program planning (e.g., the matching of consultants with past clients);
2. Variation in programming options (e.g., individual vs. groups sessions, at-home vs. agency
settings);
3. Opportunities to work with highly-skilled consultants;
4. The ability to focus families on manageable change; and
5. Effective partnerships among ASD agencies.
5. ABA Cycle Two Evaluation Report
5 | P a g e
Despite these strengths, evaluation participants also identified a number of challenges, many of which
were highlighted in Cycle 1. The six most significant challenges that emerged from Cycle 2 are as
follows:
1. Unsustainable targets, leading to staff burnout);
2. Limited program dosage, due to long wait lists and short program lengths;
3. Lack of other supports for parents during ABA and while on the waitlist;
4. Underdeveloped partnerships with schools and mental health agencies, leading to difficulties in
achieving and maintaining goals;
5. Limitations with social skills groups, with respect to matching clients and parental involvement.
Outcomes: The evaluation found that many of the outcomes anticipated by the program were in fact
being achieved. Often, families are seeing significant positive changes in their children’s behaviours and
skills. Most children either reached or exceeded their goals as indicated by both evaluation participants
and the GAS scores. Moreover, new data from Cycle Two revealed the presence of secondary outcomes
within ABA programming as well. Over 70% of children who were tracked using this tool experienced at
least one secondary outcome, such as getting along better with siblings.
Parents and caregivers themselves are also experiencing positive outcomes including heightened levels
of confidence and an increased capacity to cope with challenging situations. Nevertheless,
parents/caregivers reported some difficulty in maintaining goals over the long-term.
At the system level, there has been strong coordination between ASD agencies as well as an increase in
collaboration with external agencies (e.g., schools, mental health agencies, etc.) in some regions.
However, participants also indicated that significant challenges remain regarding working collaboratively
with schools, as well as providing coordinated services for clients with a dual-diagnosis.
Future Directions: Participants provided numerous suggestions to enhance the effectiveness of ABA
programming. For example, parents and some key informants identified a need for a reduction in the
waitlist and for more resources and support. Below are the key recommendations that emerged from
the Cycle Two report, based on discussions with the evaluation Steering Committee as well as
suggestions from evaluation participants. Recommendations related to changes in Cycle Three of the
evaluation are provided at the end of the report.
Recommendation #1: Increase program resources to sustainably meet service targets and achieve long-
term outcomes for the diversity of clients and their families. Prioritize the following program
improvements:
Hire additional ABA consultants to reduce caseloads, shorten wait lists, and to improve the
diversity of staff.
Provide flexibility in program length, with the option of longer program cycles for families who
wish to work on goals that would require more treatment time.
Provide semi-regular capacity-building opportunities for staff, including workshops, conference
participation, and annual training sessions.
6. ABA Cycle Two Evaluation Report
6 | P a g e
Add client-oriented supplies (e.g. toys, activities, etc.) to the ABA family resource centres.
Recommendation #2: Work with community partners to provide a range of additional resources in
multiple formats and languages (as needed) for parents/caregivers on the wait list. Prioritize the
following:
Information and orientation sessions on the ABA program, its family-centered approach, and
goal setting process. Orientations (online or in-person) could be mandatory for new families.
Workshops or resources for ABA clients and parents/caregivers on strategies to support youth
with ASD during their transition to high school or adulthood, or their transition from ABA
agencies serving up to eight year-olds to agencies serving 9-17 year-olds.
An online forum for families that can be used to correspond with other families.
Resources such as video tutorials on behavioural management of children/youth with ASD.
Group social activities and related resources for children/youth with ASD.
Recommendation #3: Continue to adjust the first-come-first-serve policy for clients in the social skills
domain so that clients in each group are selected according to similar social capacities and needs. As
well, organize visits (in-person or by phone) with clients being considered for social skills groups to
ensure that clients are well matched.
Recommendation #4: Explore any possible options to start the service planning process while families
are on the wait list (e.g. provide examples of possible ABA goals or illustrations of how the program
works).
Recommendation #5: Explore or further implement group models of services, such as summer groups or
dyads.
Recommendation #6: Improve the goal setting process in the following ways:
Suggest that parents select goals that address clients’ more significant behavioural challenges.
Provide examples of possible ABA goals and program illustrations.
Set goals for parent/caregivers to boost their involvement in the program.
Further seek to involve other ASD-related professionals when appropriate to collaboratively
create the treatment plans.
Continue to attempt to consult with the client’s school when the goal that would require school
involvement.
Recommendation #7: Continue having discussions with the Ministry of Child and Youth Services
regarding the suitability of the Child and Adolescent Needs and Strengths (CANS) to the ABA program.
Recommendation #8: Continue to explore options for including more natural settings in ABA sessions,
particularly for clients in the social skill and daily living domains to support them in navigating real life
situations.
7. ABA Cycle Two Evaluation Report
7 | P a g e
Recommendation #9: Further improve reporting mechanisms and increase parental involvement in
social skill interventions to strengthen long-term outcomes for participating children/youth.
Recommendation #10: Work with community partners to provide a treatment model for older clients
that is more age appropriate and engaging.
Recommendation #11: In the absence of a short wait list, institute bi-monthly follow-ups (via telephone
or email) following the booster sessions (when requested by parents/caregivers) to ensure maintenance
of goals and to support parents/caregivers with challenges as they arise before the next program cycle.
Recommendation #12: As part of the re-referral process, continue to identify whether families would
want to work with the same consultant in the next program cycle when possible and appropriate.
Recommendation #13: Prepare more detailed informational materials on the ABA program to distribute
to other ASD agencies, mental health and health care services, and schools. These resources should
clearly outline what the program is (and is not), and for schools should highlight how the program
complements the ASD services and supports they already offer. In some instances, organize meetings
between school representatives and ABA managers, including testimonials from principals, families, and
teachers who have had a positive experience with the program.
Recommendation #14: ASD agencies to prepare a transition plan for children/youth moving from
agencies serving clients up to eight year-olds to agencies serving 9 to 17 year-olds.
8. ABA Cycle Two Evaluation Report
8 | P a g e
INTRODUCTION
This report summarizes the Cycle Two evaluation findings of the Applied Behaviour Analysis (ABA)
services for children and youth (aged 9 to 17 years) in the Central West Region of Ontario (Waterloo,
Wellington, Dufferin, Halton, and Peel). The report covers the evaluation period beginning April 1, 2014
and ending March 31, 2015, and is the second of three evaluation cycles ending in MaAs lead agency for
the Central West Region, Kerry’s Place Autism Services (KPAS) contracted the Centre for Community
Based Research (CCBR) to lead the evaluation under the guidance of a cross-stakeholder evaluation
steering committee.
Background
ABA services in the Central West Region are delivered to children and youth ages 9 to 17 years with
Autism Spectrum Disorder (ASD) through a network of service agencies known as the ABA Central West
Region. This network delivers services that are focused on establishing skills and/or addressing key
issues that will improve the quality of life for children and youth with ASD. These services are oriented
around the following four domains:
Communication
Behaviour/emotional regulation
Social skills
Daily living skills
The partnering agencies also provide support to the parents/caregivers of children with ASD, aiming to
improve their quality of life, enhance their behaviour management skills, and instill a sense of
competence. To address these goals, a flexible support model is developed collaboratively with
participating children or youth and their parents/caregivers, and can include group, individual, family,
and parental capacity supports. Training is provided to parents and caregivers while their child or youth
is participating in ABA programming.
Partnering service delivery agencies for children/youth ages 9 to 17 years include Kerry’s Place Autism
Services (KPAS; lead partner), Community Living North Halton (CLNH), Peel Behavioural Services (PBS),
Reach Out Centre for Kids (ROCK), Sunbeam Residential Development Centre (DSRC), and the Canadian
Mental Health Association Waterloo Wellington Dufferin (CMHA). Another partnership exists for ABA
services delivered to children aged zero to eight years which is led by ErinoakKids (EOK) and includes
Dufferin Child and Family Services (DCAFS), KidsAbility Children’s Centre (KidsAbility), and Woodview
Children’s Centre (WCC). A joint waitlist for both partnerships is managed by KPAS.
The ABA Central West Region is committed to the development, delivery, evaluation, and continuous
improvement of its ABA-based services and supports. To this end, they requested that an evaluation of
the ABA services be conducted to enable the program to make evidence-based decisions for future
program delivery. The evaluation was to be grounded in the perspectives of children and youth, their
families, and the systems of service in their communities. The scope of the evaluation was to be focused
on the service delivery partnership for children ages 9 to 17 (i.e., the KPAS-led partnership).
9. ABA Cycle Two Evaluation Report
9 | P a g e
Evaluation Purpose
The evaluation was designed to understand how existing ABA programming is being implemented at
each of the Central West agencies for children and youth ages 9 to 17. In particular, the evaluation was
designed to assess what is working well and what is not working well within the context of ABA
programming. The evaluation should therefore clarify what helps and hinders ABA Central West from
creating the desired change that they wish to see in children and youth, their families, and the networks
of ASD-related services in their communities. In other words, the evaluation seeks to understand the
processes of ABA programming, and how these processes link to intended outcomes.
This evaluation was also designed to explore the impact of ABA programming in the Central West
Region. Specifically, the evaluation explores the outcomes of ABA on: 1) children and youth (aged 9 to
17); 2) families/caregivers of children and youth clients; and 3) the networks of ASD-related services.
Implementing a multi-cycle evaluation allows the program to consider outcomes over time.
Finally, the evaluation was designed to be forward looking. Data from the evaluation provides concrete
recommendations on how the Central West Region can improve their ABA services in the future for
children/youth, their families, and within the local networks of ASD-related services.
Overall, the evaluation has three inter-related objectives:
To assess the implementation processes of ABA programming in the Central West Region
To assess the outcomes of ABA programming at the child-, family-, and systems-levels
To identify future directions for improving ABA services in the Central West Region
Main Research Questions
Three main research questions guided the evaluation, consistent with the three objectives mentioned
above. These questions (and corresponding sub-questions) are listed below, and form the basis of how
this report is organized.
1. How is the ABA program being implemented at agencies in the Central West Region? (process)
a) What is the underlying program theory, including the main activities and expected
outcomes?
b) What are the child and family demographics in ABA Central West Region communities?
c) What resources (e.g., human, financial, partnership) support the various aspects of program
functioning (i.e. inputs)?
d) What aspects of the program seem to be working well? Not working well?
e) What is facilitating and what is hindering effective program implementation?
f) How are the guiding principles evident in service delivery?
2. How and to what extent has ABA programming impacted participating children/youth and their
families? (outcomes)
a) What are the main accomplishments and products of the ABA program? (i.e. outputs)
10. ABA Cycle Two Evaluation Report
10 | P a g e
b) What outcomes has the ABA program had for participating children and youth in the
domains of communication, behaviour/emotional regulation, social skills, and daily living
skills?
c) What outcomes has the ABA program had for the families of participating children and
youth in terms of their quality of life, enhanced behaviour management skills, and sense of
competence?
d) What systems level outcomes has ABA programming achieved?
e) How have outputs and outcomes changed over time?
3. What recommendations can be made to improve the outcomes of ABA services for children, families,
and the networks of ASD-related services in their communities? (future directions)
a) What should be done to enhance the effectiveness of the ABA programs for children and
youth? For families? For the network of ASD-related services in their communities?
b) How should future ABA programming be evaluated?
EVALUATION DESIGN
Evaluation Approach
This evaluation is carried out using an approach that is consistent with the three hallmarks of
community-based research: community-situated, collaborative and action-oriented (Ochocka & Janzen,
2014).
Community-situated. Community-situated means that research is of practical relevance to community
members and is carried out in community settings, rather than being driven simply by researcher
interests. This evaluation therefore seeks to understand the program context across the Central West
Region, paying close attention to the program’s guiding principles. It also means that the evaluation
draws on information already collected by the program (i.e., secondary data), while involving ABA staff
in helping to collect primary data (e.g., helping to organize interviews and focus groups).
Collaborative. Collaborative means that community members and researchers share control of the
research agenda through reciprocal involvement in the research design, implementation and
dissemination. The ABA services evaluation therefore makes sure to involve different stakeholders
(including children/youth, parent/caregivers, various levels of staff, and community partners)
throughout the evaluation process. Concretely, most stakeholders are involved through the evaluation
Steering Committee that guides each step of the evaluation. This Committee creates momentum and
increases the likelihood that findings will be acted upon. In addition, stakeholders are also involved as
research participants.
Action-oriented. Action-oriented means that the process and results of research are useful to
community members in making positive change. This evaluation is therefore utilization-focused,
meaning it is designed to be as useful as possible. Action-oriented also means that there is ongoing
11. ABA Cycle Two Evaluation Report
11 | P a g e
feedback of the evaluation findings so that action can be taken following each evaluation cycle. To this
end, the evaluation report includes concrete recommendations for future action. Action-oriented also
means that the evaluation is open to improvement from one cycle to the next; recommendations to
improve Cycle Three of the evaluation are included at the end of this report.
Program Theory of Change
An understanding the ABA Central West Region’s program theory helped to guide the evaluation design.
A program theory is an explanation of how a program is expected to be an intervention for positive
change (see Chen, 2005). Such a theory is useful for evaluations as it brings clarification among
stakeholders about what the program is intended to do, and it informs the development of evaluation
data collection tools.
The program logic model is presented below. The program logic model was developed collaboratively
by reviewing project documents, holding a site visit, and through Steering Committee discussion.
Overall, the logic model emphasizes the individual and systems-level outcomes specific to ABA
programming, while acknowledging that these services are one element within the entire continuum of
care for children and youth ages 9 to 17 years with ASD.
Horizontally, the program logic model activities and outcomes are arranged according to where they
would occur within the service delivery flow. These categories include: Intake/Referral, Service Planning,
ABA Services Delivery (divided according to Child/Youth and Parent/Caregiver), Discharge and
Interfacing; and System (divided according to ‘With Partner’ and ‘With External Agencies’).
The program logic model shows the service delivery flow in a time-ordered fashion from first contact to
discharge/interfacing. Note that at the end of discharge/interfacing, a dashed arrow indicates that
program re-entry may occur. Notice also that the system-level category is not connected to the service
delivery flow with an arrow as it is assumed that the system operates externally and concurrently with
ABA programming. Waitlist resources are visually shown in the shape of a circle following intake (where
clients first access them), but in fact these resources cut across the ASD continuum of care.
Vertically, the program logic model is arranged according to Activities, Outcomes (both individual and
system), and Program Mission. Activities detail the parts of the service delivery. The set of activities for
each step in the service delivery flow unpack the work that is undertaken by the network partners.
System-level activities are also noted (both with partners and with external agencies).
Individual Outcomes are those that benefit children/youth and their parents/caregivers, while System
Outcomes are those that benefit ABA Central West partners and external agencies/institutions. Notice
that the Service Planning and the Discharge & Interfacing categories lead to the primary ABA program
outcomes at the individual and family level. It should also be noted that the Parent/Caregiver individual
outcomes fold into the Child/Youth outcomes, as ABA programming is designed to improve the capacity
of parents/caregivers to support improvements to their child’s behaviour. Similarly, system-level
outcomes at the individual level (which improve participants’ ability to navigate the ASD continuum of
care) fold into Parent/Caregiver outcomes. Finally, System Outcomes are shown to cradle the entire
service delivery flow, as their impact is reflected throughout the ABA program and continuum of care.
12. ABA Cycle Two Evaluation Report
12 | P a g e
The Program Mission is noted at the bottom of the logic model. This mission captures the guiding
principles of service delivery outlined by the Ministry of Children and Youth Services. This Ministry
document identifies the purpose of ABA services as they pertain to improving the services and supports
for children with ASD.
The ABA Central West Region theory of change has been evolving and will likely continue to evolve over
time. A deepening understanding of program theory is expected to occur as the program matures and
responds to changing circumstances and evaluation learnings. This means that the evaluation will need
to be open to exploring and capturing the unexpected, as well as the anticipated outcomes and
processes. It also means that the evaluation itself can be an intervention in further clarifying and honing
the program theory (Janzen, Seskar-Hencic, Dildar & McFadden, 2012).
13. ABA Cycle Two Evaluation Report
13 | P a g e
2014 ABA Program Logic Model
14. ABA Cycle Two Evaluation Report
14 | P a g e
Cycle Two Methods
This section outlines the methods of data gathering for Cycle Two of the ABA services evaluation. The
number of methods and the volume of data collected for Cycle Two was an expansion upon Cycle One
which was a shorter evaluation cycle.
The evaluation methods were determined through a systemic process of building an evaluation
framework. The evaluation framework process clarified the main research questions, program theory (i.e.,
program logic model), and corresponding measurement matrix (see Appendix 2). The measurement matrix
links expected program outcomes (impact measurement) and processes (performance measurement) with
indicators and corresponding data collection tools. The matrix also lists questions about future directions
(strategy measurement) linking them with corresponding data collection tools.
Based on this measurement matrix, data collection tools were identified or developed. The individual
tools include a mix of existing measures used by the ABA Central West Region and those developed by the
evaluation team. Some of the quantitative tools introduced in Cycle Two were developed to address gaps
in secondary data according to Cycle One findings. The qualitative tools that were introduced in Cycle One
were also revised and expanded to further explore major themes that emerged in the first cycle, and to
examine the relationship between the clients’ domain area or diagnosis and their experience in the
program. Combined, the tools include a mix of quantitative and qualitative measures to provide breadth
and depth of insight. Appendix 3 includes primary data collection tools with corresponding consent forms.
Cycle Two methods are listed below, organized into quantitative methods (representing secondary data
already gathered by the program) and qualitative methods (representing primary data that was collected
specifically for the evaluation).
Quantitative Methods (Secondary Data)
ABA Roll-up. The ABA Roll-up is an Excel spreadsheet that is regularly updated and maintained by clinical
supervisors across the Central West Region. Each region independently completes their own spreadsheet,
which includes quantitative information about program participants and program delivery. More
specifically, the spreadsheet tracks program outputs by client (e.g., number of hours of service delivery,
type of service, and region). Included within the ABA Roll-up are client results from the Goal Attainment
Scale (GAS).
GAS quantifies the achievement of goals set by parents and ABA consultants based on a 5-point rating
scale from -2 to +2. If a chosen goal (for example, reduction in aggression) is recorded as -2 in the Roll-up,
then the outcome level was much less than expected (e.g., there was a regression in the goal area). If the
goal is recorded at +2, then the outcome was much greater than expected (e.g., the child greatly improved
and the aggression was significantly reduced). A score of zero means that the expected level of goal
attainment was reached but not surpassed (i.e., the child met but did not exceed their goal).
This report includes data collected in the ABA Roll-up for most of Cycle Two, from April 1, 2014 until
December 31, 2015. In total, outcomes from 658 clients aged 9 to 17 years (across all agencies) were
tracked in the Roll-up during this time, and were included in the analysis.
Consumer Feedback Survey. The Consumer Feedback Survey (CFS) is self-administered by
parents/caregivers of children and youth participating in ABA programming. Upon completing a round of
15. ABA Cycle Two Evaluation Report
15 | P a g e
service, parents/caregivers are given a hardcopy of the CFS, and are asked to fill it out and return it in an
envelope (in-person or by mail) to the program manager at their agency. Central intake staff at the various
ASD agencies will then insert the individual CFS data into the online version (i.e., on Survey Monkey),
where all CFS data from agencies in the ABA Central West Region is stored.
The CFS quantitatively evaluates program outcomes in the domains of quality of life, behaviour
management skills, and feelings of competence at the family level. The survey also contributes
quantitative insight into program processes (i.e., what is working well and not working well), and provides
suggestions for future directions via qualitative components (i.e., boxes for parents/caregivers to provide
suggestions or comments). The CFS was revised in the fall of 2014, partially to include questions on the
generalization and maintenance of ABA goals.
In this evaluation cycle, survey responses of 423 parents/caregivers were analyzed, approximately 69%
from Peel Region, 20% from Halton Region, and 11% from the Waterloo, Wellington, and Dufferin Regions.
Central West Region (CWR) Applied Behavioural Analysis (ABA) Services- Intake Management. The CWR
ABA Services Intake Management database (referred to as ‘intake database’ here) provides demographic
information on all children/youth and their families that are currently enrolled in ABA services within the
Central West Region. The information in the intake database provides a demographic snapshot of clients
undergoing the program, in addition to total number of clients ever served and interventions delivered.
The database provides details about how many clients are on the central waitlist vs. in service, and
identifies the domain area(s) that families are working on during their sessions. Individual results of the
Child and Adolescent Needs and Strengths (CANS) tool are also available within the intake database.
Aggregate data from the intake database (for the period of April 1, 2014 to December 31, 2015) was
analyzed during Cycle Two. Due to privacy concerns, ABA staff de-identified and collated all data prior to
sending it to the external evaluating team. Data from the CANS portion of the intake was analyzed in Cycle
One and will be re-visited and re-analyzed in Cycle Three.
Time Utilization Sheets. The time utilization sheets were a tool developed out of recommendations from
Cycle One. These time sheets help ABA consultants to track the number of hours per day that they spend
on various tasks (e.g., direct client contact, client documentation, etc.). According to ABA consultants and
other program staff, a number of these tasks were not previously tracked.
In this report, data was analyzed over a three month period (October 2014 to December 2014), as these
were the only months consistently tracked across all agencies. In total, data from 25 consultants was
analyzed for this report: one from CLNH, seven from PBS, two from CMHA, two from ROCK, five from
DSRC, and 14 from KPAS.
Secondary Outcome Tool. The Secondary Outcome tool was developed out of recommendations from
Cycle One, and was implemented starting in November 2014. This tool helps to determine: 1) the
percentage of clients who exhibit positive outcomes other than the achievement of their main goal; 2) the
extent to which these secondary outcomes are exhibited; and 3) the degree to which these outcomes
occur in different environments (i.e., at home, school, or in public). Consistent with the ABA program
goals, secondary outcomes are tracked by the four main domain areas.
For the purposes of this evaluation report, 50 secondary outcome sheets were collected and analyzed
from various agencies in the Central West Region.
16. ABA Cycle Two Evaluation Report
16 | P a g e
Qualitative Methods (Primary Data)
Focus groups with parents/caregivers. A total of nine focus groups were conducted in Cycle Two with
parents/caregivers (n=32) of ABA clients. Five were held via teleconference, and four were held in-person.
Participants were selected and recruited in cooperation with program managers on the evaluation Steering
Committee, and according to purposive sampling criteria (i.e., diversity in geographic representation
across the Central West Region, length of involvement in ABA program, gender, ethnic diversity, range of
diagnosis, range of child/youth ages, and range of models of services). All main research questions were
addressed in these focus groups.
Focus groups with staff. Three focus groups were held with ABA staff during Cycle Two, including one with
clinical supervisors (n=5) and two with ABA consultants (n=19). Participants were selected and recruited in
cooperation with program managers and according to purposive sampling criteria developed by the
evaluation Steering Committee (i.e., representation across Central West Region, diversity in clinical and
frontline staff, and diversity in length of involvement with program). All three main research questions
were addressed to gain front-line insight on program processes and outcomes for child/youth and
parent/caregivers, as well as recommendations for improvement (see Appendix 3).
Key informant interviews. Twelve key informant interviews were held in Cycle Two via telephone, to gain
insight into the three main research questions related to systems-level processes, outcomes, and
recommendations for improvements. Participants were selected and recruited in cooperation with the
Steering Committee and according to purposive sampling criteria. Participants were located across the
Central West Region and included senior staff at ABA agencies, a Ministry of Child and Youth Services
representative, school representatives, and senior staff at ABA support agencies.
Case studies. Case studies were introduced as a new qualitative tool in Cycle Two to give further insight
into some of the common strengths and challenges of the program, while showcasing the diversity in
clients, families, and program experiences. Three clients were selected according to purposive sampling
criteria (i.e., diversity in geographic representation across the Central West Region, length of involvement
in the ABA program, gender and ethnic diversity, range of child/youth ages and diagnoses, and range of
models of service). For each case study, the child’s parent or caregiver was interviewed, as well as their
ABA consultant. When possible, the child/youth was also interviewed, or a family friend if the child was
uninterested or unable. In total, eight interviews were conducted across the three case studies.
EVALUATION FINDINGS
The findings of the evaluation are organized according to the three main research questions related to
program implementation, program outcomes, and future directions. Data was analyzed across all research
methods with main themes highlighted below.
Program Implementation
Overall, evaluation participants expressed an appreciation for the ABA program, and were mostly satisfied
with the service they delivered or received. While there remain significant challenges with program
implementation, data revealed that families are “wanting the service, needing the service, [and] liking the
17. ABA Cycle Two Evaluation Report
17 | P a g e
service” (Key Informant). This section begins by describing the program overall (i.e., number of clients
served, percentage of children in each domain), followed by a list of the top strengths and challenges
revealed in Cycle 2. Similarities and differences to Cycle 1 are highlighted throughout.
Description of ABA Participants
The ABA program is available to all children/youth under the age of 18 years who have a confirmed ASD
diagnosis (Autism/Autism Disorder, Pervasive Developmental Disorder, or Asperger’s Disorder).
Confirmation of diagnosis must come from a certified and qualified professional (i.e., a family physician,
pediatrician, psychiatrist, psychologist, or psychological associate) (Autism Ontario, 2015).
Chart 1 shows a snapshot of the total number of clients (aged 9 to 17 years) in service on March 6, 2015,
within all areas of the Central West Region. The majority of children who were in service were between
the ages of 11 to 14 years. In all areas, the 15-17 year age bracket had the fewest number of clients in
service. Chart 1 also demonstrates that Peel served the greatest number of clients during this time (135),
accounting for 55% of all children/youth who were in service across the Region. This is due to the fact that
Peel has a higher population than the other regions, and is thus allocated more of the funding and
resources.
Chart 1: Number of children/youth in service by age group and region (N=356)
Source: CWR ABA Service- Intake Management (Snapshot of those in service on March 6, 2015)
Chart 2 shows the total number of eligible children/youth in 2014 (ages 9-17 years) by service status and
by service provider cumulative for April 1, 2014 to December 31, 2014. In this report, ‘eligible’ is defined as
all children who: (a) are on the waitlist; (b) are in service; or (c) completed a block of service in the 2014
fiscal year. The total number of children served is a cumulative figure for the period of April 1 2014 to
December 31, 2014. “Other Eligible Children” refers to any clients who are on the waitlist, or have
completed ABA service. The majority of eligible children were clients in KPAS Peel (26%) and PBS (24%).
40
12
5 6 7
66
17
28
15
2
29
7 7
3
0
0
10
20
30
40
50
60
70
Peel Halton Waterloo Wellington Dufferin
TotalNumerofChildren/Youthin
Service
Ages 9-10
Ages 11-14
Ages 15-17
18. ABA Cycle Two Evaluation Report
18 | P a g e
Chart 2: Total number of eligible clients (aged 9-17 years) by service status (served vs. other eligible
children) and service provider (N=2264)
Source: CWR ABA Service- Intake Management (cumulative for April 1, 2014 to December 31, 2014)
Discussions with evaluation participants revealed that families represent a range of income levels. Given
that ABA services are offered at no cost, low-income families can participate in the program. Discussions
also revealed that ABA families are ethnically diverse. This is particularly true in the large urban areas such
as Peel Region. Some families have recently immigrated and are still in the process of learning English.
Some ABA consultants can speak multiple languages. Interpreters are also available, if needed, but at a
cost to the program. As was noted in the Cycle One report, some key documents are not always translated
(e.g. consent forms and individual plans).
Description of Program
As explained in the Cycle 1 report, children or youth can work on goals related to one of four domains:
communication, social skills, daily living skills, and behaviour/emotional regulation skills. Chart 3 depicts
the percentage of clients within each domain, from April 1, 2014 to January 21, 2015. Children/youth can
only work within one domain per session.
Chart 3 reveals that the majority of clients focused on either social skills (40%) or behavioural/emotional
regulation skills (31%). Clearly, the majority of families feel that social skills and behavioural/emotional
regulation are the most important domains to work on during their ABA program.
19. ABA Cycle Two Evaluation Report
19 | P a g e
Communication
12%
Social Skills
40%
Daily Living
Skills
17%
Behavioural /
Emotional
Regulation
31%
79
125
32 32
48 48
21 30 26
81
48
19
4 6 3 7
35
10
0
20
40
60
80
100
120
140
TotalNumberofChildren/Youth
Service Providers
Individual
Group
Chart 3: Skill domains in ABA services for children/youth aged 9-17 years (N=256)
Source: CWR ABA Services- Intake Management (CANS April 1, 2014-January 21, 2015)
ABA is offered as an individual, one-on-one program or as a group program with other children/youth of a
similar age, level of functioning, and specified goal area. Chart 4 reveals the total number of clients per
ASD agency who participated in each type of service during Cycle Two. Clearly, most agencies served a
greater number of clients through one-on-one sessions, with the exception of PBS and DSRC who served
slightly more clients through their group sessions.
Chart 4: Form of service delivery by service provider for children/youth aged 9-17 (N=662)
Source: ABA Roll-up (April 1, 2014 – December 31, 2014)
Chart 5 demonstrates the total number of hours spent on client intervention (i.e., directly with the
child/youth) versus parent training from April 1, 2014 until December 31, 2014. Definitions of client
intervention hours and parent training hours are provided in Appendix 4. All agencies spent more time
working with clients as opposed to training parents or caregivers. The average amount of time that
consultants spent with each client was between 9 and 18 hours per round. The average amount of time
that consultants spent training parents or caregivers was between four and 15 hours per round. In
20. ABA Cycle Two Evaluation Report
20 | P a g e
comparison to all other service providers, KPAS Peel spent the most amount of time, on average, with
clients (approximately 18 hours/round), while CLNH spent the most amount of time, on average, training
parents or caregivers (approximately 15 hours/round). CLNH spent an almost equivalent amount of time
on both, as opposed to agencies such as KPAS Peel, DSRC and CMHA who spent much more time on client
intervention. It must also be noted that some clients, especially those with more complex cases (e.g., a
dual diagnosis) may be in service for up to six months, if a supervisor’s clinical judgment dictates that the
client cannot be released. This, according to many key informants, can contribute to the challenges
outlined later in this report (e.g., growing waitlist, staff burnout, etc.).
Chart 5: Amount of time spent on client intervention and parent/caregiver training by service provider for
children/youth aged 9-17 years (N=621)
Source: ABA Roll-Up (April 1, 2014 – December 31, 2014)
Chart 6 gives a more fulsome picture of how ABA consultants spend their time, not only with clients and
their parents/caregivers, but on paperwork, training and other duties and responsibilities. The tool to
collect this data (‘Time Utilization Sheet’) was implemented for the first time in Cycle Two; it is filled out
monthly by ABA consultants. Chart 6 demonstrates the average time spent on each task type, from
October 2014 to December 2014. The chart clearly shows that consultants spend the greatest percentage
of their time on documentation (~26%) followed by direct client contact (~23%) and administration (~12%).
Definitions of each type of task are provided in Appendix 5.
15.4
17.7
14.5
9.4
15.0
16.9
15.9
13.9
12.7
12.9
4.9
8.7
5.0
11.3
12.9
15.0
4.1
4.8
0.0
2.0
4.0
6.0
8.0
10.0
12.0
14.0
16.0
18.0
20.0
Averageamountoftimespentoninterventionand
training(perclient)
Service Providers
Client Intervention Hours
Parent Training Hours
21. ABA Cycle Two Evaluation Report
21 | P a g e
Chart 6: Total Time Spent by Task Type for October 2014 – December 2014
Source: Time Utilization Sheets, October 2014 – December 2014
Strengths of Program Implementation
Evaluation participants highlighted several strengths in regards to ABA program implementation, many of
which were similar to those revealed in Cycle One. Overall, parents and caregivers were grateful for the
program and appreciated the various components involved in service planning and delivery.
“This ABA program is like gold. It is an untapped hidden resource that is extremely effective.”
(Parent)
“The tracking was good. The feedback was good. The tools were excellent. Knowledge of the child
and an understanding of the child’s needs was excellent. It worked.” (Parent)
Chart 7 further depicts parents’/caregivers’ satisfaction through their response to the statement “Overall, I
found the program helpful.” 93% of all responders agreed or strongly agreed.
Direct Client
Contact
23.0%
Indirect Client
Contact
5.6%
Client
Documentation
25.9%
Administration
12.3%
Case Review
4.8%
Supervision
1.9%
Committee
Meeting
& Relations
1.3%
Staff Education
2.9%
Student Education
0.3%
Community
Education
0.5%
Other
3.2%
Travel
8.9%
Sick
1.9%
Vacation
7.6%
22. ABA Cycle Two Evaluation Report
22 | P a g e
Chart 7: Overall, I found the program helpful (N=414)
Source: Consumer Feedback Survey (April 1, 2014 to January 21, 2015)
Data from Cycle Two revealed five key strengths of program implementation. These included: (1) the
consistency in program planning (e.g., the matching of consultants with past clients); (2) the choice of
various programming options (e.g., individual vs. groups sessions, at-home vs. agency settings); (3) the
opportunity to work with highly-skilled consultants; (4) the ability to focus families on manageable change;
and (5) the partnerships among ASD agencies. These strengths are further explored below.
Consistency in program planning
Several parents/caregivers and frontline consultants appreciated the program’s consistency in terms of
client-consultant matching as well as grouping for children in social skills groups. Such consistencies in
service led to greater efficiencies in programming and higher rates of success. For example, when clients
are matched with the same consultant as in previous sessions, service delivery is strengthened due to
existing relationships and trust. In addition, the consultant may be better able to tailor the program to
meet the needs of the client, if they are already aware of their strengths and needs from previous sessions.
Consultants can also help their client to build upon skills that they had worked on previously.
“Having the same person be with your child on an ongoing, more fluid basis… makes all the
difference.” (Parent)
For similar reasons, parents and caregivers appreciated when their children were matched with the same
children from previous social skills groups. Bringing the same children together (when the children are well
matched) can improve the quality of their service and help the children to develop stronger relationships
with each other.
According to several evaluation participants, having a consistent program also means that parents and
children/youth can expect a similar structure each time they come off the waitlist. Children are more
aware of the expectations of the program, and understand what will happen during the sessions. Children
can also build on skills from previous sessions, and continue learning from where they left off.
Strongly Agree
70%
Agree
23%
Neutral
6%
Disagree
1%
Strongly Disagree
0%
23. ABA Cycle Two Evaluation Report
23 | P a g e
“We see more compliance [in subsequent sessions]… The kids are understanding what the
expectations coming back in are.” (Key Informant)
“I also feel that the families who are coming back in for 2nd
and 3rd
and 4th
round of service are
taking the option to build on the skills that have already been learned… We have some clients that
come through and only work on communication, and that’s because they’re building on what they
have previously learned.” (Key Informant)
Choice in programming options
According to many evaluation participants, another strength of program implementation is the choice in
programming options for families. Parents/caregivers appreciated the option of one-on-one or group
sessions, as well as the location of service (in the home, ABA agency, or community). For many children,
depending on their goal, individual sessions work best; they can work directly with a consultant to focus on
a goal specific to their needs. Some of these sessions occur in the family home, which many parents and
caregivers considered an effective approach for working towards change. Skills can often be more
effectively taught, learned and applied in the natural home setting, where many behaviours occur. In
addition, home-based sessions are sometimes more convenient for parents who are balancing multiple
responsibilities and may have other children to care for. Some clients also prefer to keep their diagnosis to
themselves, especially those in their teenage years, and thus prefer at-home sessions where they won’t be
seen with their consultant.
The option of having ABA sessions in an agency setting is also appreciated by parents and caregivers.
Children are sometimes less distracted in a more formal setting, and can work in a room where they won’t
be distracted by the goings-on in a home. Group sessions also happen most often in the agency setting; for
some clients and some goals, group sessions are the most effective as they provide ample opportunity for
children to interact with others and to develop social skills in a safe and supportive environment.
Evaluation participants appreciated the large and growing number of agency settings; having a greater
choice in where the program can occur makes it more accessible for families in the region.
“It’s really good that families are given a choice between in-home and group supports.” (Key
Informant)
“Agencies continue to open up centres to make the services more accessible…in an effort to make
the services and spaces more accessible to children with ASD.” (Key Informant)
Another option for ABA programming emerged last summer, as several agencies began offering ABA
summer groups that typically lasted one week (approximately 30 hours). Families who came off the waitlist
during these months could choose to attend a summer group, rather than have the standard eight-week
session. These groups were often situated in or involved field trips to natural environments (malls, bowling
alleys, etc.). In general, parents/caregivers found that their children benefitted from having sessions in
natural environments, whether in the context of the summer group or during regular social skills groups.
ABA consultants generally agreed that despite the challenges (e.g., logistics, finance, safety), using natural
environments contributed to the generalization and maintenance of goals.
24. ABA Cycle Two Evaluation Report
24 | P a g e
0
50
100
150
200
250
300
350
Strongly
Agree
Agree
Neutral
Disagree
Strongly
Disagree
325
76
12
0 1
NumberofRespondents
Skilled consultants
As was found in Cycle 1, the majority of key informants, clinical supervisors, and parents/caregivers in this
evaluation expressed gratitude for their highly skilled and extremely dedicated consultants. The
consultants showed creativity in how they tailored their tools and their approach to meet the needs of
individual clients. Consultants were a valued resource for families who felt they lacked the knowledge and
tools to help their children/youth on their own. One family, for example, mentioned how they only came
to understand the root of their child’s behaviour with the help of their consultant. A number of families
felt that their consultants went above and beyond to support them, and provided guidance on non-ABA
related matters as well (e.g., referrals to other services).
“The caliber of therapists is outstanding.” (Parent)
“I think our consultants… they are very skilled in the services they deliver, and I don’t think they shy
away from difficult situations.” (Key Informant)
Chart 8 illustrates one aspect of parents’ and caregivers’ satisfaction with their ABA consultant: how well
they understood the information presented to them. The chart shows that almost all parents and
caregivers believed that the information presented by the consultant was done so in a clear, easy-to-
understand way.
Chart 8: The information presented by the therapist was presented in a clear, easy to understand manner
(N=417)
Source: Consumer Feedback Survey (April 1, 2014 to January 21, 2015)
As with Cycle 1, the ABA consultants reported a strong commitment to the program both professionally
and personally, and desired to see it succeed. In general, they appreciate opportunities to advance their
skills (training workshops, conferences, etc.), although they have limited time to pursue capacity-building
opportunities.
25. ABA Cycle Two Evaluation Report
25 | P a g e
Focusing families on manageable change
Another strength of the program is that it helps to focus families on manageable change, despite the often
overwhelming challenges they face in caring for their children/youth with ASD. To begin, most key
informants and program staff as well as a few parents/caregivers discussed the benefits of focusing on one
goal during each round of ABA. As they explained, concentrating on one simple goal in the allotted two to
six month session can increase the probability of success for clients and parents/caregivers. This program
strength was seen to be a key factor in leading to many of the client and parent/caregiver outcomes, as
outlined in the outcomes section below.
“It’s one thing at a time… Just having that one goal takes away that overwhelming piece for
parents and it… it helps them kind of work on one thing.” (ABA Consultant)
“The beauty of this program is that it gets down to the small goals that are really important to
identify and work on and master.” (Key Informant)
“When [families] focus on one specific thing, it does build resiliency.” (Key Informant)
Despite the benefits of focusing on one goal, many parents/caregivers sometimes felt limited by this
approach. As they explained, their children face numerous interconnected challenges, many of which
cannot be resolved by focusing on one small goal. A few parents talked about the need for a more well-
rounded program which considers small goals while focusing on a bigger picture of change. Choosing a
specific goal was described as difficult by many parents/caregivers, especially those who are new to the
program and had not narrowed down their child’s needs before. In these cases, many parents felt
overwhelmed by the process and said they lacked direction in helping to choose their child’s goal.
Yet overall, the focus on manageable change (that is, doable given the program timeframe) seemed to be
working. Another facilitating factor was how consultants worked with parents and caregivers to teach
them skills they will need to maintain or generalize the client’s goal. By learning skills relevant to their
child’s behaviour, they were better able to work towards manageable change. As demonstrated in Chart 9,
the majority of parents/caregivers found that the training in skills/behavioural techniques helped them
with their child’s goals (87% agreed or strongly agreed). Only a small number of parents/caregivers did not
find the training useful (2%).
26. ABA Cycle Two Evaluation Report
26 | P a g e
Chart 9: The training in skills/behaviour techniques helped me with my goals (N=416)
Source: Consumer Feedback Survey (April 1, 2014 to January 21, 2015)
Furthermore, as demonstrated in Chart 10, 87% of parents/caregivers found that they were able to carry
out the strategies/techniques of the behaviour support plan (54% strongly agreed and 33% agreed).
Chart 10: I was able to carry out the procedures/strategies of the plan (N=414)
Source: Consumer Feedback Survey (April 1, 2014 to January 21, 2015)
Partnerships among ASD agencies
Finally, partnerships among ASD agencies emerged as a key strength in Cycle 2. According to ABA
consultants, clinical supervisors and several key informants, the agencies involved in the delivery of ASD
services work well together. Each Central West region formed committees of ABA and/or ASD-related
agencies and institutions that meet semi-regularly to share insights and best practices, to address service
gaps for ASD children/youth, and to strengthen regional coordination. In a few regions, the ABA program
Strongly Agree
57%
Agree
30%
Neutral
11%
Disagree
2%
Strongly Agree
54%
Agree
33%
Neutral
12% Disagree
1%
27. ABA Cycle Two Evaluation Report
27 | P a g e
was said to foster stronger relationships between agencies due to their increased interactions. Agencies
had a clearer sense of each other’s services, thereby making accurate referrals to each other’s programs
with greater regularity. To a lesser extent, there was also some evidence of effective coordination with
mental health agencies, and some examples of strong partnerships with schools and school boards.
“We can overall serve people better because we know more about these agencies.” (Key Informant)
“Working as a community is so important… We would never be able to deliver this program as one
agency.” (Key Informant)
Clinical supervisors from various agencies also meet on a regular basis, as do managers; these meetings
were considered a valuable space for knowledge exchange. Given that the program is implemented
differently across the Central West Region, clinical supervisors appreciated the opportunity to learn from
best practices employed in other agencies, and to brainstorm solutions to unique program challenges in
their jurisdiction.
“I found the clinical advisory committee meetings really wonderful… I feel like I still have a team. I
think we work nicely together.” (Clinical Supervisor)
Challenges of Program Implementation
Despite the large number of strengths associated with ABA program implementation, evaluation
participants also identified a number of challenges, many of which were highlighted in Cycle 1. The six
most significant challenges that emerged from Cycle 2 are as follows: (1) unsustainable targets (leading to
staff burnout); (2) limited program dosage (due to long wait lists and short program lengths); (3) lack of
other supports for parents (during ABA and while on the waitlist); (4) underdeveloped partnerships with
schools and mental health agencies (leading to difficulties in achieving and maintaining goals); and (5)
limitations with social skills groups (in terms of matching clients and parental involvement).
Unsustainable targets
The number one challenge with program implementation is undoubtedly the high number of children (i.e.,
targets) that consultants are expected to support each ABA cycle. As various consultants explained, the
amount of work required to provide high quality service to their assigned number of clients is
unsustainable and leads to burnout among staff. Due to their heavy workload, consultants are also unable
to provide flexibility in service that might benefit some clients. For example, even if children/youth would
benefit from having additional sessions with their consultant, consultants cannot often provide this
support due to time constraints. Similarly, if ABA sessions are missed due to illness or vacation, ABA
consultants are unable to make these up. This inflexibility has led to frustration among several evaluation
participants. For clients who need more time in service due to complex needs and/or a complex diagnosis,
consultants will extend the program. However, this adds additional pressure to their workload as there is
little flexibility in service targets to account for these circumstances. In other words, consultants are still
expected to reach their targets, even if some clients present particularly challenging cases and take more
time, energy and resources.
28. ABA Cycle Two Evaluation Report
28 | P a g e
“I have great staff who are dedicated to this program and who are very determined to provide top
quality clinical service, but they are killing themselves to do it.” (Key Informant)
“In this region, we are implementing the most clinically sound program we can. However we are
stretching our staff extremely thin in order to do so.” (Clinical Supervisor)
Part of the issue regarding consultants’ schedules is the amount of time spent on documentation. As
shown in Chart 6, ABA consultants spend more time on documentation than they do on direct client
contact. Moreover, some of this documentation is considered not very useful, by both ABA consultants
and parents/caregivers. The CANS tool, for example, was described by consultants as overly detailed (i.e.,
takes a long time to complete), yet not very helpful as the ratings are overly subjective and do not
contribute to ABA service delivery.
“The CANS has got to go… It’s not helpful for our type of service delivery… It’s very subjective and
not relevant at all, and it’s extra work; a formality. It just gets filled out and stuck in their folder.”
(ABA Consultant)
Another component of the program that takes a significant amount of time is travel. Time spent traveling
to meet clients was substantial, at an average of 8.9% of their total working hours. These and other
aspects are often not reported to the Ministry, due to standardized reporting mechanisms. According to
ABA consultants, this partial reporting gives an erroneous perception of the work required to deliver ABA
services, and could perpetuate unrealistic ministry service targets.
In sum, ABA consultants reported being overworked and seemed doubtful that they could manage their
caseloads long-term. Seeing as the program’s success has been found to partially hinge upon consistency
in consultants, there was widespread concern among evaluation participants that the ambitiousness of
ministry targets would jeopardize the quality of service delivery for ABA clients.
Limited program dosage
As found in Cycle One, another significant challenge with program implementation is the length of the
waitlist, both at initial intake and between ABA sessions. A growing number of families are participating in
the ABA program, which is a positive indication of community demand. However, because program
resources have not increased to meet the growing demand, many families are waiting up to a year and a
half for service. Chart 11 shows the number of clients on the waitlist vs. in service by agency. Clearly, there
are significantly more children on the waitlist in all agencies across the Central West Region.
29. ABA Cycle Two Evaluation Report
29 | P a g e
Chart 11: Number of clients (aged 9-17 years) on waitlist and in service by service provider on December
31, 2014 (N=1612)
Source: CWR ABA Services- Intake Management (Q4 April 1, 2014-January 21, 2015)
The waitlist was considered discouraging by various evaluation participants, especially because the
program itself is often only two to three months, and very few resources are available to families when on
the waitlist.
“The wait times…are discouraging. We need treatment for our kids so badly.” (Parent)
“The wait times are our challenge, every day.” (Key Informant)
With such long wait times, parents/caregivers reported having high expectations for the program, yet felt
that only limited gains were possible within such a short period of time. Since the first few weeks of the
program are dedicated to service planning and relationship building between the child and their
consultant, actual treatment time was seen as insufficient to have any significant impact on families or to
maintain and generalize goals. Parents also felt limited in the types of goals they could choose, as they
were not able to choose goals that take more time.
“I feel like we don’t spend enough time in getting them to generalize the skills across
environments.” (Key Informant)
Parents and caregivers of high-functioning children were especially frustrated with the short length of the
ABA program, as they are ineligible for most other ASD services and rely solely on ABA for support.
Similarly, parents and caregivers of children aging out of the system expect more from the program while
158
94 87
65
315
144
113 107
373
15 14 17 9
66
5 7 4
19
0
50
100
150
200
250
300
350
400
TotalNumberofChildren/Youth
Service Provider
On Waitlist
In Service
30. ABA Cycle Two Evaluation Report
30 | P a g e
the service is still available to them. Longer gaps between program cycles can also impede upon program
outcomes for re-referred clients, who struggle to maintain ABA goals over the long term without
professional support.
“The only problem is, my son doesn’t qualify for anything else. He never got IBI or anything, so this
is all he gets… So you get the eight weeks, and then wait almost a year until it happens again. And
he needs something on an ongoing basis, and there is nothing for him.” (Parent)
One type of support available to families after completing an ABA cycle is booster sessions. About a month
after completing service, consultants will attempt to contact their families to ask about goal maintenance.
If parents/caregivers feel that their child needs a booster session, the consultant will attempt to arrange it.
While some parents/caregivers reported receiving such a service, many others reported never being
contacted after the completion of a cycle. As they explained, in-person booster sessions rarely occurred,
and were in many cases ineffective as parents required more regular follow-up and support to maintain
goals. They generally felt positive about the idea of booster sessions, but said that in reality, they are few
and far between.
From the perspective of ABA consultants, many reported difficulties with the follow-up process, saying
that families are difficult to reach or had limited availability for in-person booster sessions. It was also
stressful for consultants to schedule these sessions whilst starting programs with new clients
“Lots of times it’s a struggle to get that [booster sessions] worked in, for a variety of reasons. I
mean the consultants have already moved on to their next group of children and youth that they’re
working with.” (Key Informant)
For consultants that did conduct follow-up over the phone, most had a positive response from families and
reported that goals were successfully maintained. Even still, consultants questioned whether families
were being honest or just appeasing them with positive feedback.
Limitations in social skills groups
Similar to Cycle 1, a final key challenge of program implementation was the difficulty around developing
relevant and meaningful social skills groups. While ABA staff considered the first-come-first-serve nature of
the wait list to be fair in many respects, it also creates challenges in forming social skill groups with well-
matched clients (i.e., similar diagnosis, gender, behavioural challenges, and program goals). In some areas
of the Central West Region, the policy was recently changed so that clients could be selected up to six
months down the wait list for social skills groups. With the wave of new clients entering the program,
however, some consultants reported that they were not familiar enough with clients to match them
effectively.
31. ABA Cycle Two Evaluation Report
31 | P a g e
Case-in-Point: Rishi’s Case Study
Unfortunately, Aarav has little understanding of what
occurred during his son Rishi’s first ABA session, as the
program did not facilitate parent observation. Rather, he
waited in the agency’s lobby until each session was complete.
While parents are now encouraged to observe some of their
children’s groups, Aarav still feels as though “when you [the
parent] are left out, all of the added value that you bring to
the session is gone”. In the end, Aarav explained:
“I’m not sure how successful that was... I was literally shut
off, so I didn’t have much idea of what was going on.... Did
I gain anything? Did [Rishi] gain anything? No idea,
whatsoever”.
Even though Aarav was provided the reports at the end of
ABA, he still found them difficult to read. As he said, “I
couldn’t tell if there was any perceptible gain or even if he
achieved the goal”.
To read Rishi’s full case study, see Appendix 1.
“This [creating social skills
groups] has been challenging
lately because newer families are
coming onto the list. So you
don’t know that client and you
don’t know that family and so
it’s very hard to pick them out
and pair them appropriately.”
(ABA Consultant)
Forming well matched girls-only groups
was reported as especially difficult, as
there are significantly fewer female
clients to select from. As well, due to the
large geographic area in places like
Halton, it was difficult to form groups
with children living in the same area.
According to several evaluation
participants, social skills groups provided
few opportunities for parents to observe
and participate. This limited parents’
understanding of the program’s process
and outcomes, as well as their ability to
help maintain and generalize skills.
“I find because we are heavy in our group supports, the parents’ participation is limited, which
impacts generalization and even just their capacity to understand the program and see that, you
know, my child is actually making gains based on the goal that was set.” (Key Informant)
Finally, a small number of families were dissatisfied with the lack of confidentiality in how feedback
regarding their child/youth was provided at agencies during social group sessions, as other families and
staff were often present in the room and could overhear the conversation.
Lack of Other Supports
Another challenge reported by most parents/caregivers as well as some key informants was the lack of
supports for families outside of the short program sessions. While a select few received resources while
on the ABA waitlist, the majority of parents and caregivers felt as though they were “off the radar”
(parent) during this time. Throughout the focus groups, they consistently expressed frustration regarding
their feelings of isolation and lack of support. Program managers and other key informants agreed that
there’s not much available for families while they wait a year or more for ABA.
Parent training and workshops have become increasingly available since the start of this evaluation, yet
are not offered in all areas of the Region, nor are they actively attended. Many parents and caregivers
cannot take the time away from their families to attend a workshop in person.
32. ABA Cycle Two Evaluation Report
32 | P a g e
Case-in-Point: Emily’s Case Study
As a single dad, John relies on services like these to
help support him with his daughter Emily’s needs.
He stated, “the waitlist is horrible. I applied when
[Emily] was 11, and she is now 14 and only getting
her second [cycle].” Both John and Lisa [the ABA
consultant] acknowledged there is a need to have
supports for parents during this time and John
recalls being able to contact Lisa outside of the
program when he needed her help or advice.
“I did have a couple of challenges after [Emily]
was discharged. And it was kind of, ‘okay, now
what do I do?’ Luckily, I had [Lisa’s] phone
number, so I called her up and said ‘hey’. And she
was very helpful, but it wasn’t something that
was covered by the program. It was something
she did on her own time. And I didn’t think that
was fair to her, and I almost felt guilty doing that
to her. But she was extremely helpful.” (John)
To read Emily’s full case study, see
Appendix 1.
“I know that there were some services that were tried…
to try and have them [parents] prepared for when they’re
coming into service, but I’m not sure. My recollection is
that they weren’t well attended, and they weren’t
accessed by families as they were waiting.” (Key
Informant)
“Unfortunately when you have a kid with, you know,
special needs and stuff… an hour is gold. You know? Like
your time is really precious, I find anyway.” (Parent)
Two examples of types of supports that parents and
caregivers desired are: (1) opportunities to talk to other
parents and share insight and support with others facing
similar challenges; and (2) opportunities to talk to
professional ASD consultants outside of the ABA setting.
Both of these supports would help parents to feel more
capable of handling day-to-day challenges, and less
overwhelmed by the ABA wait times.
One resource available to parents is Service
Coordination. However, while most families have access
to a Service Coordinator in their region, many do not
access the service and others find it unhelpful. As some
parents explained, Service Coordinators will answer specific questions related to service, but cannot
always spend time explaining the different options available, if the family is not clear in what they’re
looking for. This, in part, is due to the tremendous case load that each Service Coordinator is given. As one
key informant explained, some Service Coordinators have over 200 clients that they serve. As a result,
Service Coordinators end up largely helping families with complex cases or crisis situations, rather than
those who need help with navigation or understanding services. Consequently, ABA consultants often take
on a service coordination role for their clients.
“They [Service Coordinators] ask you, what do you need? I don’t know what I need… [So] I gave up
on them.” (Parent)
Finally, a number of parents/caregivers reported that they could not afford to purchase resources that
were recommended during service delivery (e.g. toys, books, puzzles). While some supplies were made
available at agency libraries, these were mostly adult resources. Libraries with client-friendly resources
were desired by parents and caregivers.
“The financial aspects of autism are equally as horrible as the problem with autism. Seeing people
put third mortgages on their house or cashing their RRSPs or working three jobs or whatever it
takes to get the money, it’s just a terrible thing.” (Parent)
33. ABA Cycle Two Evaluation Report
33 | P a g e
Underdeveloped partnerships with schools and mental health workers
Another challenge with program implementation is underdeveloped partnerships in many regions, both
with schools and school boards as well as various mental health agencies. This is an important challenge
as many goals cannot be attained or generalized without the cooperation of these agencies or institutions.
According to evaluation participants, schools across the Central West Region vary in the degree to which
they work with ABA consultants. For example, communication between school boards and ABA
consultants has been fairly effective in regions such as Wellington and areas of Peel. Some successful
stories were shared by evaluation participants, in which schools supported the attainment of a goal. In
cases where a child’s goal was related to behaviours at school, and schools were receptive to the ABA
program, there was often greater success in goal attainment, maintenance, and generalization. In regions
where these partnerships are less developed, however, consultants are finding that they must work with
schools on a case-by-case basis. A reluctance and even resistance to the ABA program was noted by some
evaluation participants at the school, school board and even union level.
“I stopped getting the school involved. Why bother? They will just say no.” (Parent)
“Hit and miss, I think. Some schools are willing to have conversations and some are not. Some are
willing to let you come in and observe, and some are like ‘we’re fine’.” (ABA Consultant)
According to evaluation participants, some principals, ASD itinerants, and special education teachers see
the value in connecting with ABA consultants, whereas others see it as an entrenchment on their own
treatment models and service. For example, a key informant from the school board explained that schools
in his region are generally willing to entertain conversations about collaborating with ABA consultants, and
are receptive to sharing knowledge and learning from each other.
“There has to be a respect for our [schools’] expertise and our knowledge. We do have behavioural
services, and I don’t need somebody to tell me how to do behavioural programs.” (Key Informant)
“I would say they’re receptive to it as a whole, in terms of supporting the need for ABA. In terms of
whether a worker can go in and do some of that implementation of ABA in the school or in
daycares or those settings, I’ve heard mixed messages.” (Key Informant)
ABA consultants and a few key informants noted that schools seemed to be more open to working
together when the ABA goal was behavioural in nature, especially in cases when the school faced difficulty
managing the behaviour of the child/youth. Schools were also more receptive when families were actively
working to bring the school representatives and ABA staff together.
ABA consultants and key informants from some regions also discussed the issue of underdeveloped
partnerships with mental health agencies and workers. Oftentimes, children/youth with both ASD and a
mental health issue face many unique challenges that cannot be resolved through the treatment approach
of the ABA program. ABA consultants do not have the abilities or skills to tackle a mental health concern,
which often takes precedence over the one goal of ABA. Yet in many regions, mental health workers and
ABA consultants have not been able to work collaboratively to address these challenges. A few consultants
reported that mental health workers sometimes overlook the mental health diagnosis and see it as a side
effect of ASD, and thereby assume that ABA treatment is sufficient for dual diagnosis clients. Without
34. ABA Cycle Two Evaluation Report
34 | P a g e
proper treatment, the client may ultimately end up in a crisis situation, possibly putting themselves and/or
their ABA consultant in danger. Thus, better partnerships with mental health agencies need to be
developed in many regions, so they can work together to best serve the child or youth in need.
Limited response to age transitions
Another key challenge of ABA program implementation revolved around age transitions, for children
switching agencies (i.e., at the age of 9 years), for youth entering high school, and for youth aging out of
the system (i.e., at the age of 17 years). Overall, there is an immediate need to make these transitions
more seamless, and to provide more support for families at these different stages of life.
A number of key informants reported inefficiencies in the transition for clients moving from ABA delivery
agencies serving 0-8 year olds to those serving clients between the ages of 9 to 17. For the most part,
minimal or no information is passed between agencies, which leads to confusion for families who felt that
the receiving agencies had no sense of their child/youth’s story. Staff and key informants also reported
frustration with this process, stating that consistent reports should be passed between agencies in order to
better serve clients. In response to these challenges, ABA delivery agencies have begun to discuss how this
transition could be improved.
“We don’t want parents to have to retell their stories every time… And they have to.” (Key
Informant)
Secondly, youth who are transitioning into high school also face unique challenges, and parents/caregivers
talked about the need for support from ABA during this time. For example, some parents said that their
children did really well in structured elementary school settings, but struggled upon entering high school
as they did not receive the same degree of direction when presented with recreational or free time. As
well, some parents wanted to talk to their children about issues such as puberty and romantic/sexual
relationships, but their children were uninterested or unable to speak about these topics with them.
Parents felt that the ABA program should be more proactive in guiding youth with ASD diagnoses through
these issues, as they may have different or unanticipated challenges than other children/youth.
“Going into the teenage years, it’s hard enough for a ‘normal’ kid to go through. But then [ASD] kids
who are struggling with the social – and social becomes uber huge – then it becomes even more
important to have these supports as they get older. It’s like [helping them] deal with hormones, or
opposite gender, or same gender if they’re struggling with that.” (Parent)
Thirdly, youth aging out of the ASD system struggle with this transition. ABA is only offered to youth up
until their 18th
birthday, after which there are few or no ASD services to support them. Various evaluation
participants expressed the limitations of this design, saying that many youth over the age of 17 still
desperately need services.
35. ABA Cycle Two Evaluation Report
35 | P a g e
Program Outcomes
As was found in Cycle 1, many of the anticipated outcomes that are outlined in the program logic model
(page 10) were in fact achieved. Most of the children and youth within the program reached their short-
term goals, and in this year’s evaluation we also found evidence that many experienced secondary
outcomes as a result of participating in the program. Family outcomes were not quite as positive; while
parents gained skills relevant to their child’s goal and often felt more confident in working with their child,
many still felt relatively unsupported outside of ABA. Thirdly, while some system-level outcomes were
positively reported, most evaluation participants still reported this to be the weakest outcome area in
terms of partnering with other agencies/institutions (schools, etc.), providing a stronger continuum of
support, and creating greater awareness of the ABA program itself.
This outcome section is structured similarly to the Cycle 1 report, with outcomes outlined according to the
three major outcome areas as shown in the program logic model: client (individual child/youth) outcomes,
individual family outcomes, and system-level outcomes.
Client Outcomes
Almost all children in the ABA program reached or exceeded their goal. Chart 12 shows the percentage of
clients (aged 9-17 years) across all agencies that attained each of the five levels of outcomes from the Goal
Attainment Scale (GAS). The two most common GAS outcomes were +2 (outcome much more than
expected; 37%) and +1 (outcome more than expected; 37%). In total, 97% of all clients (aged 9-17 years)
who completed the program between April 1, 2014 and December 31, 2014 reached or exceeded their
expected outcome.
Chart 12: Goal Attainment Scale (GAS) scores for children/youth aged 9-17 years (N=619)
Source: ABA Roll-Up (April 1, 2014 – December 31, 2014)
Chart 13 shows GAS scores by service provider. All regions had higher numbers of clients meeting or
exceeding goals than clients with unsatisfactory results. The most common GAS score across all regions
was +1 (six out of nine agencies). While an average of +1 might contribute to the perception that ABA
36. ABA Cycle Two Evaluation Report
36 | P a g e
programming is very successful, a few clinical supervisors explained that such a high score should not be
the norm. If the average score is +1, with a large number of children achieving +2, then programs have the
potential to be a lot more challenging and to push children to achieve more difficult goals. As they
explained, the average score should be zero (expected outcome), with +1 and +2 being less common.
Chart 13: Goal Attainment Scale score by service provider (N=619)
Source: ABA Roll-Up (April 1, 2014 – December 31, 2014)
In speaking with evaluation participants, most agreed that children and youth were achieving or exceeding
their goal(s). This was reported to be consistent across domain areas and most diagnoses.
“There’s an overwhelming amount of children that are meeting goals and making progress in their
skill development and their reduction of the maladapted behaviours.” (Key informant)
“The behaviours have lessened significantly. It’s incredible.” (Parent)
Chart 14 provides additional insight into parents’ and caregivers’ perspectives on client outcomes, as
reported in the Consumer Feedback Survey. As shown in this chart, the majority of parents (72%) either
agreed or strongly agreed that ABA goals were met.
0
10
20
30
40
50
60
70
80
90
100
110
TotalNumberofChildren(age9-17)
-2 Outcome
much less
than expected
-1 Outcome
less than
expected
0 Expected
outcome
reached
+1 Outcome
more than
expected
+2 Outcome
much more
than expected
37. ABA Cycle Two Evaluation Report
37 | P a g e
Chart 14: In my opinion, my son/daughter met his/her goal (N=415)
Source: Consumer Feedback Survey (April 1, 2014 to January 21, 2015)
Interestingly, 21% of parents/caregivers who completed the Consumer Feedback Survey felt “neutral” and
7% of parents/caregivers either disagreed or strongly disagreed that their child’s goal was met. These
numbers are higher than the 3% of clients who did not meet their goal according to the GAS. This
discrepancy demonstrates that parental or caregiver perceptions of ‘success’ (whether or not a goal was
achieved) appears to sometimes differ from the results of the GAS completed by staff. While the questions
were worded somewhat differently in these two quantitative measures, this apparent (and occasional)
discrepancy between GAS scores and parent/caregiver perspectives was further mirrored in several focus
groups. While most parents acknowledged that their child “met their goal” according to the GAS, some still
felt their child had a long way to go in order to maintain or generalize it. Some ABA consultants had similar
views, stating that the GAS scores do not always reflect what they would define as success for the client.
Chart 15 provides additional information on parent/caregiver perspectives. According to this chart, 83% of
parents agreed or strongly agreed that there was a positive change in the skills and/or behaviours
addressed in the ABA program. Thus, in comparison to data from Chart 14, more parents/caregivers (83%)
acknowledged a positive change as opposed to an achieved goal (72%).
38. ABA Cycle Two Evaluation Report
38 | P a g e
Chart 15: There has been a positive change in skills/behaviours addressed (N= 414)
Source: Consumer Feedback Survey (April 1, 2014 to January 21, 2015)
Some parents directly commented on their child’s ability to maintain or generalize the goal, post ABA.
However, because follow-up phone calls and in-person booster sessions are difficult to schedule, much of
this data is not tracked within program documentation.
“For the first ABA, he met his goals at home. But it had a ripple effect because he was less
anxious…He was also able to carry that forward into other areas.” (Parent)
Some parents/caregivers appreciated that after the program, their children exhibited fewer negative
behaviours (hitting, scratching, etc.) in relation to the goal or skill addressed. If the child, for example,
learned how to better communicate within the ABA program, than they might feel less frustrated when
interacting with people and exhibit fewer negative behaviours as a result. Parents and other evaluation
participants appreciated the preventative rather than punitive approach of ABA; ABA helps families to
reduce the incidence of negative behaviours, rather than react to these behaviours after they happen.
“I think it’s the one program that actually does hands-on work with families on how to approach a
problem… I think a lot of times families [receive] services that are consultative in nature, and I think
this is a hand-on, step-by-step approach to why this is happening, and what can be done to change
it.” (Key Informant)
Several evaluation participants reported heightened levels of self-confidence and independence in their
children following the ABA program. Some parents/caregivers reported that their child felt less stigmatized
and more “normal” when they learned to do things independently and to interact with others. According
to parents/caregivers and staff, it was very rewarding for the clients to see themselves achieve goals, and
to recognize their strengths.
“For him, it will be confidence for life.” (Parent)
39. ABA Cycle Two Evaluation Report
39 | P a g e
Case-in-Point: Eric’s Case Study
By the end of the program, Eric had achieved his first goal. He
learned how to follow a recipe and became confident in his
abilities to prepare different foods and to work with another
person – Rachel [his ABA consultant]. Eric appreciated that
Rachel “…was very enthusiastic”; his favourite part was “that I
had someone to help me with working on some areas that I
wasn’t necessarily good at”. Rachel equally enjoyed working
with Eric. As she explained:
“[Eric] warmed up quite quickly, just doing something
that he was enjoying. For [Eric], by the end, he was feeling
very confident and very proud of himself that he could do
these things he wasn’t able to do before.” (Rachel)
To read Eric’s full case study, see Appendix 1.
“To interact normally with others in society, this is a huge skill.” (Parent)
Parents and caregivers consistently expressed positive outcomes when their child had home-based
sessions. On the other hand, perceptions of outcomes from the social skills groups varied. As some
evaluation participants explained, this may be partly due to the limited opportunities for parents to
observe or participate in these groups. Those who reported a positive change in their child explained that
social groups helped their son/daughter to build friendships and socialize with others. Clients were also
generally able to apply what they learned in their social skills groups to other environments with their
peers.
“My child relates to other kids better
and uses some of the strategies to
interact with peers.” (Parent)
“My child has more confidence and is
willing to try new things.” (Parent)
“With my son, it’s reduced his anxiety.
Because if he has some relationships
with some people, then he’s less
stressed when he goes places.”
(Parent)
Secondary outcomes
One recommendation from Cycle One was to
augment GAS scores with secondary
outcomes, recognizing the limitation of GAS in
capturing the full range of client outcomes. In response, secondary outcomes were tracked in Cycle Two.
“Parents will always ask you questions [about secondary outcomes] …So, yeah, the GAS is there but we’re
doing a lot more that we’re not able to show.” (ABA Consultant)
Of the 50 clients who were reported as having secondary outcomes, five were seen to have one secondary
outcome, and 31 had two or more. In contrast, 14 clients were seen to have no secondary outcomes. In
total, there were 94 secondary outcomes reported across the 50 clients. Some examples of the most
common secondary outcomes achieved by clients during Cycle Two are listed below. For the complete list
of secondary outcomes reported, see Appendix 6.
Social/Interpersonal
Getting along better with siblings
Increased self-confidence
Initiating positive play with same-age peers
Increased eye contact
40. ABA Cycle Two Evaluation Report
40 | P a g e
Communication
Requesting items using sentences
Increased depth and complexity of conversations
Communicating effectively with new people
Using greetings consistently
Behavioural/Emotional Regulation
Decreased challenging behaviours
Increased compliance and ability to listen/compromise
Increased flexibility with new tasks
Daily Living
Increased initiative with chore completion and ability to complete independently
Better able to navigate purchasing items in stores
Able to engage in larger variety of activities
Secondary outcomes were reported for 72% of clients tracked. On average, clients acheived two secondary
outcomes. As shown in Chart 16, secondary outcomes were fairly evenly distributed across domains.
Secondary outcomes were found most commonly in the Social/Interpersonal domain (at 29% of the time)
and communication the least (21% of the time).
Chart 16: Frequency of Secondary Outcomes by Secondary Outcome Domain Area (N=94)
Source: ABA Secondary Outcomes from November 2014 – February 2015
Chart 17 shows that most of the secondary outcomes occur in the home, especially in the case of Daily
Living, in line with the nature of ABA programming. On average, clients are making considerable positive
change in all four secondary outcome domain areas but the degree to which these outcomes are exhibited
are always most considerable in the home. The school environment is where occurrences of secondary
41. ABA Cycle Two Evaluation Report
41 | P a g e
outcomes were found the least. It is important to note, that though there were only 94 secondary
outcomes reported, some of these would occur in multiple environments simultaneously and to different
degrees of success.
Chart 17: Occurrence of Secondary Outcomes by Environment (N= 94)
Source: ABA Secondary Outcomes from November 2014 – February 2015
Chart 18 shows the relationship between one’s primary domain and related secondary outcomes. There
seems to be a close relationship between the social/interpersonal and communication domain areas on
the one hand, and behavioural and daily living goals on the other. That is to say, for clients whose primary
goal was social/interpersonal, they were most likely to have secondary outcomes within the
communications domain (and vice versa). Similarly, for clients whose primary goal area was behavioural,
they were most likely to experience secondary outcomes in the daily living domain (and vice versa). Within
this sample of 50 clients 100% of those with daily living as their primary goal reported seeing secondary
outcomes. In contrast, those with behaviour as their primary goal only reported 62% of clients having
achieved any secondary outcomes.
42. ABA Cycle Two Evaluation Report
42 | P a g e
Chart 18: Frequency of Secondary Outcomes According to Primary Goal Domain:
Source: ABA Secondary Outcomes from November 2014 – February 2015
Parent/Caregiver Outcomes
Many parents/caregivers reported some positive changes in their own lives as a result of participating in
the ABA program. Several parents/caregivers expressed increased confidence in their own ability to
support their child. They had a better understanding of how to engage with their child as a result of the
program, and felt more confident that change was possible after seeing their child achieve goals. As well,
families felt more self-assured in their ability to support their children, and felt that they gained a better
understanding of how to relate to their child. These perceptions were often mirrored by ABA consultants
and other program staff.
“It is such a big help to understand my child.” (Parent)
“[I feel] less stress, less frustration. The behaviour’s reduced. But once in a while when it pops up,
I‘m more able to understand why.” (Parent)
“I really like the parent involvement piece [of the ABA program]. I think, as a parent, if you can
learn different ways of relating to your child, whatever it may be, that’s always helpful.” (Key
Informant)
With increased confidence, parents reported feeling better able to cope with their child’s behaviour. Chart
19 reveals that 74% of parents agreed that after ABA, they feel more capable of helping their child with
their new skill, or in dealing with their behaviour.
43. ABA Cycle Two Evaluation Report
43 | P a g e
Chart 19: I feel better because I am better able to cope with this skill/behaviour (N=410)
Source: Consumer Feedback Survey (April 1, 2014 to January 21, 2015)
Parent training is an important part of the ABA program and provides parents/caregivers with techniques
and ideas to help them address their child’s skills and behaviours. According to Chart 20, the majority of
parents (79%) were able to apply the techniques they were taught by ABA consultants. Moreover, several
parents who participated in this evaluation reported that they had leveraged what they had learned in ABA
to address other maladaptive behaviours and to teach their children new skills.
44. ABA Cycle Two Evaluation Report
44 | P a g e
Chart 20: I am able to use these techniques to address the skills/behaviours on my own (N=413)
Source: Consumer Feedback Survey (April 1, 2014 to January, 21 2015)
The success of the program was seen to be family-dependent. Many evaluation participants noted that
better outcomes were achieved for clients when parents were actively involved. According to program
staff, there was less likelihood that ABA goals would be maintained if the parents were unengaged in the
sessions. Yet, even parents/caregivers who actively participated in the program reported difficulty
maintaining the goals over the long-term without some additional professional support.
System-Level Outcomes
There was comparatively less data regarding system-level outcomes compared to other program
outcomes. Program stakeholders seemed to have difficulty commenting on system-level outcomes
achieved as a result of the ABA program to date. Not surprisingly, it was key informants who spoke most
about system-level outcomes, identifying continued strength in some ASD interagency coordination and
collaboration, however also acknowledging limitations in terms of partnering with external
agencies/institutions (schools, etc.), providing a stronger continuum of support, and creating greater
awareness of the ABA program itself.
According to key informants there continues to be stronger coordination evident among ASD agencies
across the Central West Region. Agencies that did not previously collaborate were now more aware of
each other’s services.
“ABA providers certainly get together to improve the program and discuss issues about the
program.” (Key Informant)
“I think our partnership is definitely working well. As I said, we meet with the clinical partners on a
monthly basis and then bi-monthly usually there is the ABA service provider meeting...[We are]
always trying to make it more seamless and transparent with families and how can we go about