This document summarizes the development of performance indicators to measure the impact of Ontario's Accessibility for Ontarians with Disabilities Act (AODA). It involved background research, consultation with stakeholders, and a priority sorting process. Twelve indicators across five domains (customer service, employment, information, transportation, public spaces) were recommended to capture changes in quality of life. The process highlighted that people with disabilities want to provide input and see the AODA taken seriously. It also showed that online surveys can effectively engage stakeholders if organizations represent populations.

1. 124 Merton St., Suite 502
Toronto, Ontario M4S 2Z2
Telephone: (416) 469-9954
Fax: (416) 469-8487
www.cathexisconsulting.ca
Developing Performance
Measures Through a
Consultative Process
June 2013

2. Purpose of the Presentation
 To provide an example of transformative use of
performance measurement

3. We will cover. . .
 Background on the Accessibility for Ontarians
with Disabilities Act
 An overview on the development of performance
indicators to measure the extent to which the
Act is really making a difference
 An introduction to a method that can help with
the selection of indicators when there are
multiple stakeholders
 An overview of the monitoring methods

4. Background on AODA
 The Accessibility for Ontarians with Disabilities Act, 2005 (AODA)
was adopted to improve accessibility for the 1.85 million people
with disabilities in Ontario
 The legislation covers five domains:
• Customer Service
• Information and Communication
• Employment
• Transportation
• Built Environment
 Standards have been development relating to each of those
domains

5. Questions to be Addressed
 What dimensions (health, education, recreation, etc)
should be considered when developing performance
indicators to measure the AODA’s impact on people with
disabilities, older adults and their families
 Which indicators can best account for the diversity among
people with disabilities, older adults and their families (i.e.
types of disabilities, degree of disability, age)
 Which indicators will best measure quality of life
improvements for people with disabilities, older adults and
their families
 What data collection tools can be used measure the
performance indicators over time and with a limited
budget

6. Key Focus
 The extent to which the standards have had a
positive impact on Ontarians with disabilities,
older adults and their families

7. Developing the Indicators
 Understanding the ADO Standards
 Literature review
 Consultation with people with disabilities, older adults and
their families
• Out reach through a number of vehicles
• On-line survey with 426 respondents
• Focus groups in Huntsville, Toronto, Ottawa, Thunder Bay and
London with a total of 58 participants
• Social media blog with no response
• Submission (1)
 Priority Sort process including a rapid sort and a more
thoughtful sort
 Finalizing indicators

8. Findings from the Literature
 Not agreed-upon definition – subjective to each
individual
 Schalock and his colleagues developed a
framework that focuses on empowerment rather
than disability
 Canadian Index of Well-being looks at indicators
for that cover all Canadians
 Quality of Life Research Unit at the University of
Toronto’s Center for Health Promotion: “The
degree to which a person enjoys the important
possibilities of his or her life”

9. Quality of Life Research Unit, Center for Health Promotion, University of Toronto
Key Elements of a QOL Framework
 Being: Who One Is
• Physical Being
• Psychological Being
• Spiritual Being
 Belonging: Connection with One’s Environment
• Physical Belonging
• Social Belonging
• Community Belonging
 Becoming: Achieving Personal Goals, Hopes and
Aspirations
• Practical Becoming
• Leisure Becoming
• Growth Becoming

11. Findings from the Consultation
 Most respondents indicated that most indicators
were important
 Provision of health care services was considered
to be the most important
 Gave advice on wording and consolidation of
indicators
 Based on the findings seven high level indicators
and 26 specific indicators emerged

12. The Priority Sort Process
 Want you to know . . .
• What Group Priority Sort is
• Why you might use Group Priority Sort (what
the benefits are)
• When it is (and is not) appropriate to use
Group Priority Sort

13. What is Priority Sort?
A participatory priority-ranking exercise
It can be used to:
 Define the scope of an evaluation
 Select performance measures/indicators
 Prioritize strategic planning goals
 Define a complex concept

14. What is Priority Sort?
 Has small groups of stakeholders or
“experts” rank-order specified items
 The outputs are:
a) comparative rankings:
b) rich qualitative data; and
c) engaged participants
 Evolved out of Q Methodology and
produces similar information as Delphi

15. Overview of Process

16. The Priority Sort:
Two Step Process

17. Recent Group Priority Sort: Quality of Life
Indicators for People with Disabilities

18. Rapid Sort Outcome

19. Sample Instructions:
Forced Sort
Now it gets more difficult...
 Refine your sort so that there are no more than
6 cards in each category
 Try to come to agreement about where each
benefit should go
 Use blank cards to record any other important
indicators (do not sort these)
 You have 25 minutes

20. Forced Sort Outcome

21. Analysis

22. Benefits of Priority Sort
Informed decision-making;
Enhanced understanding of the topic among
participants;
Strengthened community of stakeholders;
Confidence in the process and the resulting
decisions; and
An enjoyable and engaging experience for
participants.

23. When to use Priority Sort
 Not the right technique if you are exploring a
new area that nobody knows much about.
 Not a brainstorming exercise.
 Priority Sort captures subjective opinions. If
you are seeking objective facts, this is not the
right method.
 The Priority Sort is most useful when there are
divergent opinions.

24. Recommended Indicators
 Agreement on the high level indicators
• The over-arching indicators are:
• Being treated with dignity
• Feeling an integrated part of school, work and the community
• One high level indicator for each Standard
• Customer service: Being served in a way that maintains dignity and
supports inclusion
• Employment: Have access to employment consistent with experience,
abilities and training
• Information: Have access to information
• Transportation: Able to get where you need to go, when you need to go
• Design of Public Spaces: Able to move independently around the
community
 Reduced the 26 indicators to 12

25. Recommended Indicators
Customer Service
Level of confidence that health care providers can provide services in a way
that takes into account the person’s disability
• links back to physical being, psychological being and physical belonging in the
project’s methodological framework.
Educational institutions provide services in a respectful manner
• links to psychological being, social belonging, practical becoming and growth
becoming in the project’s methodological framework.
Emergency response services are provided in a manner that takes into
account the person’s disability
• links to physical being, psychological being, and community belonging in the
project’s methodological framework.
Recreation/fitness facilities provide services are provided in a way that allow
people with disabilities to use and benefit from them
• links to physical being, physical belonging and leisure becoming in the
project’s methodological framework.

26. Recommended Indicators
Employment
Accommodation is provided in the workplace for people with
disabilities
• links to practical becoming in the project’s methodological
framework.
Managers and co-workers at all levels accept and make
accommodation for people with disabilities
• links to social belonging and practical becoming in the project’s
methodological framework.
Career development opportunities are provided in a manner that
accommodates people with disabilities
• This indicator links to psychological being and practical becoming in
the project’s methodological framework.

27. The Recommended Indicators
Information
Information about local warnings/emergencies is available to and
can be accessed by people with disabilities
• links to physical being, community belonging and practical becoming
in the project’s methodological framework.
Accessibility planning includes input from people with disabilities
• links to psychological being and community belonging in the project’s
methodological framework.
 Websites are designed so that people with disabilities can access
them
• links to social belonging, community belonging, and practical
becoming in the project’s methodological framework.

28. Recommended Indicators
Transportation
Public transportation, taxis and GO trains are equally accessible for
people with disabilities as for people without disabilities
• Links to psychological being, and community belonging in the
project’s methodological framework
Design of Public Spaces
People with disabilities are able to get to stores, community centres
and other public facilities
• This indicator links to social belonging and community belonging in
the project’s methodological framework

29. Learnings from this Process
 People with disabilities are interested in participating in consultations
related to accessibility want to see that the ADO is taking the impact of
the AODA seriously.
 The most effective recruitment occurred through organizations
representing or serving the population
 The survey provided an excellent sense of what was important for people
with disabilities and supported reaching a large number of people
 The face-to-face consultation augmented this understanding plus
provided input into acceptable wording.
 A large number of people with disabilities have access to the technology
required to participate in on-line surveys
 The steps taken to accommodate people with disabilities, as part of the
consultation process, was noted and appreciated by participants.
 It is still challenging to find facilities that are fully accessible
 When conducting consultations, it is important to set the parameters in a
way that keeps the discussion focused, but does not disrespect the
importance of the other issues being raised.

30. Research Questions
 To what extent have there been changes in
quality of life as it relates to the five AODA
Standards for people with disabilities, older
adults and their families?
 What impact do the Accessibility Standards have
on key areas of daily living areas for people with
disabilities, older adults and their families?

31. Suggested Monitoring Method
On-line survey
Interviews
Use of existing data

32. Monitoring Cycle
 Baseline (2013/14)
• Recruit monitoring participants
• Survey administered and analyzed
• Interviews conducted and analyzed
• Existing data analyzed
• Baseline report produced
 Interim year (2014/15)
• Contact with participants
 Second monitoring cycle
(2015/16)
• Survey administered and analyzed
• Interviews conducted and analyzed
• Existing data analyzed
• Second report produced
 Interim year (2016/17)
• Contact with participants
 Third monitoring cycle
(2017/18)
• Survey administered and analyzed
• Interviews conducted and analyzed
• Existing data analyzed
• Review indicators to ensure
continued relevancy
• Third report produced

33. Strengths
 Builds on the consultation for developing the
indicators – people expressed interest in being
involved on an ongoing basis
 Covers all of the selected indicators
 Cost-effective
 Multiple lines of inquiry
 Able to measure change over time
 Able to measure statistical significance of change

34. Limitations
 Participants dropping off from the survey
• Annual contact will help mitigate this
• A small token of appreciation can encourage
ongoing participation
 Self-selecting so cannot be generalized to the
total population
• With a large enough sample size and tri-angulation
with other lines of inquiry we can speak to trends

35. Questions?
Accessibility Directorate of Ontario
Charene Gillies: Charene.Gillies@ontario.ca
Cathexis Consulting
Martha McGuire: martha@cathexisconsulting.ca
Melissa McGuire: melissa@cathexisconsulting.ca