124 Merton St., Suite 502
Toronto, Ontario M4S 2Z2
Telephone: (416) 469-9954
Fax: (416) 469-8487
www.cathexisconsulting.ca...
Purpose of the Presentation
 To provide an example of transformative use of
performance measurement
We will cover. . .
 Background on the Accessibility for Ontarians
with Disabilities Act
 An overview on the development ...
Background on AODA
 The Accessibility for Ontarians with Disabilities Act, 2005 (AODA)
was adopted to improve accessibili...
Questions to be Addressed
 What dimensions (health, education, recreation, etc)
should be considered when developing perf...
Key Focus
 The extent to which the standards have had a
positive impact on Ontarians with disabilities,
older adults and ...
Developing the Indicators
 Understanding the ADO Standards
 Literature review
 Consultation with people with disabiliti...
Findings from the Literature
 Not agreed-upon definition – subjective to each
individual
 Schalock and his colleagues de...
Quality of Life Research Unit, Center for Health Promotion, University of Toronto
Key Elements of a QOL Framework
 Being:...
Findings from the Consultation
 Most respondents indicated that most indicators
were important
 Provision of health care...
The Priority Sort Process
 Want you to know . . .
• What Group Priority Sort is
• Why you might use Group Priority Sort (...
What is Priority Sort?
A participatory priority-ranking exercise
It can be used to:
 Define the scope of an evaluation
 ...
What is Priority Sort?
 Has small groups of stakeholders or
“experts” rank-order specified items
 The outputs are:
a) co...
Overview of Process
The Priority Sort:
Two Step Process
Recent Group Priority Sort: Quality of Life
Indicators for People with Disabilities
Rapid Sort Outcome
Sample Instructions:
Forced Sort
Now it gets more difficult...
 Refine your sort so that there are no more than
6 cards i...
Forced Sort Outcome
Analysis
Benefits of Priority Sort
Informed decision-making;
Enhanced understanding of the topic among
participants;
Strengthene...
When to use Priority Sort
 Not the right technique if you are exploring a
new area that nobody knows much about.
 Not a ...
Recommended Indicators
 Agreement on the high level indicators
• The over-arching indicators are:
• Being treated with di...
Recommended Indicators
Customer Service
Level of confidence that health care providers can provide services in a way
that...
Recommended Indicators
Employment
Accommodation is provided in the workplace for people with
disabilities
• links to prac...
The Recommended Indicators
Information
Information about local warnings/emergencies is available to and
can be accessed b...
Recommended Indicators
Transportation
Public transportation, taxis and GO trains are equally accessible for
people with d...
Learnings from this Process
 People with disabilities are interested in participating in consultations
related to accessi...
Research Questions
 To what extent have there been changes in
quality of life as it relates to the five AODA
Standards fo...
Suggested Monitoring Method
On-line survey
Interviews
Use of existing data
Monitoring Cycle
 Baseline (2013/14)
• Recruit monitoring participants
• Survey administered and analyzed
• Interviews co...
Strengths
 Builds on the consultation for developing the
indicators – people expressed interest in being
involved on an o...
Limitations
 Participants dropping off from the survey
• Annual contact will help mitigate this
• A small token of apprec...
Questions?
Accessibility Directorate of Ontario
Charene Gillies: Charene.Gillies@ontario.ca
Cathexis Consulting
Martha McG...
Developing Performance Measures Through a Consultative Process
Upcoming SlideShare
Loading in …5
×

Developing Performance Measures Through a Consultative Process

512 views

Published on

Cathexis presentation for CES Toronto 2013 Evaluation Conference

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
512
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
12
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Developing Performance Measures Through a Consultative Process

  1. 1. 124 Merton St., Suite 502 Toronto, Ontario M4S 2Z2 Telephone: (416) 469-9954 Fax: (416) 469-8487 www.cathexisconsulting.ca Developing Performance Measures Through a Consultative Process June 2013
  2. 2. Purpose of the Presentation  To provide an example of transformative use of performance measurement
  3. 3. We will cover. . .  Background on the Accessibility for Ontarians with Disabilities Act  An overview on the development of performance indicators to measure the extent to which the Act is really making a difference  An introduction to a method that can help with the selection of indicators when there are multiple stakeholders  An overview of the monitoring methods
  4. 4. Background on AODA  The Accessibility for Ontarians with Disabilities Act, 2005 (AODA) was adopted to improve accessibility for the 1.85 million people with disabilities in Ontario  The legislation covers five domains: • Customer Service • Information and Communication • Employment • Transportation • Built Environment  Standards have been development relating to each of those domains
  5. 5. Questions to be Addressed  What dimensions (health, education, recreation, etc) should be considered when developing performance indicators to measure the AODA’s impact on people with disabilities, older adults and their families  Which indicators can best account for the diversity among people with disabilities, older adults and their families (i.e. types of disabilities, degree of disability, age)  Which indicators will best measure quality of life improvements for people with disabilities, older adults and their families  What data collection tools can be used measure the performance indicators over time and with a limited budget
  6. 6. Key Focus  The extent to which the standards have had a positive impact on Ontarians with disabilities, older adults and their families
  7. 7. Developing the Indicators  Understanding the ADO Standards  Literature review  Consultation with people with disabilities, older adults and their families • Out reach through a number of vehicles • On-line survey with 426 respondents • Focus groups in Huntsville, Toronto, Ottawa, Thunder Bay and London with a total of 58 participants • Social media blog with no response • Submission (1)  Priority Sort process including a rapid sort and a more thoughtful sort  Finalizing indicators
  8. 8. Findings from the Literature  Not agreed-upon definition – subjective to each individual  Schalock and his colleagues developed a framework that focuses on empowerment rather than disability  Canadian Index of Well-being looks at indicators for that cover all Canadians  Quality of Life Research Unit at the University of Toronto’s Center for Health Promotion: “The degree to which a person enjoys the important possibilities of his or her life”
  9. 9. Quality of Life Research Unit, Center for Health Promotion, University of Toronto Key Elements of a QOL Framework  Being: Who One Is • Physical Being • Psychological Being • Spiritual Being  Belonging: Connection with One’s Environment • Physical Belonging • Social Belonging • Community Belonging  Becoming: Achieving Personal Goals, Hopes and Aspirations • Practical Becoming • Leisure Becoming • Growth Becoming
  10. 10. Findings from the Consultation  Most respondents indicated that most indicators were important  Provision of health care services was considered to be the most important  Gave advice on wording and consolidation of indicators  Based on the findings seven high level indicators and 26 specific indicators emerged
  11. 11. The Priority Sort Process  Want you to know . . . • What Group Priority Sort is • Why you might use Group Priority Sort (what the benefits are) • When it is (and is not) appropriate to use Group Priority Sort
  12. 12. What is Priority Sort? A participatory priority-ranking exercise It can be used to:  Define the scope of an evaluation  Select performance measures/indicators  Prioritize strategic planning goals  Define a complex concept
  13. 13. What is Priority Sort?  Has small groups of stakeholders or “experts” rank-order specified items  The outputs are: a) comparative rankings: b) rich qualitative data; and c) engaged participants  Evolved out of Q Methodology and produces similar information as Delphi
  14. 14. Overview of Process
  15. 15. The Priority Sort: Two Step Process
  16. 16. Recent Group Priority Sort: Quality of Life Indicators for People with Disabilities
  17. 17. Rapid Sort Outcome
  18. 18. Sample Instructions: Forced Sort Now it gets more difficult...  Refine your sort so that there are no more than 6 cards in each category  Try to come to agreement about where each benefit should go  Use blank cards to record any other important indicators (do not sort these)  You have 25 minutes
  19. 19. Forced Sort Outcome
  20. 20. Analysis
  21. 21. Benefits of Priority Sort Informed decision-making; Enhanced understanding of the topic among participants; Strengthened community of stakeholders; Confidence in the process and the resulting decisions; and An enjoyable and engaging experience for participants.
  22. 22. When to use Priority Sort  Not the right technique if you are exploring a new area that nobody knows much about.  Not a brainstorming exercise.  Priority Sort captures subjective opinions. If you are seeking objective facts, this is not the right method.  The Priority Sort is most useful when there are divergent opinions.
  23. 23. Recommended Indicators  Agreement on the high level indicators • The over-arching indicators are: • Being treated with dignity • Feeling an integrated part of school, work and the community • One high level indicator for each Standard • Customer service: Being served in a way that maintains dignity and supports inclusion • Employment: Have access to employment consistent with experience, abilities and training • Information: Have access to information • Transportation: Able to get where you need to go, when you need to go • Design of Public Spaces: Able to move independently around the community  Reduced the 26 indicators to 12
  24. 24. Recommended Indicators Customer Service Level of confidence that health care providers can provide services in a way that takes into account the person’s disability • links back to physical being, psychological being and physical belonging in the project’s methodological framework. Educational institutions provide services in a respectful manner • links to psychological being, social belonging, practical becoming and growth becoming in the project’s methodological framework. Emergency response services are provided in a manner that takes into account the person’s disability • links to physical being, psychological being, and community belonging in the project’s methodological framework. Recreation/fitness facilities provide services are provided in a way that allow people with disabilities to use and benefit from them • links to physical being, physical belonging and leisure becoming in the project’s methodological framework.
  25. 25. Recommended Indicators Employment Accommodation is provided in the workplace for people with disabilities • links to practical becoming in the project’s methodological framework.  Managers and co-workers at all levels accept and make accommodation for people with disabilities • links to social belonging and practical becoming in the project’s methodological framework. Career development opportunities are provided in a manner that accommodates people with disabilities • This indicator links to psychological being and practical becoming in the project’s methodological framework.
  26. 26. The Recommended Indicators Information Information about local warnings/emergencies is available to and can be accessed by people with disabilities • links to physical being, community belonging and practical becoming in the project’s methodological framework. Accessibility planning includes input from people with disabilities • links to psychological being and community belonging in the project’s methodological framework.  Websites are designed so that people with disabilities can access them • links to social belonging, community belonging, and practical becoming in the project’s methodological framework.
  27. 27. Recommended Indicators Transportation Public transportation, taxis and GO trains are equally accessible for people with disabilities as for people without disabilities • Links to psychological being, and community belonging in the project’s methodological framework Design of Public Spaces People with disabilities are able to get to stores, community centres and other public facilities • This indicator links to social belonging and community belonging in the project’s methodological framework
  28. 28. Learnings from this Process  People with disabilities are interested in participating in consultations related to accessibility want to see that the ADO is taking the impact of the AODA seriously.  The most effective recruitment occurred through organizations representing or serving the population  The survey provided an excellent sense of what was important for people with disabilities and supported reaching a large number of people  The face-to-face consultation augmented this understanding plus provided input into acceptable wording.  A large number of people with disabilities have access to the technology required to participate in on-line surveys  The steps taken to accommodate people with disabilities, as part of the consultation process, was noted and appreciated by participants.  It is still challenging to find facilities that are fully accessible  When conducting consultations, it is important to set the parameters in a way that keeps the discussion focused, but does not disrespect the importance of the other issues being raised.
  29. 29. Research Questions  To what extent have there been changes in quality of life as it relates to the five AODA Standards for people with disabilities, older adults and their families?  What impact do the Accessibility Standards have on key areas of daily living areas for people with disabilities, older adults and their families?
  30. 30. Suggested Monitoring Method On-line survey Interviews Use of existing data
  31. 31. Monitoring Cycle  Baseline (2013/14) • Recruit monitoring participants • Survey administered and analyzed • Interviews conducted and analyzed • Existing data analyzed • Baseline report produced  Interim year (2014/15) • Contact with participants  Second monitoring cycle (2015/16) • Survey administered and analyzed • Interviews conducted and analyzed • Existing data analyzed • Second report produced  Interim year (2016/17) • Contact with participants  Third monitoring cycle (2017/18) • Survey administered and analyzed • Interviews conducted and analyzed • Existing data analyzed • Review indicators to ensure continued relevancy • Third report produced
  32. 32. Strengths  Builds on the consultation for developing the indicators – people expressed interest in being involved on an ongoing basis  Covers all of the selected indicators  Cost-effective  Multiple lines of inquiry  Able to measure change over time  Able to measure statistical significance of change
  33. 33. Limitations  Participants dropping off from the survey • Annual contact will help mitigate this • A small token of appreciation can encourage ongoing participation  Self-selecting so cannot be generalized to the total population • With a large enough sample size and tri-angulation with other lines of inquiry we can speak to trends
  34. 34. Questions? Accessibility Directorate of Ontario Charene Gillies: Charene.Gillies@ontario.ca Cathexis Consulting Martha McGuire: martha@cathexisconsulting.ca Melissa McGuire: melissa@cathexisconsulting.ca

×