Your SlideShare is downloading. ×
Developing Performance Measures Through a Consultative Process
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Developing Performance Measures Through a Consultative Process

233
views

Published on

Cathexis presentation for CES Toronto 2013 Evaluation Conference

Cathexis presentation for CES Toronto 2013 Evaluation Conference


0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
233
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
8
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. 124 Merton St., Suite 502 Toronto, Ontario M4S 2Z2 Telephone: (416) 469-9954 Fax: (416) 469-8487 www.cathexisconsulting.ca Developing Performance Measures Through a Consultative Process June 2013
  • 2. Purpose of the Presentation  To provide an example of transformative use of performance measurement
  • 3. We will cover. . .  Background on the Accessibility for Ontarians with Disabilities Act  An overview on the development of performance indicators to measure the extent to which the Act is really making a difference  An introduction to a method that can help with the selection of indicators when there are multiple stakeholders  An overview of the monitoring methods
  • 4. Background on AODA  The Accessibility for Ontarians with Disabilities Act, 2005 (AODA) was adopted to improve accessibility for the 1.85 million people with disabilities in Ontario  The legislation covers five domains: • Customer Service • Information and Communication • Employment • Transportation • Built Environment  Standards have been development relating to each of those domains
  • 5. Questions to be Addressed  What dimensions (health, education, recreation, etc) should be considered when developing performance indicators to measure the AODA’s impact on people with disabilities, older adults and their families  Which indicators can best account for the diversity among people with disabilities, older adults and their families (i.e. types of disabilities, degree of disability, age)  Which indicators will best measure quality of life improvements for people with disabilities, older adults and their families  What data collection tools can be used measure the performance indicators over time and with a limited budget
  • 6. Key Focus  The extent to which the standards have had a positive impact on Ontarians with disabilities, older adults and their families
  • 7. Developing the Indicators  Understanding the ADO Standards  Literature review  Consultation with people with disabilities, older adults and their families • Out reach through a number of vehicles • On-line survey with 426 respondents • Focus groups in Huntsville, Toronto, Ottawa, Thunder Bay and London with a total of 58 participants • Social media blog with no response • Submission (1)  Priority Sort process including a rapid sort and a more thoughtful sort  Finalizing indicators
  • 8. Findings from the Literature  Not agreed-upon definition – subjective to each individual  Schalock and his colleagues developed a framework that focuses on empowerment rather than disability  Canadian Index of Well-being looks at indicators for that cover all Canadians  Quality of Life Research Unit at the University of Toronto’s Center for Health Promotion: “The degree to which a person enjoys the important possibilities of his or her life”
  • 9. Quality of Life Research Unit, Center for Health Promotion, University of Toronto Key Elements of a QOL Framework  Being: Who One Is • Physical Being • Psychological Being • Spiritual Being  Belonging: Connection with One’s Environment • Physical Belonging • Social Belonging • Community Belonging  Becoming: Achieving Personal Goals, Hopes and Aspirations • Practical Becoming • Leisure Becoming • Growth Becoming
  • 10. Findings from the Consultation  Most respondents indicated that most indicators were important  Provision of health care services was considered to be the most important  Gave advice on wording and consolidation of indicators  Based on the findings seven high level indicators and 26 specific indicators emerged
  • 11. The Priority Sort Process  Want you to know . . . • What Group Priority Sort is • Why you might use Group Priority Sort (what the benefits are) • When it is (and is not) appropriate to use Group Priority Sort
  • 12. What is Priority Sort? A participatory priority-ranking exercise It can be used to:  Define the scope of an evaluation  Select performance measures/indicators  Prioritize strategic planning goals  Define a complex concept
  • 13. What is Priority Sort?  Has small groups of stakeholders or “experts” rank-order specified items  The outputs are: a) comparative rankings: b) rich qualitative data; and c) engaged participants  Evolved out of Q Methodology and produces similar information as Delphi
  • 14. Overview of Process
  • 15. The Priority Sort: Two Step Process
  • 16. Recent Group Priority Sort: Quality of Life Indicators for People with Disabilities
  • 17. Rapid Sort Outcome
  • 18. Sample Instructions: Forced Sort Now it gets more difficult...  Refine your sort so that there are no more than 6 cards in each category  Try to come to agreement about where each benefit should go  Use blank cards to record any other important indicators (do not sort these)  You have 25 minutes
  • 19. Forced Sort Outcome
  • 20. Analysis
  • 21. Benefits of Priority Sort Informed decision-making; Enhanced understanding of the topic among participants; Strengthened community of stakeholders; Confidence in the process and the resulting decisions; and An enjoyable and engaging experience for participants.
  • 22. When to use Priority Sort  Not the right technique if you are exploring a new area that nobody knows much about.  Not a brainstorming exercise.  Priority Sort captures subjective opinions. If you are seeking objective facts, this is not the right method.  The Priority Sort is most useful when there are divergent opinions.
  • 23. Recommended Indicators  Agreement on the high level indicators • The over-arching indicators are: • Being treated with dignity • Feeling an integrated part of school, work and the community • One high level indicator for each Standard • Customer service: Being served in a way that maintains dignity and supports inclusion • Employment: Have access to employment consistent with experience, abilities and training • Information: Have access to information • Transportation: Able to get where you need to go, when you need to go • Design of Public Spaces: Able to move independently around the community  Reduced the 26 indicators to 12
  • 24. Recommended Indicators Customer Service Level of confidence that health care providers can provide services in a way that takes into account the person’s disability • links back to physical being, psychological being and physical belonging in the project’s methodological framework. Educational institutions provide services in a respectful manner • links to psychological being, social belonging, practical becoming and growth becoming in the project’s methodological framework. Emergency response services are provided in a manner that takes into account the person’s disability • links to physical being, psychological being, and community belonging in the project’s methodological framework. Recreation/fitness facilities provide services are provided in a way that allow people with disabilities to use and benefit from them • links to physical being, physical belonging and leisure becoming in the project’s methodological framework.
  • 25. Recommended Indicators Employment Accommodation is provided in the workplace for people with disabilities • links to practical becoming in the project’s methodological framework.  Managers and co-workers at all levels accept and make accommodation for people with disabilities • links to social belonging and practical becoming in the project’s methodological framework. Career development opportunities are provided in a manner that accommodates people with disabilities • This indicator links to psychological being and practical becoming in the project’s methodological framework.
  • 26. The Recommended Indicators Information Information about local warnings/emergencies is available to and can be accessed by people with disabilities • links to physical being, community belonging and practical becoming in the project’s methodological framework. Accessibility planning includes input from people with disabilities • links to psychological being and community belonging in the project’s methodological framework.  Websites are designed so that people with disabilities can access them • links to social belonging, community belonging, and practical becoming in the project’s methodological framework.
  • 27. Recommended Indicators Transportation Public transportation, taxis and GO trains are equally accessible for people with disabilities as for people without disabilities • Links to psychological being, and community belonging in the project’s methodological framework Design of Public Spaces People with disabilities are able to get to stores, community centres and other public facilities • This indicator links to social belonging and community belonging in the project’s methodological framework
  • 28. Learnings from this Process  People with disabilities are interested in participating in consultations related to accessibility want to see that the ADO is taking the impact of the AODA seriously.  The most effective recruitment occurred through organizations representing or serving the population  The survey provided an excellent sense of what was important for people with disabilities and supported reaching a large number of people  The face-to-face consultation augmented this understanding plus provided input into acceptable wording.  A large number of people with disabilities have access to the technology required to participate in on-line surveys  The steps taken to accommodate people with disabilities, as part of the consultation process, was noted and appreciated by participants.  It is still challenging to find facilities that are fully accessible  When conducting consultations, it is important to set the parameters in a way that keeps the discussion focused, but does not disrespect the importance of the other issues being raised.
  • 29. Research Questions  To what extent have there been changes in quality of life as it relates to the five AODA Standards for people with disabilities, older adults and their families?  What impact do the Accessibility Standards have on key areas of daily living areas for people with disabilities, older adults and their families?
  • 30. Suggested Monitoring Method On-line survey Interviews Use of existing data
  • 31. Monitoring Cycle  Baseline (2013/14) • Recruit monitoring participants • Survey administered and analyzed • Interviews conducted and analyzed • Existing data analyzed • Baseline report produced  Interim year (2014/15) • Contact with participants  Second monitoring cycle (2015/16) • Survey administered and analyzed • Interviews conducted and analyzed • Existing data analyzed • Second report produced  Interim year (2016/17) • Contact with participants  Third monitoring cycle (2017/18) • Survey administered and analyzed • Interviews conducted and analyzed • Existing data analyzed • Review indicators to ensure continued relevancy • Third report produced
  • 32. Strengths  Builds on the consultation for developing the indicators – people expressed interest in being involved on an ongoing basis  Covers all of the selected indicators  Cost-effective  Multiple lines of inquiry  Able to measure change over time  Able to measure statistical significance of change
  • 33. Limitations  Participants dropping off from the survey • Annual contact will help mitigate this • A small token of appreciation can encourage ongoing participation  Self-selecting so cannot be generalized to the total population • With a large enough sample size and tri-angulation with other lines of inquiry we can speak to trends
  • 34. Questions? Accessibility Directorate of Ontario Charene Gillies: Charene.Gillies@ontario.ca Cathexis Consulting Martha McGuire: martha@cathexisconsulting.ca Melissa McGuire: melissa@cathexisconsulting.ca