Your SlideShare is downloading. ×
Developing Performance Measures through a Consultative Process
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Developing Performance Measures through a Consultative Process


Published on

Published in: Career, Design

  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. 124 Merton St., Suite 502Toronto, Ontario M4S 2Z2Telephone: (416) 469-9954Fax: (416) 469-8487www.cathexisconsulting.caDeveloping PerformanceMeasures Through aConsultative ProcessJune 2013
  • 2. Purpose of the Presentation To provide an example of transformative use ofperformance measurement
  • 3. We will cover. . . Background on the Accessibility for Ontarianswith Disabilities Act An overview on the development of performanceindicators to measure the extent to which theAct is really making a difference An introduction to a method that can help withthe selection of indicators when there aremultiple stakeholders An overview of the monitoring methods
  • 4. Background on AODA The Accessibility for Ontarians with Disabilities Act, 2005 (AODA)was adopted to improve accessibility for the 1.85 million peoplewith disabilities in Ontario The legislation covers five domains:• Customer Service• Information and Communication• Employment• Transportation• Built Environment Standards have been development relating to each of thosedomains
  • 5. Questions to be Addressed What dimensions (health, education, recreation, etc)should be considered when developing performanceindicators to measure the AODA’s impact on people withdisabilities, older adults and their families Which indicators can best account for the diversity amongpeople with disabilities, older adults and their families (i.e.types of disabilities, degree of disability, age) Which indicators will best measure quality of lifeimprovements for people with disabilities, older adults andtheir families What data collection tools can be used measure theperformance indicators over time and with a limitedbudget
  • 6. Key Focus The extent to which the standards have had apositive impact on Ontarians with disabilities,older adults and their families
  • 7. Developing the Indicators Understanding the ADO Standards Literature review Consultation with people with disabilities, older adults andtheir families• Out reach through a number of vehicles• On-line survey with 426 respondents• Focus groups in Huntsville, Toronto, Ottawa, Thunder Bay andLondon with a total of 58 participants• Social media blog with no response• Submission (1) Priority Sort process including a rapid sort and a morethoughtful sort Finalizing indicators
  • 8. Findings from the Literature Not agreed-upon definition – subjective to eachindividual Schalock and his colleagues developed aframework that focuses on empowerment ratherthan disability Canadian Index of Well-being looks at indicatorsfor that cover all Canadians Quality of Life Research Unit at the University ofToronto’s Center for Health Promotion: “Thedegree to which a person enjoys the importantpossibilities of his or her life”
  • 9. Quality of Life Research Unit, Center for Health Promotion, University of TorontoKey Elements of a QOL Framework Being: Who One Is• Physical Being• Psychological Being• Spiritual Being Belonging: Connection with One’s Environment• Physical Belonging• Social Belonging• Community Belonging Becoming: Achieving Personal Goals, Hopes andAspirations• Practical Becoming• Leisure Becoming• Growth Becoming
  • 10. Findings from the Consultation Most respondents indicated that most indicatorswere important Provision of health care services was consideredto be the most important Gave advice on wording and consolidation ofindicators Based on the findings seven high level indicatorsand 26 specific indicators emerged
  • 11. The Priority Sort Process Want you to know . . .• What Group Priority Sort is• Why you might use Group Priority Sort (whatthe benefits are)• When it is (and is not) appropriate to useGroup Priority Sort
  • 12. What is Priority Sort?A participatory priority-ranking exerciseIt can be used to: Define the scope of an evaluation Select performance measures/indicators Prioritize strategic planning goals Define a complex concept
  • 13. What is Priority Sort? Has small groups of stakeholders or“experts” rank-order specified items The outputs are:a) comparative rankings:b) rich qualitative data; andc) engaged participants Evolved out of Q Methodology andproduces similar information as Delphi
  • 14. Overview of Process
  • 15. The Priority Sort:Two Step Process
  • 16. Recent Group Priority Sort: Quality of LifeIndicators for People with Disabilities
  • 17. Rapid Sort Outcome
  • 18. Sample Instructions:Forced SortNow it gets more difficult... Refine your sort so that there are no more than6 cards in each category Try to come to agreement about where eachbenefit should go Use blank cards to record any other importantindicators (do not sort these) You have 25 minutes
  • 19. Forced Sort Outcome
  • 20. Analysis
  • 21. Benefits of Priority SortInformed decision-making;Enhanced understanding of the topic amongparticipants;Strengthened community of stakeholders;Confidence in the process and the resultingdecisions; andAn enjoyable and engaging experience forparticipants.
  • 22. When to use Priority Sort Not the right technique if you are exploring anew area that nobody knows much about. Not a brainstorming exercise. Priority Sort captures subjective opinions. Ifyou are seeking objective facts, this is not theright method. The Priority Sort is most useful when there aredivergent opinions.
  • 23. Recommended Indicators Agreement on the high level indicators• The over-arching indicators are:• Being treated with dignity• Feeling an integrated part of school, work and the community• One high level indicator for each Standard• Customer service: Being served in a way that maintains dignity andsupports inclusion• Employment: Have access to employment consistent with experience,abilities and training• Information: Have access to information• Transportation: Able to get where you need to go, when you need to go• Design of Public Spaces: Able to move independently around thecommunity Reduced the 26 indicators to 12
  • 24. Recommended IndicatorsCustomer ServiceLevel of confidence that health care providers can provide services in a waythat takes into account the person’s disability• links back to physical being, psychological being and physical belonging in theproject’s methodological framework.Educational institutions provide services in a respectful manner• links to psychological being, social belonging, practical becoming and growthbecoming in the project’s methodological framework.Emergency response services are provided in a manner that takes intoaccount the person’s disability• links to physical being, psychological being, and community belonging in theproject’s methodological framework.Recreation/fitness facilities provide services are provided in a way that allowpeople with disabilities to use and benefit from them• links to physical being, physical belonging and leisure becoming in theproject’s methodological framework.
  • 25. Recommended IndicatorsEmploymentAccommodation is provided in the workplace for people withdisabilities• links to practical becoming in the project’s methodologicalframework. Managers and co-workers at all levels accept and makeaccommodation for people with disabilities• links to social belonging and practical becoming in the project’smethodological framework.Career development opportunities are provided in a manner thataccommodates people with disabilities• This indicator links to psychological being and practical becoming inthe project’s methodological framework.
  • 26. The Recommended IndicatorsInformationInformation about local warnings/emergencies is available to andcan be accessed by people with disabilities• links to physical being, community belonging and practical becomingin the project’s methodological framework.Accessibility planning includes input from people with disabilities• links to psychological being and community belonging in the project’smethodological framework. Websites are designed so that people with disabilities can accessthem• links to social belonging, community belonging, and practicalbecoming in the project’s methodological framework.
  • 27. Recommended IndicatorsTransportationPublic transportation, taxis and GO trains are equally accessible forpeople with disabilities as for people without disabilities• Links to psychological being, and community belonging in theproject’s methodological frameworkDesign of Public SpacesPeople with disabilities are able to get to stores, community centresand other public facilities• This indicator links to social belonging and community belonging inthe project’s methodological framework
  • 28. Learnings from this Process People with disabilities are interested in participating in consultationsrelated to accessibility want to see that the ADO is taking the impact ofthe AODA seriously. The most effective recruitment occurred through organizationsrepresenting or serving the population The survey provided an excellent sense of what was important for peoplewith disabilities and supported reaching a large number of people The face-to-face consultation augmented this understanding plusprovided input into acceptable wording. A large number of people with disabilities have access to the technologyrequired to participate in on-line surveys The steps taken to accommodate people with disabilities, as part of theconsultation process, was noted and appreciated by participants. It is still challenging to find facilities that are fully accessible When conducting consultations, it is important to set the parameters in away that keeps the discussion focused, but does not disrespect theimportance of the other issues being raised.
  • 29. Research Questions To what extent have there been changes inquality of life as it relates to the five AODAStandards for people with disabilities, olderadults and their families? What impact do the Accessibility Standards haveon key areas of daily living areas for people withdisabilities, older adults and their families?
  • 30. Suggested Monitoring MethodOn-line surveyInterviewsUse of existing data
  • 31. Monitoring Cycle Baseline (2013/14)• Recruit monitoring participants• Survey administered and analyzed• Interviews conducted and analyzed• Existing data analyzed• Baseline report produced Interim year (2014/15)• Contact with participants Second monitoring cycle(2015/16)• Survey administered and analyzed• Interviews conducted and analyzed• Existing data analyzed• Second report produced Interim year (2016/17)• Contact with participants Third monitoring cycle(2017/18)• Survey administered and analyzed• Interviews conducted and analyzed• Existing data analyzed• Review indicators to ensurecontinued relevancy• Third report produced
  • 32. Strengths Builds on the consultation for developing theindicators – people expressed interest in beinginvolved on an ongoing basis Covers all of the selected indicators Cost-effective Multiple lines of inquiry Able to measure change over time Able to measure statistical significance of change
  • 33. Limitations Participants dropping off from the survey• Annual contact will help mitigate this• A small token of appreciation can encourageongoing participation Self-selecting so cannot be generalized to thetotal population• With a large enough sample size and tri-angulationwith other lines of inquiry we can speak to trends
  • 34. Questions?Accessibility Directorate of OntarioCharene Gillies: Charene.Gillies@ontario.caCathexis ConsultingMartha McGuire: martha@cathexisconsulting.caMelissa McGuire: