Developing Performance Measures through a Consultative Process
124 Merton St., Suite 502Toronto, Ontario M4S 2Z2Telephone: (416) 469-9954Fax: (416) 469-8487www.cathexisconsulting.caDeveloping PerformanceMeasures Through aConsultative ProcessJune 2013
Purpose of the Presentation To provide an example of transformative use ofperformance measurement
We will cover. . . Background on the Accessibility for Ontarianswith Disabilities Act An overview on the development of performanceindicators to measure the extent to which theAct is really making a difference An introduction to a method that can help withthe selection of indicators when there aremultiple stakeholders An overview of the monitoring methods
Background on AODA The Accessibility for Ontarians with Disabilities Act, 2005 (AODA)was adopted to improve accessibility for the 1.85 million peoplewith disabilities in Ontario The legislation covers five domains:• Customer Service• Information and Communication• Employment• Transportation• Built Environment Standards have been development relating to each of thosedomains
Questions to be Addressed What dimensions (health, education, recreation, etc)should be considered when developing performanceindicators to measure the AODA’s impact on people withdisabilities, older adults and their families Which indicators can best account for the diversity amongpeople with disabilities, older adults and their families (i.e.types of disabilities, degree of disability, age) Which indicators will best measure quality of lifeimprovements for people with disabilities, older adults andtheir families What data collection tools can be used measure theperformance indicators over time and with a limitedbudget
Key Focus The extent to which the standards have had apositive impact on Ontarians with disabilities,older adults and their families
Developing the Indicators Understanding the ADO Standards Literature review Consultation with people with disabilities, older adults andtheir families• Out reach through a number of vehicles• On-line survey with 426 respondents• Focus groups in Huntsville, Toronto, Ottawa, Thunder Bay andLondon with a total of 58 participants• Social media blog with no response• Submission (1) Priority Sort process including a rapid sort and a morethoughtful sort Finalizing indicators
Findings from the Literature Not agreed-upon definition – subjective to eachindividual Schalock and his colleagues developed aframework that focuses on empowerment ratherthan disability Canadian Index of Well-being looks at indicatorsfor that cover all Canadians Quality of Life Research Unit at the University ofToronto’s Center for Health Promotion: “Thedegree to which a person enjoys the importantpossibilities of his or her life”
Quality of Life Research Unit, Center for Health Promotion, University of TorontoKey Elements of a QOL Framework Being: Who One Is• Physical Being• Psychological Being• Spiritual Being Belonging: Connection with One’s Environment• Physical Belonging• Social Belonging• Community Belonging Becoming: Achieving Personal Goals, Hopes andAspirations• Practical Becoming• Leisure Becoming• Growth Becoming
Findings from the Consultation Most respondents indicated that most indicatorswere important Provision of health care services was consideredto be the most important Gave advice on wording and consolidation ofindicators Based on the findings seven high level indicatorsand 26 specific indicators emerged
The Priority Sort Process Want you to know . . .• What Group Priority Sort is• Why you might use Group Priority Sort (whatthe benefits are)• When it is (and is not) appropriate to useGroup Priority Sort
What is Priority Sort?A participatory priority-ranking exerciseIt can be used to: Define the scope of an evaluation Select performance measures/indicators Prioritize strategic planning goals Define a complex concept
What is Priority Sort? Has small groups of stakeholders or“experts” rank-order specified items The outputs are:a) comparative rankings:b) rich qualitative data; andc) engaged participants Evolved out of Q Methodology andproduces similar information as Delphi
Sample Instructions:Forced SortNow it gets more difficult... Refine your sort so that there are no more than6 cards in each category Try to come to agreement about where eachbenefit should go Use blank cards to record any other importantindicators (do not sort these) You have 25 minutes
Benefits of Priority SortInformed decision-making;Enhanced understanding of the topic amongparticipants;Strengthened community of stakeholders;Confidence in the process and the resultingdecisions; andAn enjoyable and engaging experience forparticipants.
When to use Priority Sort Not the right technique if you are exploring anew area that nobody knows much about. Not a brainstorming exercise. Priority Sort captures subjective opinions. Ifyou are seeking objective facts, this is not theright method. The Priority Sort is most useful when there aredivergent opinions.
Recommended Indicators Agreement on the high level indicators• The over-arching indicators are:• Being treated with dignity• Feeling an integrated part of school, work and the community• One high level indicator for each Standard• Customer service: Being served in a way that maintains dignity andsupports inclusion• Employment: Have access to employment consistent with experience,abilities and training• Information: Have access to information• Transportation: Able to get where you need to go, when you need to go• Design of Public Spaces: Able to move independently around thecommunity Reduced the 26 indicators to 12
Recommended IndicatorsCustomer ServiceLevel of confidence that health care providers can provide services in a waythat takes into account the person’s disability• links back to physical being, psychological being and physical belonging in theproject’s methodological framework.Educational institutions provide services in a respectful manner• links to psychological being, social belonging, practical becoming and growthbecoming in the project’s methodological framework.Emergency response services are provided in a manner that takes intoaccount the person’s disability• links to physical being, psychological being, and community belonging in theproject’s methodological framework.Recreation/fitness facilities provide services are provided in a way that allowpeople with disabilities to use and benefit from them• links to physical being, physical belonging and leisure becoming in theproject’s methodological framework.
Recommended IndicatorsEmploymentAccommodation is provided in the workplace for people withdisabilities• links to practical becoming in the project’s methodologicalframework. Managers and co-workers at all levels accept and makeaccommodation for people with disabilities• links to social belonging and practical becoming in the project’smethodological framework.Career development opportunities are provided in a manner thataccommodates people with disabilities• This indicator links to psychological being and practical becoming inthe project’s methodological framework.
The Recommended IndicatorsInformationInformation about local warnings/emergencies is available to andcan be accessed by people with disabilities• links to physical being, community belonging and practical becomingin the project’s methodological framework.Accessibility planning includes input from people with disabilities• links to psychological being and community belonging in the project’smethodological framework. Websites are designed so that people with disabilities can accessthem• links to social belonging, community belonging, and practicalbecoming in the project’s methodological framework.
Recommended IndicatorsTransportationPublic transportation, taxis and GO trains are equally accessible forpeople with disabilities as for people without disabilities• Links to psychological being, and community belonging in theproject’s methodological frameworkDesign of Public SpacesPeople with disabilities are able to get to stores, community centresand other public facilities• This indicator links to social belonging and community belonging inthe project’s methodological framework
Learnings from this Process People with disabilities are interested in participating in consultationsrelated to accessibility want to see that the ADO is taking the impact ofthe AODA seriously. The most effective recruitment occurred through organizationsrepresenting or serving the population The survey provided an excellent sense of what was important for peoplewith disabilities and supported reaching a large number of people The face-to-face consultation augmented this understanding plusprovided input into acceptable wording. A large number of people with disabilities have access to the technologyrequired to participate in on-line surveys The steps taken to accommodate people with disabilities, as part of theconsultation process, was noted and appreciated by participants. It is still challenging to find facilities that are fully accessible When conducting consultations, it is important to set the parameters in away that keeps the discussion focused, but does not disrespect theimportance of the other issues being raised.
Research Questions To what extent have there been changes inquality of life as it relates to the five AODAStandards for people with disabilities, olderadults and their families? What impact do the Accessibility Standards haveon key areas of daily living areas for people withdisabilities, older adults and their families?
Suggested Monitoring MethodOn-line surveyInterviewsUse of existing data
Monitoring Cycle Baseline (2013/14)• Recruit monitoring participants• Survey administered and analyzed• Interviews conducted and analyzed• Existing data analyzed• Baseline report produced Interim year (2014/15)• Contact with participants Second monitoring cycle(2015/16)• Survey administered and analyzed• Interviews conducted and analyzed• Existing data analyzed• Second report produced Interim year (2016/17)• Contact with participants Third monitoring cycle(2017/18)• Survey administered and analyzed• Interviews conducted and analyzed• Existing data analyzed• Review indicators to ensurecontinued relevancy• Third report produced
Strengths Builds on the consultation for developing theindicators – people expressed interest in beinginvolved on an ongoing basis Covers all of the selected indicators Cost-effective Multiple lines of inquiry Able to measure change over time Able to measure statistical significance of change
Limitations Participants dropping off from the survey• Annual contact will help mitigate this• A small token of appreciation can encourageongoing participation Self-selecting so cannot be generalized to thetotal population• With a large enough sample size and tri-angulationwith other lines of inquiry we can speak to trends