Creating an outcomes framework for your organisation
Homelessness assistance sector, Australia, November 2010 Mark Planigale Research & Consultancy results by design
2 Performance measurement Compliance reporting Program logic
3Performance measurement Bigger picture of organisational performance measurement Compliance reporting currently focuses on effort Many organisations monitor inputs and client satisfaction But are we making a difference? National frameworks: National Affordable Housing Agreement, National Partnership Agreement on Homelessness ‘Manufacturing-based’ performance models need to be enhanced
4Program logic Mission Vision Environment • Problem or need • Barriers • EnablersInputs Activities Outputs Outcomes ImpactsPeople and What gets What clients Changes in Changes inresources done receive clients’ lives society• Clients • Functions • Client • Short term • Social• Human • Processes participation • Long term outcomes resources • Actions benefits • Interim• Other • Client direct resources benefits Objectives Aims
5 Useful information is produced throughwell-planned systems.
6 Purpose and scope Defined outcomes Defined measures and tools Data collection and storage processes Analysis and reporting processes Use of outcomes information Strategies for sustaining the system
7Clarify purpose and scopeWhy measure outcomes? Questions to consider Compliance Which programs? Individual assessment and Which populations? planning What do we mean by Oversight outcomes? Quality improvement Advocacy
8Define key outcome areasOutcome: Questions to consider“... a change or absence of If the program works change that results (at least in really well, how are part) from actions of staff of clients’ lives better? the organisation” If things go wrong Desired vs. undesired for our clients, where do we see this in Short term, long term, interim their lives? Domains vs. locus of change What is our core business?
9Example outcomes1. Person obtains appropriate, affordable long-term housing2. Person takes medication more regularly3. Person has a better understanding of tenancy rights and responsibilities4. Person is placed on the OOH Early Housing waiting list5. Person gains part-time employment6. Person becomes homeless
10Select measures and toolsMeasure: Questions to consider“...an observable, measurable If our key outcomes characteristic of a person or occurred, how would their situation, which is linked we know? with a state or condition of interest to us” Which measures have the greatest Base measures, derived communicative measures and KPIs power? Varied vantage points What data do we Use modules for flexibility and already have? consistency
11Example measuresStatus change/maintenance scale:1. % of clients who were homeless at the end of the periodLevel of functioning scale:2. % of clients who rate their parenting skills better at exit than they did at entryGoal attainment scaling:3. % of clients who achieved better than expected outcomes in majority of goals reviewed this period
13Collect dataData collection and entry: Questions to consider: Has a huge impact on data Who will collect the quality data? What training will they require? Should be integrated as far as All clients or a sub- possible with service delivery sample? Requires client consent How and when will May need to respond flexibly data be collected? to circumstances of client How will data be stored?
14Analyse and reportOutcomes reporting enablers: Questions to consider: A knowledge of stakeholder Who will use the requirements reports? Data analysis skills What levels of Standardised: aggregation are calculation procedures useful? report templates How can complexity database queries be acknowledged? Access to data!
15Example: control graphProportion of service episodes with housing situation improved at completion Quarterly proportion Cumulative proportion 41.3% 40% 38.4% 37.4%Percentage ofcompleted service 34.5% 35.2% 34.3% 34.4% 33.3% 34.9%episodes 31.8%(current quartern = 165, 28.5% 30%cumulative 28.6%n = 1097) 20% Jun Sep Dec Mar Jun Sep Dec Mar Jun 2009 2009 2009 2010 2010 2010 2010 2011 2011
16Use outcomes informationContexts for use: Questions to consider: Individual casework: How can outcomes assessment, planning and information be review shared with clients? Service review and reflection How can staff play a Strategic planning: “turning the part in “giving curve” meaning” to Advocacy outcomes information?
17Caution! Outcomes monitoring data by itself cannot “prove” service effectiveness To build your case, triangulate with other evidence
18Sustain the systemWhat helps? Using the information – complete the loop Strong commitment to and endorsement of outcomes measurement by senior management Performance expectations – staff to collect outcomes data Ongoing resourcing Regular review of framework
19 Risks Implementation process The bottom line
20Risks Poor value for money Outcome measurement can be resource-intensive and time-consuming Information produced may not be high quality Distortion of service delivery Staff opposition Data collection burden Feeling scrutinised Negative results
21Process tips Involve stakeholders early and ongoingly Board, Executive, management, service delivery staff, clients Combine top-down and bottom-up elements Pilot locally, implement sequentially Typically 6 – 24 months to end of pilots Varies with size of organisation, complexity of services, and level of resourcing
22The bottom line Client wellbeing comes first Client rights are respected Information produced is useful Relevant Reliable Valid System is affordable Data collection burden on staff is minimised
23Useful resourcesResults-Based Accountability www.raguide.org Friedman (2009) Trying Hard Isn’t Good EnoughUK Approaches (incl. Outcomes Star) www.homelessoutcomes.org.uk Burns & Cupitt (2003) Managing Outcomes: A Guide for Homelessness OrganisationsUS: National Alliance to End Homelessness Spellman & Abbenante (2008) What Gets Measured, Gets Done: A Toolkit on Performance Measurement for Ending HomelessnessReference – technical and implementation issues Planigale (2010) Literature Review: Measurement of Client Outcomes in Homelessness Services. http://www.homeground.org.au/assets/literature-review-measurement-of- client-outcomes-in-homelessness-services.pdf
Mark PlanigaleResearch & Consultancyresults by designPhone: 0429 136 596Mail: PO Box 754, Macleod VIC 3085Email: email@example.comWeb: www.planigale.com