The document discusses the performance measurement and compliance reporting in Australia's homelessness assistance sector, emphasizing the need to assess the actual impact on clients' lives rather than merely tracking input and satisfaction levels. It highlights the importance of defining key outcome areas, implementing effective data collection and analysis processes, and utilizing the information for continual improvement and strategic planning. It also acknowledges the challenges and risks associated with outcome measurement systems while providing resources and recommendations for better practices.

1. Homelessness assistance sector, Australia, November 2010
Mark Planigale
Research & Consultancy
results by design

2. 2
 Performance measurement
 Compliance reporting
 Program logic

3. 3
Performance measurement
 Bigger picture of organisational performance
measurement
 Compliance reporting currently focuses on effort
 Many organisations monitor inputs and client satisfaction
 But are we making a difference?
 National frameworks: National Affordable Housing
Agreement, National Partnership Agreement on
Homelessness
 ‘Manufacturing-based’ performance models need to be
enhanced

4. 4
Program logic Mission Vision
Environment • Problem or need
• Barriers
• Enablers
Inputs Activities Outputs Outcomes Impacts
People and What gets What clients Changes in Changes in
resources done receive clients’ lives society
• Clients • Functions • Client • Short term • Social
• Human • Processes participation • Long term outcomes
resources • Actions benefits • Interim
• Other • Client direct
resources benefits
Objectives Aims

5. 5
Useful information
is produced through
well-planned systems.

6. 6
 Purpose and scope
 Defined outcomes
 Defined measures and tools
 Data collection and storage processes
 Analysis and reporting processes
 Use of outcomes information
 Strategies for sustaining the system

7. 7
Clarify purpose and scope
Why measure outcomes? Questions to consider
 Compliance  Which programs?
 Individual assessment and  Which populations?
planning  What do we mean by
 Oversight outcomes?
 Quality improvement
 Advocacy

8. 8
Define key outcome areas
Outcome: Questions to consider
“... a change or absence of  If the program works
change that results (at least in really well, how are
part) from actions of staff of clients’ lives better?
the organisation”  If things go wrong
 Desired vs. undesired for our clients, where
do we see this in
 Short term, long term, interim
their lives?
 Domains vs. locus of change
 What is our core
business?

9. 9
Example outcomes
1. Person obtains appropriate, affordable long-term
housing
2. Person takes medication more regularly
3. Person has a better understanding of tenancy rights and
responsibilities
4. Person is placed on the OOH Early Housing waiting list
5. Person gains part-time employment
6. Person becomes homeless

10. 10
Select measures and tools
Measure: Questions to consider
“...an observable, measurable  If our key outcomes
characteristic of a person or occurred, how would
their situation, which is linked we know?
with a state or condition of
interest to us”  Which measures
have the greatest
 Base measures, derived communicative
measures and KPIs power?
 Varied vantage points
 What data do we
 Use modules for flexibility and already have?
consistency

11. 11
Example measures
Status change/maintenance scale:
1. % of clients who were homeless at the end of the
period
Level of functioning scale:
2. % of clients who rate their parenting skills better at exit
than they did at entry
Goal attainment scaling:
3. % of clients who achieved better than expected
outcomes in majority of goals reviewed this period

12. 12
Example tool: Outcomes Star
Triangle Consulting / London Housing Foundation. http://www.homelessoutcomes.org.uk/The_Outcomes_Star.aspx

13. 13
Collect data
Data collection and entry: Questions to consider:
 Has a huge impact on data  Who will collect the
quality data? What training
will they require?
 Should be integrated as far as
 All clients or a sub-
possible with service delivery
sample?
 Requires client consent
 How and when will
 May need to respond flexibly data be collected?
to circumstances of client  How will data be
stored?

14. 14
Analyse and report
Outcomes reporting enablers: Questions to consider:
 A knowledge of stakeholder  Who will use the
requirements reports?
 Data analysis skills  What levels of
 Standardised: aggregation are
 calculation procedures useful?
 report templates  How can complexity
 database queries
be acknowledged?
 Access to data!

15. 15
Example: control graph
Proportion of service episodes with housing situation improved at completion
Quarterly proportion Cumulative proportion
41.3%
40% 38.4%
37.4%
Percentage of
completed service 34.5% 35.2%
34.3% 34.4%
33.3% 34.9%
episodes 31.8%
(current quarter
n = 165, 28.5%
30%
cumulative 28.6%
n = 1097)
20%
Jun Sep Dec Mar Jun Sep Dec Mar Jun
2009 2009 2009 2010 2010 2010 2010 2011 2011

16. 16
Use outcomes information
Contexts for use: Questions to consider:
 Individual casework:  How can outcomes
assessment, planning and information be
review shared with clients?
 Service review and reflection  How can staff play a
 Strategic planning: “turning the part in “giving
curve” meaning” to
 Advocacy
outcomes
information?

17. 17
Caution!
 Outcomes monitoring data by itself cannot “prove”
service effectiveness
 To build your case, triangulate with other evidence

18. 18
Sustain the system
What helps?
 Using the information – complete the loop
 Strong commitment to and endorsement of outcomes
measurement by senior management
 Performance expectations – staff to collect outcomes
data
 Ongoing resourcing
 Regular review of framework

19. 19
 Risks
 Implementation process
 The bottom line

20. 20
Risks
 Poor value for money
 Outcome measurement can be resource-intensive and
time-consuming
 Information produced may not be high quality
 Distortion of service delivery
 Staff opposition
 Data collection burden
 Feeling scrutinised
 Negative results

21. 21
Process tips
 Involve stakeholders early and ongoingly
 Board, Executive, management, service delivery staff,
clients
 Combine top-down and bottom-up elements
 Pilot locally, implement sequentially
 Typically 6 – 24 months to end of pilots
 Varies with size of organisation, complexity of services, and
level of resourcing

22. 22
The bottom line
 Client wellbeing comes first
 Client rights are respected
 Information produced is useful
 Relevant
 Reliable
 Valid
 System is affordable
 Data collection burden on staff is minimised

23. 23
Useful resources
Results-Based Accountability
 www.raguide.org
 Friedman (2009) Trying Hard Isn’t Good Enough
UK Approaches (incl. Outcomes Star)
 www.homelessoutcomes.org.uk
 Burns & Cupitt (2003) Managing Outcomes: A Guide for Homelessness Organisations
US: National Alliance to End Homelessness
 Spellman & Abbenante (2008) What Gets Measured, Gets Done: A Toolkit on
Performance Measurement for Ending Homelessness
Reference – technical and implementation issues
 Planigale (2010) Literature Review: Measurement of Client Outcomes in Homelessness
Services. http://www.homeground.org.au/assets/literature-review-measurement-of-
client-outcomes-in-homelessness-services.pdf

24. Mark Planigale
Research & Consultancy
results by design
Phone: 0429 136 596
Mail: PO Box 754, Macleod VIC 3085
Email: results@planigale.com
Web: www.planigale.com