The document discusses the importance of evaluating outcomes in service delivery. It notes that while services often focus on outputs like clients served, evaluating outcomes is important to determine if a service is making a meaningful difference. Evaluating outcomes can improve services by providing insights into who benefits most and which program elements are most effective. Both monitoring data and evaluations are important, with evaluations helping to assess effectiveness, processes, and cost-effectiveness. The document outlines different evaluation methodologies and challenges in measuring outcomes.

1. The Importance of outcomes and
evaluation in service delivery
Yfoundations
10th July 2012
Prof Ilan Katz (SPRC)

2. Will cover
 Why outcomes are important
 Evaluating for outcomes
 Monitoring and evaluation
 Types of evaluation
 Some key methods
 Conclusions

3. Why are outcomes important
 Most services focus on outputs:
 Numbers of clients seen
 Numbers of service events
 Unit costs
 However this does not measure impacts ie
whether the service makes a difference
 can create perverse incentives.
 Outcome measurement can improve service
delivery.

4. How does evaluation improve
services
 Important to know what impact the service is
actually making on clients and why.
 Better understanding of who benefits most
from the service and which aspect of the
service is most beneficial.
 Results are fed back into the service so that
lessons can be learned about elements of
effective practice.

5. Outcomes and accountability
 Accountability increasingly focused on
outcomes rather than outputs.
 Funding decisions now rely on cost
effectiveness as well as assessment of need.
 Outcomes and evaluation can also increase
accountability to service users, as participants
and recipients.

6. Evaluation and monitoring
 Monitoring – regular collection of data which
is used for quality assurance, adherence to
standards and compliance.
 Evaluation – one off or episodic activity
focusing on effectiveness of implementation,
processes, outcomes and/or cost
effectiveness.
 Both are important and can build on each
other.

7. Theory of change or program logic
 Links inputs processes, outputs and
outcomes.
 Explains why activities should lead to
particular outcomes.
 Needs to be backed up by theory and
empirical evidence and be specific.
 Explains the link between inputs, processes
and outcomes.

8. Process evaluation
 Is service doing the things it is supposed to
do in the most efficient way possible?
 Has the program been implemented as
planned?
 Is it targeting the right people?
 What are the barriers to accessing the
service?
 How well is it working with other services?
 What are clients’ and other stakeholders’
views of the service?

9. Ultimate Outcome:
Young people find stable accommodation and are able to participate actively in society to their maximum potentioal
Intermediate Outcome:
Young people at risk of homelessness are provided with safe accommodation and given the
opportunity to address issues leading to homelessness
Impacts
For children and adolescents:
Improvement in housing situation
Outputs Safety
Situation What services will provide: Improvements in educational/vocational
counselling participation
Limited access to initial assessments Improvements in social functioning
Community specialist Inputs Temporary accommodation Improvements in financial stability
services brief interventions
$X put into youth Reduction in wait time for service
Benefits advice
High demand for service homelessness Better interaction with family
Peer support
Staffing resources Improvement in health and mental health
providing education and support to
Young people unable to
Policies – referrers For homeless Service System:
access mainstream
housing services Commonwealth and Lower numbers of homeless youth
NSW Who the services will interact with:
Young people with acute housing
Better inter-agency collaboration
Young people leaving issues Longer term engagement with vulnerable
Other services
families and becoming families and carers clients
vulnerable to exploitation other members of the broader youth
Client Group: Young
and abuse services For Referrers:
people at risk of
homelessness primary referrers and other service Clear referral pathway
partners Improvement in collaborative care
9

10. Methodologies
 Administrative data (employment, health,
housing)
 Worker assessments, case files
 Use of standard validated surveys
 Self completed surveys
 pre and post questionnaires
 Satisfaction, wellbeing, circumstances
 Qualitative methodologies/action research

11. Outcomes
 Immediate
 Satisfaction, appropriate referral
 Intermediate
 Improved wellbeing, skills, housing
 Long term
 Home ownership/stability, employment, social
networks

12. Challenges of measuring
outcomes
 When?
 Should you follow up clients?
 How?
 Questionnaire, feedback, admin data
 What?
 How to define a good outcome
 Comparison and benchmarking
 Attribution
 How do you know it was your service making the
difference?

13. Who should do the evaluation?
 In house
 Advantages
• Cheaper and more tailored to agency needs.
 Disadvantages
• Lack of credibility and independence.
• Need organisational expertise
 External
 Advantages
• Independent, authoritative – provides feedback
 Disadvantages
• Expensive and potentially burdensome, potential
embarrasment

14. Ilan Katz
Social Policy Research Centre
Matthew Gray
CAEPR, ANU
Ilan.katz@unsw.edu.au
www.sprc.unsw.edu.au
www.anu.edu.au/caepr
G2 Western Campus
University of New South Wales
Kensington 2052
NSW, Australia
+61 2 9385 7810