The Importance of outcomes and evaluation in service delivery Yfoundations 10th July 2012 Prof Ilan Katz (SPRC)
Will cover Why outcomes are important Evaluating for outcomes Monitoring and evaluation Types of evaluation Some key methods Conclusions
Why are outcomes important Most services focus on outputs: Numbers of clients seen Numbers of service events Unit costs However this does not measure impacts ie whether the service makes a difference can create perverse incentives. Outcome measurement can improve service delivery.
How does evaluation improveservices Important to know what impact the service is actually making on clients and why. Better understanding of who benefits most from the service and which aspect of the service is most beneficial. Results are fed back into the service so that lessons can be learned about elements of effective practice.
Outcomes and accountability Accountability increasingly focused on outcomes rather than outputs. Funding decisions now rely on cost effectiveness as well as assessment of need. Outcomes and evaluation can also increase accountability to service users, as participants and recipients.
Evaluation and monitoring Monitoring – regular collection of data which is used for quality assurance, adherence to standards and compliance. Evaluation – one off or episodic activity focusing on effectiveness of implementation, processes, outcomes and/or cost effectiveness. Both are important and can build on each other.
Theory of change or program logic Links inputs processes, outputs and outcomes. Explains why activities should lead to particular outcomes. Needs to be backed up by theory and empirical evidence and be specific. Explains the link between inputs, processes and outcomes.
Process evaluation Is service doing the things it is supposed to do in the most efficient way possible? Has the program been implemented as planned? Is it targeting the right people? What are the barriers to accessing the service? How well is it working with other services? What are clients’ and other stakeholders’ views of the service?
Ultimate Outcome: Young people find stable accommodation and are able to participate actively in society to their maximum potentioal Intermediate Outcome: Young people at risk of homelessness are provided with safe accommodation and given the opportunity to address issues leading to homelessness Impacts For children and adolescents: Improvement in housing situation Outputs SafetySituation What services will provide: Improvements in educational/vocational counselling participation Limited access to initial assessments Improvements in social functioningCommunity specialist Inputs Temporary accommodation Improvements in financial stabilityservices brief interventions $X put into youth Reduction in wait time for service Benefits advice High demand for service homelessness Better interaction with family Peer support Staffing resources Improvement in health and mental health providing education and support to Young people unable to Policies – referrers For homeless Service System:access mainstreamhousing services Commonwealth and Lower numbers of homeless youth NSW Who the services will interact with: Young people with acute housing Better inter-agency collaboration Young people leaving issues Longer term engagement with vulnerable Other servicesfamilies and becoming families and carers clientsvulnerable to exploitation other members of the broader youth Client Group: Youngand abuse services For Referrers: people at risk of homelessness primary referrers and other service Clear referral pathway partners Improvement in collaborative care 9
Methodologies Administrative data (employment, health, housing) Worker assessments, case files Use of standard validated surveys Self completed surveys pre and post questionnaires Satisfaction, wellbeing, circumstances Qualitative methodologies/action research
Outcomes Immediate Satisfaction, appropriate referral Intermediate Improved wellbeing, skills, housing Long term Home ownership/stability, employment, social networks
Challenges of measuringoutcomes When? Should you follow up clients? How? Questionnaire, feedback, admin data What? How to define a good outcome Comparison and benchmarking Attribution How do you know it was your service making the difference?
Who should do the evaluation? In house Advantages • Cheaper and more tailored to agency needs. Disadvantages • Lack of credibility and independence. • Need organisational expertise External Advantages • Independent, authoritative – provides feedback Disadvantages • Expensive and potentially burdensome, potential embarrasment
Ilan KatzSocial Policy Research Centre Matthew Gray CAEPR, ANU Ilan.firstname.lastname@example.org www.sprc.unsw.edu.au www.anu.edu.au/caepr G2 Western Campus University of New South Wales Kensington 2052 NSW, Australia +61 2 9385 7810