Evaluating permanent supportive housing: Reflections on Elizabeth Street
Evaluating permanent supportivehousing: Reflections on Elizabeth StreetDr Shannon McDermott School of Social Sciences
Overview• Evidence base for permanent supportive housing• Challenges for building the evidence• The Evaluation of the Elizabeth Street Common Ground Supportive Housing Project• Ensuring evaluation resources are well spent
What is the evidence?• Pathways to Housing has been the most rigorously studied• Have high retention rates (75-88%) in the first two years• Non housing outcomes are less clear (drug & alcohol, isolation; Johnson, Parkinson and Purcell, 2012)• Research has not kept pace with new models• We need to be cautious about applying overseas research to the Australian context
Evidence hierarchy for Australian policymakers(Leigh, 2009)1. Systematic reviews of multiple randomized trials2. High quality randomized trials3. Systematic reviews of natural experiments or pre-post studies4. Quasi experiments5. Pre-post studies6. Expert opinion and theoretical conjecture
Some challenges in evaluation design• Randomised controlled trials can be costly• Require relinquishing some control to researchers• Results are quantitative and can be devoid of context• Essentially, there is no one truth out there waiting to be discovered; evidence can and should be contested
The Elizabeth Street Program • Targeted at chronically homeless and highly vulnerable people with complex needs • 131 units – 65 for formerly homeless • Includes office space for onsite support services • Partnership between Yarra CH (owns and manages the building) and Homeground (onsite support and concierge service)
Evaluation questions1. What is the Elizabeth Street project service model and to what extent does it reflect what was originally intended?2. What impact has the Elizabeth Street project had on the mental and physical health, social well being and economic participation of the formerly homeless residents?3. How efficiently have Elizabeth Street resources been used?4. What lessons have been learnt about the supported housing model: its structure, establishment and implementation?5. What lessons have been learnt about how to support previously homeless residents?
Evaluation methodsEvaluation methods Brief descriptionQuantitative data De-identified resident data from HomeGround (demographics,collection and service use, incidents), Yarra CH (tenancy issues and outcomes).analysis Health data from DOH; resident survey with small sample of residentsQualitative data Interviews with formerly homeless and low income residents,collection and HomeGround and Yarra CH staff and DHS, DoH staff.analysisCost and benefit data Financial data on the cost of the program is being collectedcollection and from key stakeholdersanalysisObservation The researchers spent time at 660 Elizabeth Street to understand how the program operates in practice.
Some reflections - NGOs• Plan to collect excellent program data and use clever data collection systems• Partner very early on with academics• Make research and evaluation part of the organization culture• Define and document what you think the program will achieve from the beginning. Develop a logic model detailing the inputs, activities, outputs and outcomes of your program.• Think through ethical issues regarding data sharing and use in research• Keep good financial records by tracking your budgeted and actual costs. This is particularly crucial for economic evaluations.
Some reflections – government funders• Involve academics earlier in program design so that sound research design is possible• Administrative data should become more widely available for research purposes.• Reconsider intellectual property restrictions to encourage data sharing and publication of results• Think about other models of funding evaluations to ensure that they remain independent from government politics
Some reflections – academics• Need to partner with governments and NGOs to produce more high quality program evaluation• We need to work on producing research that can be inform project design and operation, and to produce these outputs in a timely fashion• We need to develop more ways of implementing theory in practice, especially in relation to economic evaluation
Acknowledgements and referencesI would like to thank:• Kristy Muir and my project team• Our funders, DHS• All the residents and stakeholders who contributed to the evaluationSelected referencesJohnson, G., Parkins, S., & Parsell, C. (2012). Policy shift or program drift? Implementing Housing First in Australia. Melbourne: Australian Housing and Urban Research Institute.Leigh, A. (2009). What evidence should social policymakers use? Economic Roundup, 1, 27-43.