• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Literature review: measurement of client outcomes in homelessness services

Literature review: measurement of client outcomes in homelessness services



Explores a wide range of practical and theoretical issues relating to introduction of client outcomes measures in welfare / human service organisations, with a particular focus on the housing and ...

Explores a wide range of practical and theoretical issues relating to introduction of client outcomes measures in welfare / human service organisations, with a particular focus on the housing and homelessness assistance sector.



Total Views
Views on SlideShare
Embed Views



2 Embeds 5

http://www.linkedin.com 4
http://www.slideshare.net 1



Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

    Literature review: measurement of client outcomes in homelessness services Literature review: measurement of client outcomes in homelessness services Document Transcript

    • Literature Review:MeasureMent of Client outCoMesin HoMelessness serviCes january 2011Mark Planigale Service Development and Research
    • Mark Planigale Research & Consultancy results by design PO Box 754 Macleod VIC 3085 Australia Tel: 0429 136 596 Email: results@planigale.com Web: www.planigale.com HoMeground ServiceS 1a/68 Oxford St Collingwood VIC 3066 Australia Tel: (+61) 3 9288 9600 Email: research@homeground.org.au Web: www.homeground.org.auCopyright © HomeGround Services & Mark Planigale 2010
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesContents1. introduction 1 1.1 About HomeGround Services 1 1.2 About HomeGround’s Client Outcome Measures Project 1 1.3 Literature review: aim and key questions 1 1.4 Literature review: methodology and scope 2 1.5 Note on language 2 1.6 Acknowledgements 22. BenefitS and cHallengeS ofclient outcoMe MeaSureMent 3 2.1 Benefits 3 2.2 Disbenefits, challenges and risks 4 2.3 Balancing benefits and risks 53. HoMeleSSneSS Sector: aPProacHeSto defining and MeaSuring client outcoMeS 7 3.1 Australia – national outcomes and indicators 7 3.2 Victorian Department of Human Services 9 3.3 SAAP pilots: Baulderstone and Talbot (2004) 10 3.4 Results-based accountability 10 3.5 Outcomes Star 11 3.6 National Alliance to End Homelessness 12 3.7 Benefits and challenges of a sector-based approach 124. MeaSureMent of client outcoMeSaS a reSearcH activity 14 4.1 Monitoring and evaluation 14 4.2 Naturalistic and experimental approaches to research 15 4.3 Monitoring systems as single-system research designs 16 4.4 Limitations of monitoring systems 175. concePtualiSing client outcoMeS 22 5.1 General definition 22 5.2 Stakeholder perspectives 25 5.3 Outcomes as an element of program logic 25 5.4 Domains 26 5.5 Locus of change 28 5.6 Point of intervention (prevention vs. amelioration) 30 5.7 Housing and non-housing outcomes 316. MeaSureS and MeaSureMent toolS 32 6.1 Criteria for selecting measures and tools 32 6.2 How many measures? 36 6.3 Types of data and types of measures 38 6.4 Possible tools and measures 427. MeaSureMent ProceSSeS 55 7.1 Sampling 55 7.2 Consent 56 7.3 Timing and frequency 56 7.4 Gathering and recording data 57 7.5 IT systems 588. uSe of outcoMeS data 60 8.1 Reporting 60 8.2 Analysis of aggregate data 62 8.3 Using the findings 659. ProceSS of introducing outcoMe MeaSureMent 67 9.1 Stakeholder involvement 67 9.2 Key stages in implementation 67 9.3 Timeframe 69 9.4 Defining and organising measures 70 9.5 Sustaining outcome measurement systems 7010. key queStionS to reSolve in develoPingan outcoMe MeaSureMent fraMework 71referenceS 73
    • Literature Review:Measurement of Client Outcomes in Homelessness Services1. Introduction1.1 aBout HoMeground ServiceS HomeGround’s Client Outcome Measures literature review document is intended toHomeGround Services is one of Melbourne’s Project aims to establish the foundations of serve as a technical reference during theleading housing and homelessness the systems needed to monitor outcomes for process of implementing an outcomeorganisations. HomeGround’s vision is people engaged with HomeGround’s services. measurement system.to end homelessness in Melbourne, and The project activities include:HomeGround’s mission is to get people housed • Researching the types of indicators that The literature review was guided by theand keep people housed. HomeGround has a can be used to measure housing and following key questions, grouped under theunique combination of experience delivering wellbeing outcomes headings of context, system design, andhigh volume housing and homelessness • Defining outcome measures for each of implementation:services to people in crisis, providing property HomeGround’s service streamsand tenancy management services, and • Identifying or developing tools to collect Context:providing short and long-term support services client outcomes data • What are the known benefits andto people who have experienced homelessness • Providing input to the development of disbenefits or risks of introducing clientin the past. electronic data systems that can store and outcome measurement systems within a report on client outcomes data human service organisation?HomeGround is a strong supporter of ‘housing • Identifying strategies to embed outcome • What current approaches to outcomefirst’ approaches to ending homelessness. measurement in practice. measurement provide guidance or directionHomeGround has played a leading role in to HomeGrounds efforts?introducing new housing and support models The project runs from January–July 2010 and isin Victoria, including the Private Rental expected to be followed by a phase of piloting System design:Access Program and a range of Supportive and implementation of outcome measures • What types of information can be producedHousing models. HomeGround has a strong within the organisation, influenced by the by outcome measurement systems andreputation for quality service provision, findings of the project. what conclusions can this informationeffective advocacy, and for bringing private support?and community sector partners to the table This Literature Review and the accompanying • What client outcomes are of relevance toto achieve change for people experiencing discussion paper A consistent set of casework homelessness services and how can theyhomelessness. domains for HomeGround (Planigale 2010a) be conceptualised and categorised? are the key outputs of Phase 2 (background • What specific measures and measurementHomeGround has a long-standing commitment research) of the project. tools may be of relevance to homelessnessto sector development and a track record of services?involvement in significant research projects 1.3 literature review: • What are the options for data collection(see e.g. Chamberlain, Johnson et al. 2007; aiM and key queStionS processes (when should data be collected,Johnson, Gronda et al. 2008). from who, by who, and in what format)? The aim of the literature review is to gain an • How can results be presented to be1.2 aBout HoMeground’S client overview of political, theoretical and logistical of maximum benefit to clients and theoutcoMe MeaSureS Project considerations related to the introduction of organisation? client outcome measurement systems withinHomeGround is committed to ensuring a homelessness organisation in Australia. The Implementation:that its approach to ending homelessness expected benefit of the literature review is • What are the best processes foris underpinned by strong evidence. The that it will allow the process of developing and introducing outcome measurementorganisation has identified that one of the implementing measures to be well-informed, systems within organisations?most important sources of evidence is well planned, and more likely to be successfulinformation on the outcomes achieved in (avoiding known pitfalls, and enabling more The literature review document is organisedpartnership with clients. useful measures to be developed). The into sections mirroring the above questions. 1
    • Literature Review:Measurement of Client Outcomes in Homelessness Services 1.6 acknowledgeMentSThe document ends with a section For example, the area of evaluation theorysummarising the main questions that an was identified as an important input to I wish to thank a number of individuals whoorganisation may need to answer in developing the literature review, and the following have generously and insightfully shared theiran outcome measurement system. sources were selected to provide a range experience and knowledge, and have therefore of perspectives on this topic: Weiss (1972), made important contributions to the directionSome elements of the literature review are Schalock (2001), Bloom et al. (2006), and content of this literature review. (Inspecific to the context of the homelessness Wadsworth (1997), Patton (Patton 1987) and alphabetical order): Deb Batterham, Danielservice system in Victoria. However, many Rossi, Freeman et al. (1999). Clements, Yann Decourt, Sally Elizabeth,of the issues discussed appear to be general Sherrill Evans, Sue Grigg, Hellene Gronda,themes within the outcomes measurement There is a plethora of information available in Lorrinda Hamilton, George Hatvani, Elaineliterature and are of wide applicability. the area of client outcomes measurement. Due Hendrick, Heather Holst, Guy Johnson, Deb to the limited time available for the literature Keys, Sue Kimberley, Anna Love, Matthew1.4 literature review: review, many relevant areas were either visited Scott, Theresa Swanborough, Chris Talbot,MetHodology and ScoPe only briefly, or not at all. Topics of interest that Quynh-Tram Trinh, Trish Watson and Zoe Vale. were not able to be adequately pursued withinThe literature review was conducted by Mark the literature review are labelled for further HomeGround also wishes to thank thePlanigale. The research process drew on two research in this document. following partner organisations as a whole formajor sets of sources: their contribution to this project: the Australian• Review of a range of source documents Early in the literature review process, the topic Housing and Urban Research Institute, including books, journal articles, reports, of life domains was identified as a priority to Department of Human Services, Hanover government publications, web pages, and both the Client Outcome Measures Project and Welfare Services, Melbourne Citymission, measurement tools; and to other work in progress at HomeGround. This RMIT University, Royal District Nursing Service• Meetings with selected stakeholders topic was therefore given a stronger emphasis Homeless Persons Program, and Sacred Heart including HomeGround staff and within the project. The discussion paper on Mission. representatives of sector and government casework domains (Planigale 2010a) should be partner organisations. read in conjunction with this literature review document.The document review was ‘seeded’ usingthree publications already known to be of high 1.5 note on languagerelevance: Baulderstone and Talbot (2004),Spellman and Abbenante (2008) and MacKeith In this paper the term client is used toand Graham (2007). These publications, along refer to a person who uses the services of awith initial meetings with stakeholders, were community or welfare organisation. This isused to derive a set of key themes and issues in line with HomeGround Services standardto be explored. These themes form the basis of terminology. It is acknowledged that in otherthe key questions noted above. contexts, a range of other terms may be preferred including consumer, service user,From the seed publications, other relevant patient or participant.sources were identified using a snowballapproach. As the research process identifiedareas where further depth was needed,other relevant sources were found either byinternet search or through library catalogues. 2
    • Literature Review:Measurement of Client Outcomes in Homelessness Services2. Benefits and challenges of client outcome measurementThe literature identifies a range of benefits what they are doing is working, to what achieved)" (MacKeith 2007: 2). While theassociated with client outcome measurement, extent, and for which clients. It helps motivational benefits may vary acrosshowever it is also worth considering potential the organisation (or service or team) different groups of staff, organisations thatdisbenefits, challenges and risks associated to answer the question of whether it is involve staff in defining desired outcomeswith outcome measurement systems. This being successful in its mission, and can and measures often report an enthusiasticallows a well-informed decision about whether, therefore play a role in guiding decision response (e.g. Hendrick 2010b).and how, to proceed. making. While outcomes measurement by • Advocacy benefits: outcomes measurement itself cannot answer questions related to can assist in demonstrating the successful2.1 BenefitS attribution, in combination with other data results of a program or intervention, as it can potentially provide some information well as potentially demonstrating levelsThe principal benefit that tends to be cited on service effectiveness. Outcomes data of client need. This information mayin favour of client outcome measurement can also at times provide a useful input to help in the task of generating supportis that it focuses staff, organisations and formal program evaluations. (partnerships, public perception, funding).service systems on the needs, goals and • Evaluative benefits (individual level): • Knowledge building: outcomesachievements of clients. Rapp and Poertner information on individual client outcomes measurement can contribute to research(1992: 16) articulate the central tenet that can provide useful data for reflection by and evaluation, can generate hypotheses“the raison d’être of the social administrator individual staff members and clients on and questions for further research, andis client wellbeing and that the principal task individual progress and the effectiveness can contribute to drawing together learningof the manager is to facilitate that wellbeing”, of the services being provided to that from across many organisations to assistand from this derive four principles of individual. the development of evidence based serviceclient-centred management, including Creating • Assessment benefits: outcomes delivery.and maintaining the focus on the clients and measurement can provide clients, serviceclient outcomes. MacKeith (2007) notes that delivery staff, managers and Boards with One note of caution is that while these benefitsa strong focus on the desired outcomes of a an overview of how the clients’ situations are often asserted, there is little documentedservice tends to positively change the way that and needs are changing over time. This research that objectively demonstratesstaff and clients work together. Having a clear is important information that can help them (Booth and Smith 1997: 42). There isshared understanding of what the goals are to drive future service delivery at the certainly anecdotal evidence that outcomes(and a shared language for talking about them) individual and program level. measurement can have positive effects on staffcan be an important basis for working together • Quality improvement benefits: outcomes morale (Clements 2010; Hendrick 2010b), and(Spellman and Abbenante 2008: 4). measurement can drive quality there is some evidence that introduction of improvement, both by identifying what outcome measures by funders can positivelyA focus on client outcomes, together with the works, and by identifying interventions or affect service effectiveness (e.g. Wells andability to measure them in a meaningful way, approaches that are less successful and Johnson 2001: 194). Friedman et al. (Friedman,can have the following benefits (Burns and are in need of review. DeLapp et al. 2001a) cite a number of caseCupitt 2003; MacKeith 2007; MacKeith and • Motivational benefits: outcomes studies in which outcomes measurement wasGraham 2007; Spellman and Abbenante 2008): measurement can help both staff important to demonstrating the population-• Evaluative benefits (system level): and clients to recognise progress and level changes resulting from partnerships outcomes measurement assists funders to celebrate achievements. This can be a to improve wellbeing. It is unclear whether assess the effectiveness of use of public significant shift for organisations whose these change efforts were better implemented funds and to consider how it may be staff are constantly attending to the or more successful as a result of the use of targeted to maximise cost-effectiveness. hard work of service delivery. "It can be outcome measurement, although it is clear• Evaluative benefits (organisational de-motivating to always be travelling (i.e. that the use of baseline data was an important level): outcomes measurement assists focussed on delivery) and never arriving motivational tool in initiating change. organisations to understand whether (i.e. recognising that a goal has been There is clearly scope for further research on the outcomes of client outcome measurement itself. 3
    • Literature Review:Measurement of Client Outcomes in Homelessness Services2.2 diSBenefitS, The information produced may be of poor including its cost: "One needs to be cautiouscHallengeS and riSkS quality that the outcomes measurement systemPotential adverse aspects of outcomes does not consume in resources more than There may be a variety of reasons whymeasurement are grouped here under its information is worth." Hudson (1997: 78) outcomes measurement systems do notheadings that deal broadly with four categories makes a related point, noting the distortion produce useful information. Outcome measuresof impacts: that can arise if the resources put into (and measurement tools) are subject to• Resourcing impacts managing the performance of a program the same range of threats to reliability and• Staff impacts (including monitoring outcomes) exceed the validity as any other psychometric measures.• Impacts relating to the value of the resources actually provided for service delivery. Some potential limitations and sources of information produced, and bias or error that are particularly related to• Impacts on service delivery. Outcomes measurement can be difficult to the organisational context may include low sustain over time response rates (Hatry 1997: 18), administrationSee also Post, Isbell et al. (2005: 6-13). Anecdotal evidence (Clements 2010; Talbot of complex measures by staff with limited 2010) suggests that one of the greatest training or knowledge (Berman and HurtOutcomes measurement can be expensive challenges in outcomes measurement is 1997: 88), collector bias (especially whereMost authors agree that implementing sustaining the measurement systems over those responsible for ratings are the sameoutcomes measurement can be an expensive time. While appropriate resources and as those delivering the service – cf. Rossiand time-consuming process. Resources organisational focus may be provided initially, (1997: 31)), and use of ratings to achieveare required for researching and developing these may be impacted by the pressures of an instrumental purpose related to serviceoutcome statements, measures and tools; service delivery and the introduction of other delivery (e.g. to demonstrate client eligibilitytraining; developing or modifying data systems; organisational initiatives. Ongoing commitment for certain resources or services) rathercollecting and entering data; supporting staff and resourcing is required to ensure the than as an accurate reflection of the clientsand trouble-shooting; analysing and reporting systems function well. status (Hudson 1997: 77). It is also possibleon data; and ongoingly reviewing the outcomes that the selection of measures themselvessystem itself. Rapp and Poertner (1992: 107) Staff may feel threatened can be subject to dumbing down due to thecaution that managers typically underestimate It is possible that some service delivery political purposes to be served by the results.the level of resources needed for data staff may feel threatened by outcomes Segal (1997: 154-155) gives the example thatcollection and data entry, while Berman and measurement systems (Rapp and Poertner stakeholders may shy away from the use ofHurt (1997: 87) note that outcomes data 1992: 101). Staff can feel they are being hard outcomes measures as they are lesssystems are more likely to contribute valuable scrutinised. Where services are delivered to likely to show change than soft measures.information if they are adequately resourced, clients with complex needs in resource-poorand operated by trained staff. Implementing environments, there is a risk that managers Another key challenge related to informationoutcomes measurement superficially in an or frontline staff will feel they are being quality is the adequacy of electronic systemsattempt to minimise cost and avoid changes held accountable for things that are outside for storing outcomes data. Most homelessnessin organisational culture may actually be less of their control (Schalock 2001: 4, 39). It is organisations operate with a suite ofcost-effective – MacKeith (2007: 4) suggests essential that both those collecting and those inconsistent and only partially functional clientthat organisations that approach outcomes interpreting the data understand the range of data systems that are not designed to capturemeasurement in this way will not achieve the factors and constraints that affect outcomes, or analyse outcomes data or integrate it withbenefits of increased focus, motivation and including the fundamental observation that service delivery. Even with the best measures,effectiveness. outcomes are largely controlled by the client. if data systems are not available to process It is possible that introduction of outcomes the data this can lead to a breakdown in theSchalock (2001: 39) argues that despite the measurement may lead to the departure of measurement process (Hendrick 2010b).value of outcome measurement, it also needs a small number of staff who are not willingto be balanced with other considerations, to make practice changes (Smith, Rost et al. 1997: 132). 4
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesThe information produced may be used poorly staff morale implications of data indicating argue that it is important to hold serviceEven where the information produced by low levels of success (Segal 1997: 155). providers accountable for both outcomesoutcomes measurement is of high quality and Stakeholders may become dissatisfied with and process (quality assurance) – "Measuringrelevance, it is only beneficial to the extent the length of time required to achieve change one to the exclusion of the other can leadthat it is actually used to improve services for in outcomes; Wells and Johnson (2001: 195) to inappropriate practices that focus on theclients (Schalock 2001: 39). Berman and Hurt suggest that achieving change tends to take measure rather than the experience of the(1997: 87) comment that there is "no greater much longer than expected. child and family" (180). A related issue is theburden" than the collection of information potential for outcomes measurement to leadthat is not used. One danger is that the One possible response is to use risk to soft targeting or to services focusing oninformational needs of some stakeholders adjustment or casemix style approaches countable outcomes to the exclusion of less(often service delivery staff) will be ignored to balance outcomes against complexity and easily measured goals (Wells and Johnsonin outcome measurement systems (Hudson severity of presenting issues (e.g. Spellman 2001: 195).1997: 73, 76). A different problem is the use and Abbenante 2008: 41-43). However, this inof outcomes data to motivate conclusions or itself is complex and adjustment formulae tend Another potential detriment to service deliverydecisions without understanding the limitations to be controversial. is that outcome measurement results mayof the data. Booth and Smith (1997: 40) note become just another way of categorising orthat the end users of the results (often agency Adverse outcomes information can also have labelling clients, and that this can becomeor governmental decision makers) are not an effect in individual casework. Outcomes disempowering for clients. Pilon and Raginsnecessarily familiar with technical aspects of tools can show regression as well as progress (2007: 11) discuss an example of a mentalresearch or evaluation design. This can lead to (MacKeith, Graham et al. 2007: 13), and health clinic that had been rating clientsan inflated view of the reliability and validity of negative ratings by a case manager of a using the MORS recovery indicators and hadoutcomes data in demonstrating causal links clients progress, if known to the client, may begun using the milestones as a shorthandbetween programs and results. For this reason, impact on the worker-client relationship (Love way to describe consumers: “She’s a three.”among others, Wells and Johnson (2001: 193) and Scott 2010). Careful thought should be “He thinks he’s a 7, but he’s really a 5.” Theadvise caution in using outcome information to given to the way that outcomes are discussed authors express concern at the dehumanisationallocate resources. in these contexts. implied by this use of language, and suggest that attention to agency culture is importantThe information produced may reflect adversely Measurement may have adverse impacts on to ensuring that outcomes tools are used in aon services service delivery strengths-based perspective.Organisations typically hope that outcome The literature identifies two types of possiblemeasurement will demonstrate the success adverse consequences to service delivery of 2.3 Balancing BenefitS and riSkSof their work. However, as Rossi (1997: 24) outcomes measurement. One is a distortionnotes, program designers and operators often of the types of intervention provided by a Organisations considering introducinghave exaggerated expectations in terms of measurement focus on certain outcomes. outcomes measurement systems may find itthe outcomes of their services. Outcome For example, in a child protection setting, useful to weigh up the potential benefits andmeasurement may in fact suggest that a a measurement focus on length of time to risks of such systems. Although the list of risksprogram is neither effective nor efficient - family reunification (with shorted durations may appear daunting, the majority of thesewhether because the environment imposes understood as better) might lead to a pressure impacts can be mitigated through a well-severe constraints on what it is possible to on staff to return children to their families planned and well-executed implementationachieve, or because the intervention itself is while significant risks or instability still existed. process, including clear communicationmisguided. It is important that organisations Therefore, a counterbalancing measure processes. This speaks to the need forentering into outcome measurement are of return to foster care within a specified adequate resourcing for implementation,prepared to face this possibility, and have period of time should also be used (Wells and for timeframes that allow for meaningfula strategy for dealing with the political and and Johnson 2001: 191). Wells and Johnson consultation with a range of stakeholders. 5
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesA question of particular interest for communitysector organisations is whether the benefitsof outcomes measurement systems areworth the resources required. The answerto this question will depend on the contextand informational needs of each individualorganisation. However, it is useful to rememberthat:• Organisations can choose the scope of the outcomes that they choose to measure, and find a scope that represents best informational value for money• A staged approach to implementation can distribute the resource burden over time and also has other benefits, including enabling assessment of the usefulness of the initial stages prior to roll-out of subsequent parts of a package.Baulderstone and Talbot (2004: vii) weighed uppractical aspects of outcome measurement inSupported Accommodation Assistance Program(SAAP) services, concluding: “Of those thatparticipated in the pilot instrument trials,some reported positive experiences and somenegative. The project team found that wherethere was negative experience, the reasons forthis were identifiable and could be dealt with,and that there was no fundamental barrier tooutcome measurement implementation in mostinstances.” 6
    • Literature Review:Measurement of Client Outcomes in Homelessness Services3. Homelessness sector: approaches to defining and measuring client outcomes3.1 auStralia – national outcoMeS The NAHA includes population-level outcomes, outputs and performance measures; selected itemsand indicatorS are summarised in Table 1.The overall framework for tacklinghomelessness in Australia is established by Table 1: the National Affordable Housing Agreement National Affordable Housing Agreement: (NAHA) (Council of Australian Governments excerpts from outcomes, outputs and performance indicators2009a), which aims to ensure that allAustralians have access to affordable, safe Outcomes • people who are homeless or at risk of homelessness achieveand sustainable housing that contributes to sustainable housing and social inclusionsocial and economic participation. The NAHA • people are able to rent housing that meets their needsis supported by three National PartnershipAgreements: one on social housing, one Outputs • number of people who are homeless or at risk of homelessnesson Indigenous Australians living in remote who are assisted to secure and sustain their tenanciesareas, and most relevant to the current • number of people who are assisted to move from crisispaper, the National Partnership Agreement on accommodation or primary homelessness to sustainableHomelessness (NPAH) (Council of Australian accommodationGovernments 2009b). Performance indicators • proportion of low income households in rental stressThese government documents may be • proportion of Australians who are homelessrelevant to measurement of client outcomes in • proportion of people experiencing repeat periods of homelessnessindividual homelessness services in two ways:• They provide an indication of government policy focus in terms of homelessness, The NPAH is designed to contribute to the NAHA outcome People who are homeless or at risk of which is one source of guidance in terms homelessness achieve sustainable housing and social inclusion. Table 2 summarises the outcomes of the types of outcomes that services explicitly stated in the agreement; other outcomes implied in the agreements outputs; and may pursue, and how outcomes could be excerpts from the performance indicators specified in the agreement (the performance indicators framed for advocacy purposes. included here are those focused on service effectiveness rather than service quality or effort). The• They provide an indication of government performance indicators also have associated baselines and performance benchmarks: see Council interest in particular data items, and of Australian Governments (2009b: 7-8). therefore of potential future reporting requirements. While some of these are population outcomes (and therefore likely to be measured through census-type approaches rather than from agency data), others may need to be built into agency data collection. 7
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesTable 2: National Partnership Agreement on Homelessness: stated outcomes, implied outcomes and selected performance indicators Stated outcomes • Fewer people will become homeless and fewer of these will sleep rough; • Fewer people will become homeless more than once; • People at risk of or experiencing homelessness will maintain or improve connections with their families and communities, and maintain or improve their education, training or employment participation; and • People at risk of or experiencing homelessness will be supported by quality services, with improved access to sustainable housing Implied outcomes • Public and private tenants sustain their tenancies • Homeless people secure or maintain stable accommodation • Homeless people (including families) ‘stabilise their situation’ Performance indicators • Proportion of Australians who are homeless • Proportion of Australians who are experiencing primary homelessness (rough sleeping) • The number of families who maintain or secure safe and sustainable housing following family violence • Increase in the number of people exiting care and custodial settings into secure and affordable housing • Reduce the number of people exiting social housing and private rental into homelessness • The proportion of people experiencing repeat periods of homelessness • Number of young people (12 to 18 years) who are homeless or at risk of homelessness who are re-engaged with family, school and workThe Supported Accommodation Assistance Program (SAAP) has long maintained a data collectionsystem. This system does include some data items which provide an indication of statusmaintenance and change, pre- and post- engagement, in areas such as accommodation, mainsource of income, labour force status, student status and living situation of clients. However, manyof the changes achieved by clients are not reflected in SAAP data reporting (Baulderstone andTalbot 2004: 1, 5). 8
    • Literature Review:Measurement of Client Outcomes in Homelessness Services Youth refuges: Support Assessment FrameworkThe SAAP V Multilateral Agreement (Commonwealth of Australia, State of New South Wales etal. 2005: 36-40) included a set of performance indicators, many of which are concerned with A pilot is currently underway of an outcomesaccess, cost, and activities or outputs. Two of the current performance indicators and three of the measurement system for youth refuges inindicators "being considered for future implementation" can fairly be seen as outcomes indicators. Victoria. The measures, recording tool andThese are summarised in Table 3. implementation process are being developed and coordinated by Sally Elizabeth forTable 3: SAAP V Multilateral Agreement: selected performance indicators related to client outcomes DHS, with considerable input from sector stakeholders. The tool is to be piloted from Current indicators Indicators being considered for future implementation May 2010. 20. The extent to which 26. Number and proportion of clients at risk of homelessness The Support Assessment Framework (SAF) clients’ case management who were assisted by SAAP to maintain their accommodation is intended to meet a range of needs as an goals are achieved at case integrated assessment, planning, review, closure 27. Number and proportion of clients at risk of homelessness who communication and outcomes measurement were assisted by SAAP to maintain family links where their health tool. The draft tool covers a range of domains 37. Number and proportion and safety will not be endangered including income, living skills, health, mental of clients not returning to health, housing / homelessness and a series SAAP within 6 months 28. Number and proportion of people at risk of homelessness of others. For each domain, the tool has who were assisted by SAAP to obtain appropriate accommodation essentially a 5-point status maintenance and upon their exit from an institution change scale. The young person is given a rating on each scale at intake, and again at exit. Each domain also has space for brief3.2 victorian dePartMent of HuMan ServiceS qualitative comments in relation to assessmentThe Victorian Department of Human Services (DHS) has an interest in the development of of support required, support provided (actionoutcome measures for homelessness and other services. Prior to the 2009 re-organisation of taken), and any action in progress at pointthe former DHS into the current Departments of Health and Human Services, the Department of exit. The qualitative items are importanthad engaged Allen Consulting to develop a set of draft Departmental, Program and aspirational because they help to validate the ratings andoutcomes (aspirational outcomes being those requiring the input of other stakeholders, such to unpack the meaning of particular ratingsas other branches of government). Subsequent to the departmental re-organisation, these at analysis stage. Some basic demographicproposed outcomes are being revised to make them more suitable to the current focus of DHS. data is also included. The tool allows for anThe process is being driven by the DHS Central policy team. However, these outcomes are not yet index of the young persons overall situationpublicly available (Trinh 2010). There is also considerable work occurring within the Department (total of ratings at entry or exit), although thein relation to development of outputs and outcomes for the new Victorian Homelessness Strategy details of how this will be calculated are still(Homelessness 2020). Again the draft outcomes are not available for public comment at this stage, being finalised. The tool is designed to reflectalthough they will be based partly on consultation that has already occurred. the short stays (average 6 weeks) and interim outcomes expected in a youth refuge serviceProgram evaluations of homelessness services contracted by the Department also provide environment.an indication of the Departments focus of interest in terms of outcomes. Recent and currentevaluation projects (including e.g. those of Elizabeth St Common Ground Supportive Housing, SAF data for all Victorian youth refugeKensington Redevelopment, J2SI, YHAP2) have focused on housing stability, social inclusion clients will be collected by agencies in an(including participation in Employment, Education and Training), and physical and mental health. Excel spreadsheet template. The data willWhere children are involved (for example in evaluations of neighbourhood renewal programs), be deidentified, aggregated and analysed oneducational continuity, retention and attainment are also of interest (Trinh 2010). a quarterly basis by DHS. At this stage the estimate is that on average 1000 records of 9
    • Literature Review:Measurement of Client Outcomes in Homelessness Servicesdata per quarter will be gathered. In addition The main outcome measurement approaches 3.4 reSultS-BaSed accountaBilityto the sector-level statistics, it is intended trialled were Goal Attainment Scaling and Results-based accountability is a high-profilethat SAF will be useful at a variety of levels Standard Goal Scaling. Several services trialled approach to managing for results thatfrom individual client self-assessment, through level of functioning scales and the BT Generic originated in the United States. In its broadworker use in case planning and review, to Outcomes Scale. Other approaches and tools sense the term implies at a minimum thatsupervision and agency-level reflection on were considered (a set of specific Status “expected results are clearly articulated andoutputs and outcomes. In this way the data Maintenance and Change scales; pre-existing that data are regularly collected and reportedgathered may also be used by organisations to standardised scales), however these were not to determine whether results have beenfacilitate practice change. pursued further as consultation suggested they achieved” (Weiss 1997: 174). Results-based did not meet the needs of the stakeholders accountability has a particular emphasis on3.3 SaaP PilotS: BaulderStone and or provide a good fit with the service delivery using outcomes definitions and measurementstalBot (2004) context (Baulderstone and Talbot 2004: 9-10). to focus service provision and to leverage collaboration among human services agenciesAs noted above, SAAP performance Mission Australia and a broad range of partners who have themeasurement has primarily concentrated on During and following the Baulderstone and potential to impact on a problem.equity and efficiency rather than on client Talbot study, a range of Mission Australiaoutcomes. However, Baulderstone and Talbot services were involved in outcomes Friedman’s approach(2004) conducted and reported on a pilot measurement pilots (Clements 2010; One of the best known proponents ofproject that tested the applicability of a variety Talbot 2010). All Mission Australia NSW/ results-based accountability is Mark Friedman,of outcome measurement tools across a range ACT Community Services were expected to who along with collaborators has developed aof SAAP service types. The intention was to complete an outcomes measurement project in particular approach to applying results-baseddevelop a system of outcome measures that 2004/5; services in other parts of Australia also accountability (Friedman, DeLapp et al. 2001b;would be acceptable to SAAP agencies and that participated in pilots. A Tool Kit was developed Friedman 2005). Results are understood aswould allow aggregation of outcomes data to to support these projects (Mission Australia “population conditions of wellbeing for children,state and national levels. The project examined 2005). Routine outcomes measurement was adults, families and communities”; results databoth housing and non-housing outcomes. also seen as feeding into formal service provides information about whether the efforts evaluation processes and reflection on practice being made to achieve results are succeedingThe project found that outcomes measurement (Mission Australia 2006: 11). (Friedman 2005: 11-13).is useful at a number of levels (client and workeruse in case management; service management; The measurement approaches and tools The core principles of results-basedprogram planning). However, the authors promoted through the Tool Kit were Goal accountability are (Friedman, DeLapp et al.concluded that due to the great diversity in Attainment Scaling, Standard Goal Scaling and 2001b: 1.1):SAAP service types and service delivery across the BT Generic Outcomes Scale. The Tool Kit 1. Start with ends, work backward to means.Australia, no one outcomes tool was applicable includes an extensive list of goals related to What do we want? How will we recogniseto all SAAP services (or even to all services SAAP support areas but tailored to Mission it? What will it take to get there?within a service type) (2004: 37). Given this, Australia’s services. 2. Be clear and disciplined about language.the authors suggested that the needs of client 3. Use plain language, not exclusionaryservice workers and managers need to be given For further research: insights of Mission jargon.priority in decisions on which forms of outcome Australia from their experience of the outcome 4. Keep accountability for populationsmeasurement are undertaken. Another key measurement pilots and subsequent work in separate from accountability for programsfinding was that staff skills and training were performance measurement and agencies.critical to the use of outcomes tools and the a. Results are end conditions of wellbeingcollection of reliable and valid data. for populations: children, adults, families and communities. 10
    • Literature Review:Measurement of Client Outcomes in Homelessness Services Melbourne Citymission: MORF 3.5 outcoMeS Star b. Customer or client results are end conditions of wellbeing for customers of a Melbourne Citymission (MCm) began The Outcomes Star approach was developed program or agency. introducing an outcomes measurement in the United Kingdom. The first version of the5. Use data (indicators and performance system in 2008. The organisation had already star tool was developed by Triangle Consulting measures) to gauge success or failure identified results-based accountability as the in 2003 for St Mungo’s, a London-based against a baseline. framework within which it wanted to develop homelessness agency (MacKeith, Burns et6. Use data to drive a disciplined business-like its outcomes systems. Initially, Friedman’s al. 2008a: 7). Subsequently the star tool has decision making process to get better. approach was piloted with seven services evolved into a suite of related tools designed7. Involve a broad set of partners. from MCm’s Homelessness and Children & for use in different sectors: the Homelessness8. Get from talk to action as quickly as Disability Services portfolios. The pilot was Star (MacKeith, Burns et al. 2008b), the Mental possible highly successful in engaging staff and led to Health Recovery Star (MacKeith and Burns a decision to roll out the Measuring Outcomes 2008), the Alcohol Star (Burns and MacKeithOne of the key concepts of results-based and Results Framework (MORF) across the 2009a), and the Work Star (Burns andaccountability is that of baselines and organisation from mid-2009 (Hendrick 2010a). MacKeith 2009b).of “turning the curve”. A baseline is arepresentation of the current state of affairs - During the pilot MCm identified the need to All of the tools are based on a commonboth a historical picture (“where we’ve been”), augment Friedman’s framework in several ways approach which includes:and a forecast (“where we’re heading if we (Hendrick 2010b): • An explicit ‘journey of change’ modeldon’t do something different”) (Friedman 2005: • Friedman’s work tends to assume that the with ten steps (ranging from “stuck” to28). The expression ‘turning the curve’ is a way aims and objectives of services are already “self-reliance”)of describing the desired change in a particular clear and can provide a starting point for • A set of life domains, which vary from toolcondition with comparison to the baseline. defining client results – in fact, this is not to tool but with substantial similaritiesTurning the curve involves doing better than always the case. The process of discussing • A single measure in each domain,the baseline forecast. One of the strengths of outcomes with services needs to include a reflecting a global measure of the person’sFriedman’s approach is that it can be strongly stage of clarifying program aims “relationship” with that domain, includingmotivating in bringing people together to • Processes for collecting, storing and how well they are managing needs oridentify which curves are the highest priority to analysing data need to be defined. Due to problems in that domainturn, and then in working in partnership to “beat the complexity and diversity of the data • A common visual presentation for thethe baseline” on these curves (Cunningham- systems in operation within MCm (as within measurement tool and supportingSmith 2010). most large human services organisations), information, using a star-shaped this is a major challenge. arrangement of axes representing differentFriedman’s approach involves clear processes domains, and a ladder representing stagesfor “getting from talk to action”, and has a clear MCm continue to work on many areas of the of changeframework for identifying the most important implementation of MORF, including working • The intention that the tool be integratedtypes of performance measures that may be with individual services on defining desired with casework processes and that ratingsused to monitor progress (Friedman 2005: 72). outcomes, standardising outcome statements be jointly agreed through discussionHowever, it does not dictate any particular set within portfolios, and developing data systems. between client and worker.of outcome measures and for this reason could At this point, the expectation is that a set ofbe flexibly combined with other theoretical standardised core outcomes will be defined The various versions of the outcomes starand practical approaches to outcomes for each portfolio; services within the portfolio have been piloted and are currently beingmeasurement. For example, it would be will be expected to work towards one or more used in a range of services, mainly in the U.K.possible for an organisation to monitor a range of these core outcomes, but may also have A web-based electronic data system has alsoof outcomes, but to use Friedman’s processes additional service-specific outcomes that are of been developed to capture and analyse dataduring planning, drawing on the monitoring data importance in their context (Hendrick 2010b). produced from the star tool.to drive a focus on particular results areas. 11
    • Literature Review:Measurement of Client Outcomes in Homelessness Services 3.7 BenefitS and cHallengeS of aMajor strengths of the outcomes star context of the problem or need that the Sector-BaSed aPProacHinclude its thorough piloting process program is designed to address, and theand appropriateness to the context of activities and outputs of the service One of the questions that hovers aroundhomelessness services. Potential limitations • A clear distinction between outputs and outcomes measurement in the homeless sectorare its focus on ‘soft’ rather than ‘hard’ outcomes is the extent to which it is possible to have anoutcomes (for example, it would not provide • Analysis of outcomes within three distinct integrated approach to outcome measurementa count of status change in respect to clients’ time frames (short-term, medium-term and across the sector. There are potential benefitshousing situations), and the limited depth of long-term) to an integrated approach, specifically:measurement in any one domain. However, • A clear definition of the target population • The ability to aggregate data to servicethere may be potential to complement the for a particular desired outcome (certain system levels and to reflect on the‘soft’ measures included in the tool with goals may only apply to a subpopulation effectiveness of the service system as aadditional measures that help to fill out the such as persons with disabilities, or to wholepicture of change for individuals. those who have achieved earlier goals) • The ability to compare the performance • A specific formula for calculating an actual of services and agencies, and to therebyMicah Projects, Brisbane outcome: the number of persons who gain an insight into factors influencingMicah Projects in Brisbane began piloting the achieved the desired goal, divided by the effectivenessHomelessness Star in 2009 (Stevens 2009). total number of persons in the target • Minimising data collection impositions onInitially the tool was trialled with one small population (i.e. percentage of population individual agencies and clients.team, with positive reactions from staff. achieving the goal)Training in the use of the tool was provided • Setting meaningful outcome targets. However, there are also complexitiesfor most of Micah’s staff in the second half of associated with implementing a sector-wide2009. Micah also intended to trial adapting the The NAEH approach is designed for use approach; these are briefly discussed here.tool for use with homeless parents, adding an by funders and communities as well as byadditional domain around parenting. service delivery organisations; it includes Developing common outcomes and measures suggested methods for comparing outcomes It is unclear to what extent desired outcomesFor further research: benefits and challenges across services (including risk adjustment are common to all stakeholders (services,of implementation of tool at Micah Projects approaches), and for examining population- funders, clients) within the homelessness level changes. sector, and to what extent they differ across3.6 national alliance to end service types and client groups. BaulderstoneHoMeleSSneSS A strength of the NAEH approach is that it and Talbot (2004: 37) found that no one provides clear mechanisms for aggregation outcomes tool was applicable to all SAAPThe United States-based National Alliance to and comparison of results; a weakness is services, due to the diversity of serviceEnd Homelessness (NAEH) strongly support that measuring outcomes solely in terms of type and contexts across Australia. Evenoutcomes measurement and have produced a percentage of target population achieving a within a single organisation there can beToolkit to support government and community goal can miss progress that clients make in tensions between standardisation versusorganisations in measuring effectiveness other areas, and can obscure differences in customisation of tools and measures. Theseoutcomes (Spellman and Abbenante 2008). the degree of change on particular measures. can reflect tensions between the needs of theThe approach is not prescriptive and does not In addition, setting targets and comparing organisation as a whole and the needs of itsprovide a particular measurement tool. outcomes across services are often highly sub-units. political processes, which may have unintendedSome features of the NAEH approach include side-effects. Putting aside the tools themselves, it is unclear(Spellman and Abbenante 2008; Barr 2009): to what extent different expressions of desired• Use of a program logic model to place outcomes reflect differences in language, as desired and actual outcomes within the opposed to differences in conception of service 12
    • Literature Review:Measurement of Client Outcomes in Homelessness Servicesgoals. While there are certainly philosophical wastage of resources within the service system. While this proposal has merit, one of theand political implications to the way outcomes difficulties with it is that the robustness ofare described, practice wisdom also suggests Ideally, it would be possible for outcomes generalisations about interventions in thethat at a practical level, service providers data to be collected by services who are best homelessness field tend to be lower thanwould be able to find areas of agreement placed to obtain the information (i.e. services those in the health field, owing to the typesabout what constitute desirable as opposed to who have the expertise, the resources and of research designs that can reasonably beundesirable outcomes. One of the problems the appropriate quality of relationship with pursued in homelessness settings.with such a process, however, may be that the client), and shared appropriately with‘consensual’ definitions of desired outcomes other services who are working in partnershipmay be high-level and generic, making with that client. This would fit with joint casethem difficult to operationalise. Another planning and review processes (Hamiltonproblem is that services may have different 2010), and would also potentially allowunderstandings of the process of change, responsibility for outcome measurement toleading to conflicting ways of describing the be shared between services. For example, asteps involved in progress (MacKeith 2007: 5). clinical mental health service might collect one set of measures while a Psychiatric DisabilityA separate but related question is the Rehabilitation and Support provider wouldfeasibility of collecting particular core data collect a different set.items across all homelessness services, orservice types (therefore providing a set of Exchange of information in such a systemcommon core outcome measures). The more would need to be governed by the informedof a ‘stretch’ it is for service providers to collect consent of the client. Clear processes fordata, the lower the reliability of the data is seeking and recording this consent would needlikely to be. to be developed.Exchange of outcomes information Burden of proofOne of the problems that can arise as more MacKeith (2007: 5-6) differentiates the useservices begin to collect outcomes data is of outcome measurement to evaluate thethe duplication of data collection by multiple effectiveness of service provision modelsservices. This particularly impacts on clients or interventions from its use by individuallinked to a range of support services – for agencies to monitor their success. She arguesexample, a client may have an outreach support that the “burden of proof” currently falls onworker, a clinical case manager, be linked to a each individual service to demonstrate thatcommunity health centre, use a day program their intervention type is worth funding.and receive occasional assistance from a crisis Instead, she proposes following the healthaccommodation provider. If each of these service model of clinical trials to assess theservices is gathering outcomes data, with no effectiveness of an intervention. Once the trialscoordination between services, the client may are complete, the value of the intervention isbe ‘bombarded’ by measurement requests established and there is no requirement forusing the same or different instruments (Love individual services to prove the value of theand Scott 2010). In addition to the annoyance intervention. Instead, agencies can focus oncaused to the client, this may well undermine monitoring their own success compared withthe reliability of the data collected. The established benchmarks.duplication of data collection also represents 13
    • Literature Review:Measurement of Client Outcomes in Homelessness Services4. Measurement of client outcomes as a research activityClient outcomes measurement systems are selected criteria (Rossi, Freeman et al. 1999: monitor. Evaluations are typically customisedusually intended as practical performance 20-21). Some commentators distinguish to the context of individual services and themeasurement and program management tools, evaluation from research on a range of time-specific needs of particular stakeholders.rather than as vehicles for theoretical research. considerations including the purpose of the Evaluations can explore how and why certainNevertheless, they do involve gathering and study, the relationships of stakeholders to effects occur and is suitable for exploringanalysing data to create new knowledge or the study, and the types of standards used to issues such as long-term impacts, causalinsight, and therefore can be viewed as a form judge the adequacy of the study (Fitzpatrick, attribution, cost-effectiveness and comparativeof research. Sanders et al. 2004: 6-7). Within the field effectiveness (McDavid and Hawthorn 2006: of program evaluation, a wide spectrum of 293; Segone 2008: 101-103).It is useful to consider outcomes measurement approaches can be used (e.g. Wadsworthsystems through the lens of research design 1997: 77-109). The borderline between evaluation andfor several reasons: monitoring is not clearly defined and outlying• To identify choices about the types of Relationship between monitoring and evaluation forms of each approach may resemble outcomes data that are desired, and how Measurement of client outcomes can be the other. A detailed and sophisticated these may be gathered and analysed considered as an evaluative activity (Kimberley monitoring system that includes a wide range• To identify ways in which measurement 2009) – that is, an activity that is designed of performance measures and supplements systems can be designed so as to provide to provide information that may assist in this with a significant amount of qualitative the most reliable and valid data assessing the value of an intervention or data may in effect be an ongoing evaluation• To identify the types of knowledge or program. However, there can be major process. However, for most homelessness insight that outcomes measurement differences in the way that this evaluative organisations such a system would be systems can typically produce, and the activity takes place in an organisational unaffordable. On the other hand, an level of certainty of the conclusions drawn. context. The key distinction that is drawn effectiveness-focused evaluation with limited This may include understanding the within the literature is between monitoring of resources and a quantitative methodology limitations on the types of inferences that client outcomes (as part of an organisational may resemble a time-limited performance can reasonably be supported by the data. performance measurement system) and measurement system. evaluation of client outcomes as part of aOutcomes measurement also has a relationship formal program evaluation. Some sources see both routine outcomeswith program evaluation. This section monitoring and in depth ad hoc evaluationconsiders these relationships and explores Monitoring typically involves systematic, as subtypes of program evaluation (e.g.the types of findings typically produced by periodic collection and analysis of data to Hatry 1997: 3-4). Some have argued thatoutcomes measurement systems. assess performance in relation to an agreed performance measurement can fulfil many of standard set of indicators. Monitoring the same purposes that program evaluations4.1 Monitoring and evaluation systems are usually designed to be ongoing have served, and that in-depth program rather than time-limited. Monitoring provides evaluation is therefore an “expensive luxury”.In general terms, evaluation refers to “the succinct, regular feedback that can assist However, there are important differencesidentification, clarification, and application of with accountability, quality improvement between the typical forms that monitoringdefensible criteria to determine an evaluation and responding to evolving trends in the systems and program evaluations take, andobject’s value (worth or merit) in relation to environment. the level of information that they are able tothose criteria” (Fitzpatrick, Sanders et al. 2004: provide (McDavid and Hawthorn 2006: 293).5). Program evaluation typically involves the Evaluation involves episodic, in-depth collection The term evaluation is used in this paper forapplication of social research procedures to and analysis of information. Evaluation can in-depth, episodic program evaluations insystematically develop a valid description of draw on a broad range of data sources and contrast to ongoing monitoring systems.particular aspects of program performance, methods, and can examine factors thatand the comparison of the performance to are too costly or difficult to continuously 14
    • Literature Review:Measurement of Client Outcomes in Homelessness Services 4.2 naturaliStic and exPeriMentalIt is worth noting that the term monitoring evaluation is also used in the literature, however it aPProacHeS to reSearcHtypically refers to process-oriented program evaluations that focus on how the program is beingdelivered, rather than on outcomes (Fitzpatrick, Sanders et al. 2004: 21). There is a long-standing and sometimes fierce debate in the social sciences (and in scienceAdvantages and disadvantages more generally) about which approachesHatry (1997: 4) provides a useful summary of the relative advantages and disadvantages of to research are the best for studying andevaluation and monitoring, which is adapted as Table 4 below. Monitoring is used to track understanding the world. This “paradigmsperformance against a limited set of measures, and for early identification of trends in the external debate” is linked to philosophical positionsor internal environment. Evaluation is used to guide significant organisational decisions or gain a about the nature of reality, truth, and of whatdeeper understanding of phenomena identified through monitoring. constitutes credible evidence (Patton 1987: 165).Table 4: Advantages and disadvantages of monitoring compared to evaluation (adapted from Hatry (1997: 4) Table 1-1) Experimental approaches to research are typically associated with the positivist tradition, Type Advantages Disadvantages which emerged from the physical sciences. These approaches tend to aim for objectivity, In depth, ad hoc evaluation • Identifies causes of • High cost emphasise separation of the researcher from outcomes (to some • Covers few services that being researched, and assume value- extent) • May take extended time to neutrality. They often rely on and valorise • Can provide relatively get results quantitative data (Patton 1987; Wadsworth strong evidence on 1997: 165). Within the evaluation literature outcomes and effect there are significant critiques of this paradigm, • Can provide guidance as which argue that subjective and objective to improvement action meanings are socially constructed, are central to understanding human phenomena, and that Regular outcomes • Covers many or most • Provides little information all research is value-driven (Wadsworth 1997: measurement agency programs on causes of outcomes 101). • Provides information on a • Provides little guidance on regular basis improvement actions In contrast, naturalistic approaches to research • Lower cost per program • Subject to a variety of are typically associated with phenomenological covered interpretations and constructivist philosophical positions. • Hints at improvement These approaches do not attempt to actions artificially manipulate the environment or the phenomenon being studied, and typically proceed inductively from practice to theory.For greatest benefit, organisations use a combination of ongoing monitoring and episodic Naturalistic approaches aim to describe andevaluation. Data from monitoring can contribute to evaluations, by providing historical data understand naturally unfolding processes,streams (Hatry 1997: 4). Program evaluations provide a much richer understanding of context and including variations in experience, and tendof the factors that may be impacting on client outcomes. to emphasise qualitative data (Patton 1987: 13-15; Wadsworth 1997: 95-96). NaturalisticEvaluation designs may also influence monitoring systems – for example, an evaluation might approaches have been criticised at times forcollect measures on particular domains, which are then partially carried on by the organisation a perceived lack of objectivity (Patton 1987:in routine outcomes monitoring. Evaluations may also identify particular areas of strength or 166).weakness which an organisation may wish to monitor in an ongoing way. 15
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesAt times these approaches and their possible intended and unintended outcomes caused change are inferred. Instead theassociated methods have been characterised are being explored. The Most Significant comparisons depend on individual oras fundamentally opposed and irreconcilable. Change approach (Davies and Dart 2005) is groups for which equivalence either is notHowever, more recently authors have argued particularly relevant to this context. However, established, or is known to be absent.that both approaches are useful and that qualitative approaches will be less appropriate • Single-system designs are those in whichthe key challenge is to match appropriate where the informational need is a broad the comparisons are for one individual,approaches to particular research or evaluation understanding of overall patterns of outcomes group or collectivity, at different pointsquestions and issues (e.g. Patton 1987: 169). across a large group of people. in time (at a minimum, before and duringMost evaluations use several designs or application of the treatment).combinations of designs to address different Given the focus of the outcomes measurementquestions. Mixed method designs that combine literature, the bulk of this literature review It is often not feasible to use true experimentalqualitative and quantitative approaches often focuses on quantitative methodologies for designs in human services evaluations, for ayield richer insight and can increase validity monitoring outcomes of large groups of variety of reasons. In particular, there may be(Fitzpatrick, Sanders et al. 2004: 263, 305). people. ethical issues and risks related to withholding treatment from a ‘control’ group, althoughThe large majority of literature on client For further research: explore possible ways in in certain situations where demand outstripsoutcomes measurement assumes a which qualitative and naturalistic approaches supply a control group may be possible (Cookquantitative methodology, and by association could be used as the basis of an outcomes and Campbell 1979: 347-350; 374). In aimports much of the focus of the positivist measurement system monitoring system, with limited resources, ittradition on reliability and validity. A few is very unlikely that an experimental designsources do acknowledge the usefulness 4.3 Monitoring SySteMS aS would be attempted.of qualitative data (e.g. Burns and Cupitt Single-SySteM reSearcH deSignS2003), although generally as an adjunct to Classic outcomes measurement systemsquantitative measures. This overwhelming The literature distinguishes ‘classical’ are most closely aligned with single-systemfocus on quantitative methods may reflect experimental designs from quasi-experimental research designs. They share the definingthe informational needs that tend to drive designs (including single system designs) (Cook characteristic of single system models: themonitoring systems (e.g. senior management and Campbell 1979: 4-6; Bloom, Fischer et al. planned comparison of a noninterventiondesire for oversight of results for an entire 2006: 44-49): (“baseline”) period with observations ofcohort of clients), as well as the genesis of • Experimental designs involve at least intervention period(s) or in some cases, aoutcomes monitoring in governmental and two groups of participants (a ‘treatment’ post-intervention period, for a single client orbusiness performance measurement systems group and a ‘control’ group), with system (Bloom, Fischer et al. 2006: 322).(McDavid and Hawthorn 2006: 282-288). different treatment provided to the two groups. Random selection and random Within the area of single system designs,Qualitative methods clearly have the assignment are used to attempt to obtain again there are a wide variety of differentpotential to offer considerable insight into equivalence (on average) between the designs (Bloom, Fischer et al. 2006: 352).client outcomes, especially in programs in two groups on all relevant characteristics. The basic single-system design is A-B (wherewhich outcomes are expected to be highly Data is aggregated within these groups A represents the baseline phase and B theindividualised rather than standardised. and analysis of comparisons between intervention phase). More complex designsQualitative methods have strengths in the average scores of the groups are involve replication of original conditions and/capturing diversity and subtlety of experience undertaken. or treatment periods (e.g. A-B-A-B designs);and of outcomes (Patton 1987: 24-30). ‘Open • Quasi-experimental designs depart from others may involve successive or alternatinginquiry’ approaches may be of particular value the experimental model in that they do interventions (e.g. A-B-A-C or A-B1-B2-B3in the early stages of developing outcomes not use random assignment to create designs, where the subscripts representmeasurement systems, when the range of the comparisons from which treatment- varying intensity of the same intervention). 16
    • Literature Review:Measurement of Client Outcomes in Homelessness Services 4.4 liMitationS ofIn general, it is suggested here that an Monitoring SySteMSoutcomes monitoring system should beconsidered as tracking a large number of A relevant concern of all forms of research is to understand the level of ‘trustworthiness’ orparallel instances of a basic A-B single system ‘credibility’ of the research findings. The traditional scientific paradigm tends to concentrate ondesign. Even though services provided by reliability and validity as the two key measures of trustworthiness. Critiques of this paradigmhomelessness agencies may often have suggest other indicators and techniques for demonstrating trustworthiness (Wadsworth 1997).multiple intervention components that might Regardless of the paradigm, different research designs may lead to differences in the level ofbe applied at different times, these different certainty or credibility of the findings, and therefore differences in the strength of the argumentscomponents are not necessarily applied in that can be advanced based on the research (Schalock 2001: 68).isolation or in any particular planned sequence.In individual casework situations, it may be Given the primarily quantitative focus of outcome monitoring systems, it is useful to considerpossible to discriminate different phases issues relating to the design validity of these systems – their ability to validly answer questionsof service delivery, and therefore view the about change, causality and generalisability. (Validity of individual measures and tools is a relatedcasework process with a single client as but separate issue and is discussed in Section 6). Table 5 summarises three fundamental questionsone of the more sophisticated designs with that outcomes measurement might try to answer, and the types of validity that are relevant todifferent intervention phases. However, from these questions (Bloom, Fischer et al. 2006: 338-346). One way of conceptualising this is to askthe aggregated perspective of a monitoring whether the data provided by outcomes measurement is sufficient to reject the “null hypothesis”system, these different stages of interventions - i.e. the hypothesis that no change occurred, or that intervention and outcome are unrelated (dewill tend to be invisible and the ‘intervention Vaus 2002). More powerful research designs are more able to exclude alternative explanations forphase’ needs to be considered as a single B their findings.phase. The withdrawal of intervention (returnto baseline) which is characteristic of some Table 5: Key questions that research may try to answer, and associated validity considerationsof the more powerful single system designs(such as experimental replication or successive Key questions Validity considerations Threats to validityintervention designs) would usually be seenas unethical or wasteful of resources in the Did change occur? Statistical conclusion • Unreliable measurescontext of homelessness service delivery. validity • Inconsistency in implementation of interventionIt is possible that routine outcomes Are there sufficient • Random changes in the intervention settingmeasurement might be considered as a grounds to believe • Small numbers of observation pointsmultiple baseline design across clients, that dependent andi.e. application of the same intervention at independent variablesdifferent times to different clients, comparing covary?the results across clients. However, technicallymultiple baseline designs require that a stable Was the change Internal validity • Unrelated events occurring during thebaseline be obtained for all of the target cause by the intervention periodproblems, clients or settings, and that an intervention? Are there sufficient • Maturation – independent physical orintervention is introduced to only one of the grounds for ruling psychological processes within the subjecttargets at a time while the baselines of the out the hypothesis • Effect of previous testingother targets are continued (Bloom, Fischer et that one or more • Instrumentation - changes in measuremental. 2006: 421). This is clearly not the case in extraneous tools or their usesroutine service delivery, where there may be variables influenced • Attrition in sampleno temporal overlap at all between baselines or produced the • Regression of initial extreme scores to theand service provision for different clients. observed changes? mean • Diffusion or imitation of intervention 17
    • Literature Review:Measurement of Client Outcomes in Homelessness Services Successful replication strengthens the case Key questions Validity considerations Threats to validity for generalisability of results. It is unclear if it strengthens arguments for causal connections Construct validity • Mono-operation and mono-method bias – use between interventions and effects. of only one measure and/or way of recording Are there sufficient per target Outcomes monitoring systems in service grounds to believe • Hypothesis guessing by participants delivery settings have some particular that the research is • Valuation apprehension – people not limitations that further affect the strength actually measuring representing themselves accurately of the conclusions that they support. These what it intends to • Participants able to perceive practitioner limitations are summarised below. measure? expectations • Interaction of multiple interventions Attribution Schalock (Schalock 2001: 141) points out that If the same External validity • Interaction between the intervention and other outcomes in the human services are integrative intervention Are there sufficient variables related to client or environment - that is, they reflect the contributions of all was applied grounds to believe • Individual styles of practice who provide care. Service delivery partnerships under different that the effect of • Different conceptions of target variables in are a necessary part of doing business in the circumstances, the intervention can different studies sector, and are widely seen as an important would the same be generalised to • Interaction of history and intervention aspect of best practice (e.g. Department of change occur? other populations • Measurement differences between studies Human Services (2006: 3.2.7, 3.2.9)). However, (population • Differences in characteristics of client group it can make it very difficult to disentangle the external validity) (representativeness of sample) contributions of multiple practitioners and or other settings • Interaction of testing and intervention multiple services, especially in a monitoring or environments • Reactive effects to evaluation – awareness by system where (unlike in a program evaluation) (ecological external participants that they are involved in a study there will generally not be sufficient data or validity)? time to enable in-depth consideration of other environmental factors.Single system designs vary in their ability to deal with the ‘threats to validity’, and hence in the Many authors have identified this problem.strength of the conclusions that they may support. A basic single system design (A-B), if well Rossi (1997: 25) distinguishes ‘gross’ fromimplemented has the potential to answer the first key question (“did change occur”), including ‘net’ outcomes. Gross outcomes are the totalidentifying how likely it is that changes could have occurred by chance alone. However, basic change in the target problem, consisting of thesingle system designs are generally not powerful enough by themselves to provide strong effects (if any) of the program plus all of theevidence about causality, or to support generalisation of findings (Stedman, Yellowlees et al. 1997: other processes, including other interventions,20; Bloom, Fischer et al. 2006: 351, 377). the client’s own independent processes, and environmental factors, that led to suchOne of the advantages of monitoring systems is that over time they potentially provide data on outcomes. Net outcomes are those that wouldoutcomes for a large cohort of clients. Clinical replication involves the repeating of an intervention not have occurred without the program.package, applied by the same practitioner or group of practitioners to a series of clients in Unfortunately, within a monitoring system it isthe same setting, with multiple problems that generally cluster together (Bloom, Fischer et al. usually difficult or impossible to establish the2006: 347). In some circumstances outcomes monitoring could be viewed as clinical replication; net effect (Baulderstone and Talbot 2004: 3-4).this would depend on the level of similarity of client characteristics, presenting problems, andthe degree to which the interventions are applied uniformly and consistently across clients. 18
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesThere may be ways to modify the design of the can be expected to have an immediate In these cases, if outcomes data is consistentoutcomes monitoring system so as to reduce effect – perhaps providing material aid or with the results expected according to pre-the probability of alternative explanations for emergency housing), it may be useful in existing evidence or theory, this may bolsterchange: some situations. the argument that the program is operating• Collect and analyse additional quantitative as expected and that there is some causal data that indicates other potential causal McDavid and Hawthorn (2006: 62-62, 361) connection between activities and results. factors operating in consumers situations discuss the concept of the ‘core technologies’ (Stedman, Yellowlees et al. 1997: 20). For used by programs. They comment that human Another way to view questions of attribution example, there may be ways to analyse services programs tend to use ‘low probability may be to see the combined work of information on the number and types of technologies’ in which our knowledge of what all services involved with a client as the support links that clients have to gain works is often unclear, and in which results intervention. From this viewpoint, the issue an understanding of the possible other are heavily influenced by a wide range of of the differential contribution of various service contributions to an outcome. If it variables which it is impossible to control. The services become irrelevant. The attribution is known that a client is not engaged with corresponding argument is that attribution question then becomes, with what level of other support services, this rules out at will always pose more challenges with low certainty is it possible to claim that change was least some alternative explanations for probability technologies. However, even within caused by the actions of the service system, outcomes. a single program it may be possible to discern as opposed to other chance or environmental• Assess clients and practitioners opinions a range of ‘technologies’ within use, with factors (economy, political changes, the client concerning potential change and the varying levels of probability. winning the lottery, ...). While in one sense this meaning of change scores (Stedman, approach reduces the number of alternative Yellowlees et al. 1997: 20). For example, For some categories of outcome (especially explanations that must be considered, it may be possible to ask clients to what interim hard outcomes concerning status stakeholders may view it as unsatisfactory for extent they think any change resulted change), with sufficient knowledge of the two reasons: from their involvement with the service, as service it may be reasonable to conclude • it still does not allow a well-founded claim opposed to external factors. that the service was a necessary (if not the that the service system caused the change,• Triangulate quantitative data from the only) contributing factor to the outcome. For as opposed to other factors monitoring system with other forms example, all clients of a particular service • it does not meet the needs of individual of data, including qualitative data that might be provided with Transitional Housing organisations who wish to know how provides insight into program processes (as for example in HomeGrounds Justice effective their programs are. and staff and client perceptions of change. Housing Support Program). For clients that For example, the use of case studies in are homeless at assessment and housed in Depth of understanding of intervening variables conjunction with monitoring data can Transitional Housing during engagement with Intervening variables are those factors that greatly deepen the understanding of the program, the program can clearly claim a affect the process of change, between inputs the processes that have led to observed causal contribution to that interim outcome. and outcomes. Weiss (1972: 48-49) categorises outcomes. intervening variables into:• An immediate pretest-posttest design Other situations in which claims of causality • program-operation variables – factors (Bloom, Fischer et al. 2006: 386), in may be more solid include: concerning the way particular interventions which measures are taken immediately • Situations where there is an existing or services are provided, for example, before and after a single session of body of evidence that demonstrates the frequency of contact, continuity of staff intervention, can reduce the likelihood effectiveness of the particular intervention providing service, extent of coordination of that other factors intervened in the causal or program being monitored services, etc. relationship between intervention and • Situations where the intervention is effect. While this approach clearly has situated within a well-established limited applicability (to interventions that theoretical framework and other aspects of the framework are supported by existing evidence. 19
    • Literature Review:Measurement of Client Outcomes in Homelessness Services• bridging variables – intermediate points required, in general three points are the It may be possible to obtain additional milestones or interim outcomes that minimum necessary to establish any sort of data points for the baseline phase using (according to the explicit or implicit theory clear pattern, while having ten or more points a reconstructed or retrospective baseline of the program) are necessary to achieve enables better use of statistical techniques (Bloom, Fischer et al. 2006: 365). This may the desired end result; for example, such as tests of significance (Bloom, Fischer involve asking the client for their estimates or change in attitude might be seen as a et al. 2006: 364). For greatest validity it is recollection of their condition over the months bridging variable for a program attempting ideal if baseline and intervention phases are preceding intake. Alternatively, it might involve to produce behavioural change. roughly equal in length, as this allows external gathering historical data from other services and internal factors to influence behaviours involved with the client. Another possibility, forOutcome monitoring systems may have limited in baseline and intervention phases equally those services using waiting lists, would be toinformation to draw on to assess the effects (Bloom, Fischer et al. 2006: 327). regularly collect assessment data from clientsof intervening variables. This may be because while they are on the waiting list. However,only limited data is able to be collected or In service delivery situations, especially for all of these options require additional dataanalysed on programs-operation factors; longer-term case management services, it is collection resources which may be unavailablelikewise, it may only be practical to collect highly unlikely that baseline and intervention in high-pressure service environments.data on a few (or no) interim outcomes. A phases would be of equal length. Further, it isrelated issue is that research designs (whether often the case that only a single baseline data Follow upexperimental or naturalistic) require the point is available (at point of initial assessment Many authors emphasise the importance ofclearest possible definitions of the intervention, or intake). It would generally be unethical to gathering follow-up data after the interventionso that it is clear what was actually done and withhold service delivery until a stable baseline (service delivery) has ended. This allowshow it might be replicated. However, in a data picture can be established, especially an understanding of whether any changesservice delivery context, interventions may be where there was some imminent threat to observed during engagement have beenquite amorphous, flexible, and/or may fluctuate the safety or health of the client or a third ongoingly maintained in the clients life,in response to a wide range of factors over party. Clients are often in crisis at intake, and and hence whether the desired medium ortime and between staff members (Bloom, there is pressure to commence intervention long term outcomes have occurred (Bloom,Fischer et al. 2006: 322, 329). immediately (Bloom, Fischer et al. 2006: Fischer et al. 2006: 337). This is particularly 365). It should be noted, however, that for important given evidence that gains made byCareful selection and use of process measures many longer-term services, service delivery clients in human services programs may notin relation to quality of service delivery may does begin with a period of assessment be sustained (Rapp and Poertner 1992: 107).allow some understanding of how program- and relationship establishment and this may Booth and Smith (1997: 42) suggest that aoperation variables affect outcomes. provide an opportunity for collecting baseline follow-up rate of 80% of the original sample is data. In other cases, service delivery may only required for generalisability of findings to theShortened research phases occur over a relatively short timeframe, and original population.In general, quantitative research designs there may be only one data point available insupport more robust conclusions about the intervention phase also. Given only a single However, there are many logistical andwhether change occurred, and what caused data point, it is impossible to tell from this resourcing challenges involved in follow-upthe change, when the periods of data information alone whether it represents the of clients, especially those who are transientcollection at baseline and during intervention typical condition of the client at that time or (Post, Isbell et al. 2005: 12). For short-termare of sufficient length (Bloom, Fischer et not. For this reason, multiple assessments over services (such as many SAAP services),al. 2006: 327). Sufficient length may be time are preferred (Stedman, Yellowlees et al. the transitory nature of engagementinterpreted as involving a sufficient number 1997: 20). further reduces the possibility of follow-upof data points to provide a clear picture of a (Baulderstone and Talbot 2004: 3). Therelatively stable pattern of events or trend. limited follow-up possible within an outcomeWhile there is no absolute number of data monitoring framework limits conclusions about the stability of changes achieved during engagement. 20
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesConclusionQuestions of causality are notoriously difficultin research in the social sciences. Even with themost robust research designs, conclusions aboutcausality are still generally a case to be arguedon the balance of probabilities, rather than afact than can be definitively proven. Given thelimitations of outcome monitoring systems,it is unrealistic to expect that these systemsin themselves can support strong argumentsfor program effectiveness. However, whencombined with other methods and sources ofdata they may form one plank of an argumentfor effectiveness.It is worth remembering that in general it is notnecessary or appropriate to try to demonstratethat the work of a particular program or agencywas the sole cause of a client outcome. Apartfrom anything else, this ignores the central roleplayed by the client in achieving change. It willgenerally be more appropriate to try to showthat the program made a contribution to theoutcome observed. However, even this lesserclaim may be difficult to argue persuasively.When well designed and implemented, outcomemonitoring systems are potentially capableof providing valid and credible informationregarding changes occurring for clients. Thisis valuable information for planning servicedelivery at the individual and program level.It is also important to recognise that data doesnot have to achieve academic standards ofrigour to be useful (Friedman, DeLapp et al.2001b: 2.7). No matter what data collectiontechniques are used, there will always be limitson the reliability and validity of outcomesdata. Within these constraints, outcomesmeasurement can still provide a useful gauge ofwhat is happening for clients. 21
    • Literature Review:Measurement of Client Outcomes in Homelessness Services5. Conceptualising client outcomes5.1 general definition In the narrow view, outcomes refer to effectiveness only - i.e. the change that occurs with respectIn its most general sense, the term outcome to the client’s problem that is the target of intervention. This may also be referred to as ‘clinicalrefers to a condition that arises from an action outcomes’ (Berman and Hurt 1997: 82-83).or set of actions. In the field of human servicesthe term outcome tends to be used in a more The two different ways of using the term outcome are linked to the observation that effectivenessfocused sense: a personal or organisational outcomes are only part of a larger picture of an organisation’s performance, and that attentionchange or absence of change that results to inputs and to process is equally important to ensuring that an organisation is ‘doing a goodfrom an action or set of actions carried out job’ by providing quality services (Weiss 1972: 46; Planigale 2010b). Schalock (2001: 10-11,by staff of the organisation (Schalock 2001: 18-19) provides a framework for outcomes (in the broad sense) which includes four quadrants,3-4; Baulderstone and Talbot 2004). Similarly, summarised as below. The distinction between performance measurement and value assessmentBurns & Cupitt (2003: 6) define outcomes for in this model reflects a difference between ‘hard’ outcomes (role status and behaviour) vs. ‘soft’homelessness services as “… the changes, outcomes (quality of life and personal experience). It should be noted that most authors usebenefits, learning or other effects that happen the term performance measurement more loosely to encompass any measure that provides anas a result of your activities.” indication of the quality of the organisations work.This use of the term outcome is neutral with Figure 1: Outcome categories as per Schalock (2001)regard to whether the change or effect isregarded as positive or negative. An outcome Evaluation standardof service delivery may be an improvement ordecline in a clients wellbeing. In contrast, the Performance measurement Value assessmentterm desired outcome can be used to describethose effects that a stakeholder positively Evaluation Organisational Organisational performance Organisational valuevalues and aims to produce. focus outcomes outcomes: outcomes: • Service coordination • Access to servicesEffort and effect: organisation-focused vs. • Financial stability • Customer satisfactionperson-focused outcomes • Health and safety • Staff competencies • Program data • Family / consumerThe term outcome tends to be used in two • Staff turnover supportsdifferent ways in outcomes measurement • Community supportliterature related to the human services. Thedistinction might be termed the ‘broad’ and Individual Individual performance Individual value outcomes:‘narrow’ uses of the term. outcomes outcomes: • Quality of life • Physical wellbeing • Emotional wellbeingIn the broad view, outcomes refer to a range • Material wellbeing • Personal developmentof organisational and personal measures that • Self-determinationreflect the quality of the organisation’s service • Interpersonal relationsdelivery across a range of dimensions including • Social inclusionquantity and type of services provided, • Rightscost, satisfaction (of clients, staff and otherstakeholders), equity, timeliness, relevanceto client or community need, effectiveness, Outcomes in the narrow sense (clinical effectiveness) are located in the bottom two quadrants.and other dimensions (Berman and Hurt 1997:82-83). 22
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesAnother way that clinical outcomes are of poor outcomes such as dependence on a • performance accountability isdistinguished from the broader picture of service). They are often positively skewed accountability by managers to stakeholdersorganisational outcomes is through the terms (Segal 1997: 155-156) and may be susceptible for the performance of a program, agency‘effort’ and ‘effect’ (Friedman, DeLapp et al. to validity problems depending on the scales or service system.2001b: 1.1). Effort refers to the quantity and used (Ware 1997).quality of service delivery activity, while effect The distinction is crucial to the implementationrefers to quantity and quality of change for Outcomes for groups and third parties of outcomes measurement in individualthe better that was produced by the activities. Some services may aim to achieve outcomes organisations. It prevents staff at an agencySimilarly, Hudson (1997: 70) distinguishes that are changes not in individuals but in larger level from feeling that they are responsibleoutcomes with respect to the problem from groupings of people - for example, families, for population outcomes (which individualoutcomes relating to professional behaviour or communities, other organisations (Burns staff cannot hope to impact on at a globalactivity of the organisation or its employees. and Cupitt 2003: 6-8). While the distinction level), while also making clear the contribution between effort and effect is still relevant for that an agency makes to population resultsFor the purposes of this paper, outcomes will work with these larger groups, the definitions (Cunningham-Smith 2010).be used in the narrow sense, i.e. to refer to of the types of outcomes that may occurchanges in the presenting problem. becomes more complex. There are well-tested Some population results may reflect the outcome definitions and measures for work direct aggregation of individual results. ForConnection between effectiveness outcomes with families, and to some extent community example, an individual client achieving aand other organisational outcomes outcomes may be measureable through a better level of income may contribute to anFollowing the above discussion, performance census or population sampling approach. agency percentage of clients with sufficientmeasures such as client satisfaction or income, and to a population indicator aroundaccessibility of services are not defined as For further research: outcomes and tools for level of poverty in the community. Individualoutcome measures for the purposes of this groupings other than individuals, including results may also contribute indirectly topaper. However, these aspects of service tools for measuring changes in organisations other population-level results such as morequality may still have some connection with appropriate use of welfare services, loweredeffectiveness. In particular, there is some Population, program and individual outcomes crime rates, and so forth (Blunden andevidence that particular service delivery While homelessness agencies typically work to Johnston 2005: 1).behaviours may tend to result in better achieve positive outcomes for individual clientsoutcomes than others (Rapp and Poertner (or households), these individual changes also Change and maintenance1992: 121-122). It may therefore be useful contribute to a larger picture of change at an While outcomes are often thought of in termsto use productivity measures as a proxy agency, program, or community level (Weiss of change, it is important to recognise that atfor outcomes in some circumstances (for 1997: 175). This is not an issue about different times the desired outcome may seem to be aexample, in short-term services where it is very types of outcomes, but about the way that lack of change – i.e. maintenance of a person’sdifficult to get follow-up data), although the outcomes are aggregated so that change (or current situation (Burns and Cupitt 2003: 6).assumed relationships between outputs and lack of change) can be observed in larger An outcomes measurement system needs tooutcomes also need to be tested. Productivity populations. allow for maintenance goals as well as change(activity or output) measures can also help to outcomes. However, maintenance goals maymeasure fidelity in implementation of a service Friedman, DeLapp et al. (2001b) distinguish often be accompanied by a goal of reductionmodel (Schalock 2001: 22) and to identify results accountability and performance of risk, improvement of prognosis or avoidanceopportunities for improvement in practice. accountability: of future negative consequences. Even a • results accountability is accountability by maintenance goal may therefore have someClient satisfaction measures, while useful to the community (city, county or state) to change indicators associated with it.some extent as an indicator of service quality, the community (city, county or state) forare of little use in assessing client outcomes the wellbeing (results) of a population(and may in some cases actually be markers (children, adults, families, all citizens...) 23
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesIntended and unintended outcomes will take different lengths of time to occur, Intermediate or interim outcomes areWhile human services programs are either because of the internal process that indicators of progress towards a final desiredestablished with intended outcomes, they can may need to occur within the individual who is outcome. For example, if the desired outcomealso have a variety of unintended outcomes. changing, or because of the external process is housing stability for at least 6 months, anThese unintended outcomes may be positive that needs to occur within the individual’s interim outcome would be placing a client inor negative (and this may vary from the environment. appropriate permanent housing (Spellman andperspective of different stakeholders). The Abbenante 2008: 16) – or assisting the client tounintended outcomes may not necessarily be One approach in the literature is to use a set have their public housing application approved.unexpected – i.e. they may be predictable by of temporal durations to separate out sets Given the difficulty of long-term follow-up withthose with expertise in the field – but they of outcomes. For example, Spellman and clients in some settings, interim outcomes maymay be ‘side-effects’ of the operation of the Abbenante (2008: 12-13) categorise be feasible to measure and may provide someprogram. Rossi (1997: 23) gives the example outcomes as: proxy indication of long-term outcomes. Theof introduction of improved income support • short-term (occurring within the first month difference between a short-term outcome andbenefits for people in particular categories; an of involvement in the program) an interim outcome is not always clear – butexpected but unintended consequence was • intermediate (occurring within one year in a sense a short-term outcome can bereduction in work effort. Blunden and Johnston of commencement of involvement in the regarded as ‘complete’ whereas an interim(2005: 1) make a similar point in relation to program) outcome is always understood as a step onprovision of public housing. In some cases, the • long-term (after three years or more from the way to somewhere else. Other authorsunintended consequences may be ‘flow-on’ commencement of involvement with the use the terms ‘proximal’ and ‘distal’ outcomes,effects to other parts of the system that the program). where proximal outcomes are similar to interimindividual is a part of. For example, changes in outcomes, i.e. outcomes that occur on the waythe wellbeing of homeless people (a primary These timeframes appear appropriate to the to a final state and contribute incrementally togoal of homelessness services) may lead to types of outcomes that may occur in the that state: “… it may be possible to measuredifferent patterns of usage of a variety of other environment of homelessness service provision outcomes that are consistent with [theparts of the service system (Rossi 1997: 22). – for example, some outcomes may well be program’s] goal but that occur close enough in observed within a month of engagement time to be practical to use in the measurementBy definition, unintended consequences will (provision of material aid), while others will of that program’s effectiveness and efficiency”tend to be more difficult to detect, particularly take several years (or more) to ‘mature’ (Rossi 1997: 29). Weiss (1972: 48-49) discussesin a monitoring (as opposed to evaluation) (e.g. obtaining public housing via an Early ‘bridging variables’ – sub-goals on the wayenvironment where the number of measures Housing Application). These timeframes may to achievement of a final goal – and notesthat can be tracked is limited. However, this be dependent on the type of service provided that these bridging variables may affect thedoes speak to the value of having a range – for example, in a Housing First approach, ultimate effectiveness of a program.of wellbeing measures in place rather than obtaining permanent housing would often benarrowly focusing measurement only on the seen as a short-term rather than a long-term Interim outcome data can be useful for aspecific goals of the program. In medical outcome. number of reasons (Burns and Cupitt 2003: 8):terms, just because a treatment is effective in • it can demonstrate progress on thecontrolling a particular condition, this does not Another approach would be to link sets of way to longer-term goals, providingmean it has improved a patient’s overall quality outcomes to stages within a theory of change encouragement to both clients and workersof life (Ware 1997: 52-53). model. Different types of outcomes and levels • in situations where end goals may be of outcome stability might be expected in early difficult to reach, it can recognise theShort, medium and long term outcomes stages of change as opposed to subsequent valuable contributions of the organisationThe question of stages of change is important stages. to steps along the wayto outcome measurement for a number of • it can help to acknowledge thereasons. It is expected that different outcomes contributions of different services to resolution of the same problem. 24
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesA more subtle question is whether long-term while service providers may see this as a 10). Logic models are useful for purposes ofoutcomes are based on gradual change over relevant goal for only a minority of clients and communication, shaping of service delivery, andtime (linked to constant intervention - for may place more priority on physical health and for evaluation, to guide the types of evaluationexample, in a long-term case management safety. questions that may be asked and to interpretsetting), or whether they may result from findings. There are three main types of programchanges put in place by a relatively brief A related distinction is that between client logic approaches: outcome approaches (basedperiod of intervention, perhaps years status and case status variables (Baulderstone on an implicit theory of program functioning),earlier. Although behavioural theorists tend and Talbot 2004: 5). Client status variables activities approaches (documenting into be dismissive of ‘sleeper’ effects (Rossi refer to information which indicates outcomes detail the sequence of activities in program1997: 29), the example of a public housing of direct relevance to clients, such as higher implementation), and theory approachesapplication which results in an offer several quality accommodation; case status variables (documenting the theoretical and causal basisyears later is a practical example of this type refer to information about outcomes of direct of the program’s functioning) (DHS Evaluationof outcome in the homeless setting. It is relevance to agencies, often pertaining to Support Unit n.d.).also possible for medium-term successes to how a client is categorised within the servicediminish over time, speaking to the value of system (e.g. removal of a client from an at-risk Logic models play an important role infollow-up measurements to establish long-term register). contextualising outcomes measurement. Logicoutcomes. models commonly document the connections At times there will be a disagreement between between inputs, activities, outputs, outcomes5.2 StakeHolder PerSPectiveS client and societal or program goals. A and impact. In this way they are useful in program may value the outcome of reduction distinguishing client outcomes from otherHuman services programs have multiple in problematic drug use, while some clients aspects of a service’s operations, but also instakeholders – clients, service delivery might see an increase in drug use as positive. pinpointing the ways that outcomes may beworkers, managers, funders, service delivery Involuntary services, such as Child Protection, connected to or dependent on other elementspartner agencies, and communities or ‘the often involve conflicts between the program of the model.public’, amongst others. While it is easy to and client goals. Rapp and Poertner (1992:assume that everyone shares the same view 106) note that using ethical analysis to blend HomeGround has developed a program logicof the aims of a program, often there are apparently conflicting goals is a standard social framework (Planigale, Kimberley et al. 2009);substantial differences in perspective (Weiss work skill, applied in practice daily. Blending the use of terminology in that framework1972: 27). This can be a hurdle in program and/or choosing between conflicting desired is consistent with its usage in this paperevaluation, but also affects monitoring outcomes is a necessary activity that is the (including usage of the word outcomes) andsystems. responsibility of those implementing outcomes discussion in this section should be referenced systems; however, it can be supported by to that paper.Stakeholder perspectives necessarily impact consultation and by clear articulation of theon decisions about what outcomes to value basis for particular decisions. Outcomes and goalsmeasure, and about the interpretation of A program logic model is usually linked toresults. Whether a given outcome is positive 5.3 outcoMeS aS an eleMent of a program’s goals, as expressed throughor negative is a value judgement; the effects PrograM logic its mission, aims and objectives. Goals mayof programs may therefore be positive in potentially be quite broad (e.g. “to improvethe eyes of some actors and negative in the Program logic is a systematic way of family functioning”). To be practical foreyes of others (Rossi 1997: 22). Even where documenting the connections between the outcomes measurement, goals need to bethere is agreement on valuation of outcomes, various aspects of a program’s operations, clear, specific and measurable (Weiss 1972: 26).there may be differences in views of priority. and in particular the connections between Rossi (1997: 28) suggests looking at the coreGovernment departments may have an interest effort (inputs and activities) and effect activities and emphasis of casework within ain measuring participation in EET activities, (outcomes) (Spellman and Abbenante 2008: particular service to determine which aspects 25
    • Literature Review:Measurement of Client Outcomes in Homelessness Servicesof a general goal should be operationalised as example of such an ‘endogenous’ outcome line between one domain and another)outcomes. Program objectives may correspond might be a program that claims that one of and ‘subsumption questions’ (is a domainto outcomes (e.g. to place clients in emergency its outcomes is avoiding the out-of-home independent, or part of a higher-level domain).accommodation) or to outputs (to provide three placement of children while their families aresessions of personal counselling to each client). participating in the program. If the program For the purposes of this literature review, the is structured so that families recruited into term domain is used to refer to an aspectOutcomes and outputs the program are offered participation as an of human existence that is understood toMany authors stress the difference between alternative to out-of-home placement, then be relevant to most or all people within aoutcomes and outputs. The distinction is the outcome is essentially assumed as an particular community or population – fortypically understood to be that outputs element of the program’s activities. “The fact example, physical health, material wellbeing,quantify the level and types of activities that a child … is not taken into custody while and so forth.provided by a service, while outcomes indicate in the program restates the existence of ahow the need/problem is affected by these moratorium and is not itself an outcome of the Domains (in the sense used in this paper) areactivities. Outputs can be important as process program” (Rossi 1997: 30). also referred to as:measures, and help provide the context • ‘life domains’ – especially within the Qualityfor explaining results, but do not measure Outcomes and targets of Life literature (e.g. Felce and Perry 1995)whether the program is effective (Spellman Targets set particular (generally numerical) • ‘life areas’ – especially within social work /and Abbenante 2008: 7, 16). Baulderstone benchmarks for the level of particular types of practice focused literature (e.g. Kaufmanand Talbot (2004: 3) define outputs as “the outcomes that the program wishes to achieve 2007)completed service transactions or immediate (Spellman and Abbenante 2008: 27). Targets • ‘outcome areas’ – especially withinresults created by the program often defined are commonly set in relation to the percentage literature focusing on outcomeas units of service (e.g. support hours of a given client population who will achieve a measurement (e.g. MacKeith and Grahamdelivered, referrals made, beds provided)”, particular type of change, however they might 2007).which they note tend to be easier to count and also be set in relation to the degree or qualitymeasure than outcomes. of the change (for example, to remain stably Domains as cultural constructs housed for at least 12 months, to improve at Often, domains are thought of as universalsIt is worth noting that not all sources embrace least three points on a particular scale). - aspects of human life that are common orthis distinction between outputs and outcomes. relevant to all people. For example, the domainIn particular, Duignan (2005) defines an 5.4 doMainS of physical health is often asserted to beoutcomes hierarchy as including claims about universally applicable to human beings. Onea range of “causes” leading to higher-level Many approaches to outcome measurement implication of this statement is that it should beoutcomes – the lower-level causes, which organise outcomes and measures into possible to measure the physical health statusmight be termed outputs in some systems, are domains. Within evaluation and outcome of any person. However, it is also possible tohere regarded as lower-level outcomes. measurement literature, the concept of have a domain which is only of relevance to a domains is typically assumed to be useful and particular community, population or society.Outcomes and activities valid. While individual domains are discussed For example, employment is a domain oftenIn general, the distinction between a program’s in detail, the concept of domain itself is rarely used to assess outcomes of social serviceactivities and outcomes is relatively clear. defined or critiqued. From a practical point programs within industrialised societies. LabourThe activity may be transporting a client to of view, this paper also assumes the concept markets and employment relationships area dental appointment; the outcome may be of domains to be useful. However, it is worth cultural phenomena and are not necessarilyan improvement in the client’s dental health. defining the concept clearly and pointing found within all human societies.However, there are some situations where out some of its limitations and complexities.an outcome may be defined in terms that Among other factors, complexities can relate In the context of health care, Booth andare equivalent to the program’s activities. An to ‘boundary questions’ (where is the dividing Smith (1997) and Ware (1997) discuss the 26
    • Literature Review:Measurement of Client Outcomes in Homelessness Servicesdifference between generic and specific example, is understood in different societies model has three top-level domains (being,measures, and generic and specific health (or even by different sub-groups within a single belonging and becoming), each of whichconcepts. Specific measures are those that society) can be widely different. Any set of have three sub-domains (for example, theare targeted to specific conditions or diseases; data items designed to measure a person’s being domain is divided into physical being,while the presence or absence of these status or functioning within a particular psychological being and spiritual being)conditions may be generally applicable as a domain therefore expresses a particular (University of Toronto Centre for Healthpopulation measure, tools focused on specific model or conception of the domain. These Promotion 2010).condition domains will only be applicable to the models are grounded in particular cultures andsub-population of people who experience the professions, and should therefore be applied Other approaches (particularly those with acondition. Generic measures relate to generic with caution (if at all) to other cultural settings. practice rather than theoretical orientation)components of health or wellbeing that are enumerate a range of domains with minimalunderstood to be of relevance to an entire A related point is that the relative importance attempt to group these together into largerpopulation. and significance attached to particular domains units. For example: varies from society to society, and from • Kaufman (2007) lists 16 life domainsAn analogous distinction can be applied outside individual to individual. This observation is ranging from ‘housing and basic needs’of the realm of health care. For example, particularly relevant to tools such as Quality to ‘spirituality and values’ - these are notparenting and child wellbeing are examples of of Life scales that aggregate scores from a grouped into larger units;domains that are specific to families; measures number of domains to reach an overall rating. • The BT Generic Outcome Scalein these domains are therefore specific rather When assessing the relative contribution (Baulderstone and Talbot 2004: 65-74)than generically applicable to the entire of each domain to the aggregate score, it includes 17 domains related to ‘knowledge,population (or to an entire client group). is useful to weight the domains to reflect feeling and behavioural change’ (including individual and societal appraisals of the two which are identified as aspects ofSome suggested domains may be contested importance of this life area (Felce and Perry health), and three related to ‘client-either in terms of their universality, or their 1995; University of Toronto Centre for Health environment change’. The division intovalidity as a model of some aspect of human Promotion 2010). personal vs. environment-focused changelife. For example, some Quality of Life and could be regarded as a top-level domainoutcome measurement tools include domains Sub-domains division. However, the intent of the tool isor items relating to spirituality. The Ridgway Many authors (particularly in the Quality of that relevant domains can be excerpted forRecovery-Enhancing Environment Measure Life literature) group domains in a hierarchical use with particular clients without affecting(Campbell-Orde, Chamberlin et al. 2005) fashion, with a small number of top level the validity of the tool. From this point ofincludes the item ‘I have a positive spiritual domains, each of which includes a number of view it is an advantage that the domainslife/ connection to a higher power’. Some sub-domains. For example: are arranged in a flat structure rather thanauthors might dispute the universality of • The World Health Organisation’s WHOQOL- being embedded in larger units.spirituality as a domain of human experience, BREF is based on four domains (physicalor model it as an aspect of emotional health, psychological, social relationships Where domains are arranged into largerwellbeing. and environment) - each of these has a groupings, in some cases these groupings are number of sub-components (for example, based on empirical evidence – for example,Even where a convincing argument can physical health is subdivided into activities through use of statistical techniques such asbe made for the general applicability of of daily living, dependence on medicinal factor analysis (DeCoster 1998); see also Worlda particular domain, there will be cultural substances and medical aids, energy and Health Organisation (1996: 6). In other cases,differences in the way these domains are fatigue, mobility, pain and discomfort, the grouping is based either on the author’sconceptualised in different populations. The sleep and rest, and work capacity) (World own preferred grouping, or on a meta-analysisway that emotional or mental health, for Health Organisation 1996) of common themes and approaches across • The University of Toronto’s Quality of Life other models (e.g. Felce and Perry 1995: 60). 27
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesA further issue is that some approaches to • knowledge – acquisition of information, • environmental structure – in particular,assessing status or measuring outcomes understanding, self-awareness, insight policies and constraints imposed by theinclude a range of indicators but do not • behaviour – skills (a person’s ability to do environment that impact upon the person.attempt to group these into domains at some behaviour); performance (a person’sall. For example, the Illness Management use of particular knowledge and skills in For further research — explore the concept ofand Recovery Scales include 15 items that their life) environmental structure / constraints and lookeach relate to a different aspect of illness • status – categorisation of someone’s at how it could be defined more clearlymanagement and recovery, with no explicit situation into one of a set of mutuallyattempt to group these in any way (Campbell- exclusive categories (for example, Rapp and Poertners category of affectOrde, Chamberlin et al. 2005: 32). ‘homeless’ and ‘housed’ might be examples includes a number of areas that other authors of high-level status categories) categorise separately. Weiss (1972: 39)The discussion paper A consistent set of • environment – the relationship between separates attitudes, values and personalitycasework domains for HomeGround (Planigale a person and their social and physical variables, while Ware (1997: 59) expands on2010a) looks at examples of a variety of context (including e.g. a person’s access the concept of values to include how peopledomains sets or trees that are in use to resources, relationships with caregivers, as evaluators rate themselves – a conceptand relevant to outcomes measurement in degree of control or choice). related to Quality of Life approaches such ashomelessness services. Multiple Discrepancies Theory that consider This taxonomy can be refined in a number how people perceive themselves in relation to5.5 locuS of cHange of ways. Rapp and Poertner acknowledge a variety of reference points (Hubley, Russell that the change category of ‘environment’ et al. 2009). Schalock (2001: 23) distinguishesAnother way of dividing up outcomes is is the least well-developed and least clearly adaptive behaviour (self care, language,according to what might be called the locus conceptualised (1992: 199-120). In Rapp mobility, independent living) from role statusof change – what is it about the client or the and Poertner’s formulation, environmental (a set of valued activities considered normativesituation that actually changes? For example, change includes a number of areas that could for specific age group). Ware (1997: 59) alsodoes the client feel better about their situation, be viewed as specific life areas rather than touches on the concept of role performanceare they behaving differently, or do they have general types of change (e.g. adequacy of a and role disability. Finally, in the area ofsome physical resource now which they did not person’s residence, food and finances). It is status it is possible to look at both a personshave previously? suggested here that these are better viewed as current status, and their potential future status domains (particularly in the case of residence (prognosis or risk).In understanding the types of changes that / housing). However, the other four changesocial service programs aim to produce, the categories (affect, knowledge, behaviour Burns and Cupitt (2003: 6) also present whatrelationship between life domains and locus and status) are primarily person-focused. To is essentially a taxonomy of locus of change,of change can be confusing. This section discard a focus on the environment altogether however it combines elements of both lifediscusses different taxonomies of locus of would be problematic in ignoring a range of domains and locus of change. The discussionchange and then looks at how these can be factors that often have a major impact on the below suggests that these are better keptintegrated with the concept of life domains. work that occurs between staff and clients, separate. and also have a major impact on outcomes.Locus of change taxonomiesOne well-known categorisation of locus of The suggested resolution is to replace thechange is that of Rapp and Poertner (1992: change category of environment with two108), who identify five major categories in more precisely defined loci of change:which social work interventions are designed to • resources – the person’s access towork with clients to produce outcomes: resources (broadly defined – including• affect – the way people feel about assets, access to services and other something; their emotional response to a supports) situation; their attitudes 28
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesTable 6 presents a taxonomy of locus of change based on Rapp and Poertners model but Soft and hard outcomes (or measures) couldintegrating the refinements noted in this section. be seen as one way to divide up the locus of change set, where the first two locus of changeTable 6: Integrated taxonomy of locus of change categories (affect and knowledge) can be seen as soft outcomes and the remaining four as Broad category Specific locus of change more associated with hard outcomes. Affect Emotion / mood Segal (1997: 151) suggests that hard and soft outcomes are not mutually exclusive - i.e. that Attitude it does not necessarily make sense to only measure hard outcomes or soft outcomes. The Value (including self-evaluation) two types can be closely intimately connected; if the objective is improving a person’s sense Knowledge Information of optimism, the “hard-core resources” for this process may be an apartment of the person’s Insight choice, a job that accommodates a disability, or the opportunity for social relationships. Behaviour Skill Relationship between Performance - adaptive behaviour locus of change and domains It is useful to have a model of how domain and Role performance locus of change fit together. The suggestion made here is that locus of change cross-cuts Status Status the domain set. In other words, in any domain it may be possible to observe changes in Risk one or more locus of change. For example, in the life domain of employment, staff Resources Physical resources and clients may work together to produce changes in affect (how the client feels about Support their employment situation), knowledge (for example, understanding of expectations within Environment Structure (especially constraints) an employment environment), behaviour (job search skills), status (employed full time, part time, unemployed), resources (an up-to-dateHard and soft outcomes CV) and environment (cultural or policy changeSome authors use the terms ‘hard’ and ‘soft’ to broadly categorise outcomes and measures. Hard within a workplace to better accommodate theoutcomes are those that are concrete, countable and externally observable; for example, an needs of the client).increase in the number of days that someone was in paid employment, a move from one housingtenure to another, or the occurrence of a particular behavioural pattern. Soft outcomes are relatedto a person’s internal state and skills, and are typically measured by self-assessment on a scalewith relative response categories (e.g. ‘very satisfied’ through ‘very dissatisfied’) (Segal 1997: 153;Butcher and Marsden 2004; Anderson 2008). 29
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesOne way to think about the distinction between domains and locus of change might be in terms A similar issue exists with regard to theof content and process. Domains relate more to the “subject area” of change, while locus relates domain of education and the knowledge locusmore to process – ‘how’ the client is changing in relation to these domains. Table 7 shows this of change. Following the above suggestion,diagrammatically with a sample set of domains. the knowledge locus of change would be seen as relating to a person’s knowledge inTable 7: Matrix illustrating cross-cutting relationship between domain and locus of change relation to a particular life area (housing, physical health, or in this case the process of Domains Locus of education). Measurement of change in terms change of a person’s understanding of the content of education would fit either with measurements Affect Knowledge Behaviour Status Resources Environment within the status locus of change for the education domain (such as whether a person Housing had completed and passed a particular course), or in the knowledge domain for the life area Physical health corresponding to the content of the course. Mental health 5.6 Point of intervention (Prevention vS. aMelioration) Legal In recent years, government policy has raised etc. the profile of efforts to prevent homelessness for at-risk groups, placing this on a par with the importance of assistance for peopleThis cross-cutting arrangement appears useful, as it allows both workers and clients to be clearer already experiencing homelessness (e.g.about the types of change they might pursue within particular life areas, and to identify and Commonwealth of Australia 2008).celebrate a range of achievements within life areas, rather than limiting the focus to (for example)status change. The distinction between prevention and amelioration relates to the timing of serviceOne area that needs clarification in this model is where a domain appears to cover the same provision in relation to a potential or actualterritory as one of the loci of change, or a domain is conceptualised in terms of a locus of change. episode of homelessness. This is a separateFor example, the domain of mental health could be seen as occupying similar territory to the dimension that can be added to the domainsaffect locus of change. There are two possible responses to this: and locus of change model without requiring• Define the affect locus of change as pertaining to a person’s emotional response or attitude changes to the model. For example, a towards a particular life area, rather than their affect in general. In the domain of mental preventative intervention may be in relation to health, therefore, affect change would relate to how a person feels about or regards their a threatened tenancy breakdown. In addition mental health. A persons global mood condition would then be seen as fitting within the to the domain of housing, an assessment may status locus of change within the mental health domain. explore some of the reasons why the tenancy• For the mental health domain, the measurement of a persons affectual state (whether global has become at risk, for example in domains of or specific to a particular area) is seen as fitting within the affect locus of change. a person’s mental health, family situation or employment. Support and resources might beI would suggest that the first approach is clearer as it fits with the approach that locus of change provided in each of these areas, and outcomeis about a persons capabilities in a particular area. Thus, affect locus of change is always seen as measurement would also focus on these areas.a persons affect in relation to a particular life area. 30
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesSpecific goals and outcomes measured may differ, however it is suggested that the framework 5.7 HouSing and non-HouSing outcoMeSof domains and locus of change is equally applicable to work at any stage along the timeline ofan episode of homelessness. Table 8 below includes some examples of outcomes that might be Conceptualising and categorising outcomes inobserved in the domain of housing, for preventive- and amelioration-focused service delivery. terms of domains and loci of change can give the impression that multiple outcome types forTable 8: Possible outcomes of prevention and amelioration activities, categorised by domain and an individual are independent of each other. Tolocus of change some extent this is true; as Segal (1997: 150) notes, "given the complex hostile environment Domain Locus of change Prevention Amelioration of each case, it would be unusual to find all outcomes moving in the same direction at the Housing Affect Person less anxious about their Person feels more satisfied same time. Some good things and some bad housing situation with their housing situation things will result simply from the nature of the environment..." Knowledge Person has better Person has increased understanding of tenancy knowledge of housing options However, it is also true that outcomes in rights and responsibilities particular domains may influence outcomes in other domains. One example is the relationship Behaviour Person pays rent more Person able to complete private between housing and non-housing outcomes. regularly rental application form Hulse and Saugeres (2008: 39-42) found that housing insecurity is closely related to other Status Continuation of tenancy Positive change in housing types of insecurity (for example, insecurities status (e.g. primary homeless in family life and employment insecurities), Reduced risk of tenancy > transitional, transitional > and impacts on physical and mental health. breakdown housed) Blunden and Johnston (2005: 6, 34) argue that provision of public housing can dramatically Resources Person linked to tenants’ legal / Person linked to private rental improve other outcomes for people who are advocacy service support worker homeless and/or who suffer from mental illness. However, it is also the case that Environmental Landlord agrees to arrears Improved standards in housing stability outcomes may be dependent structure repayment agreement that is rooming houses used by the on support and intervention in other life reasonable for the tenant to person, due to enforcement of domains including mental health and substance maintain standards misuse. For this reason, housing first programs build in case management of varying levels of intensity and duration following a persons access to permanent housing (National Alliance to End Homelessness 2006).See also (Culhane, Parker et al. 2007; Gray 2008) re: outcomes for prevention and earlyintervention programs. The example of public housing also provides a caution about causal inferences. Living in public housing is correlated with poorer socioeconomic outcomes across a range of domains, when compared with other tenure groups. However, this is predominantly not a result of public housing tenure itself, but an allocation effect – i.e. a result of eligibility criteria which target public housing to people with low socioeconomic status and complex needs (Blunden and Johnston 2005: 33-35). See also (Phibbs 2002). 31
    • Literature Review:Measurement of Client Outcomes in Homelessness Services6. Measures and measurement toolsMeasures are the dependent variables of study Table 9 lists a range of criteria for selection of measures and tools, based on NMHWG Information(Weiss 1972: 34). Outcome measures "describe Strategy Committee Performance Indicator Drafting Group (2005: 98-101), Spence, Donald etobservable, measurable characteristics or al. (2002: 24-29), Stedman, Yellowlees et al. (1997: 13-15), and MacKeith, Graham et al. (2007).changes that represent achievement of Evans, Greenhalgh et al. (2000: 383) provide a general framework for organising the criteriaa desired outcome. Outcome measures into four categories: psychometric validation, stakeholder perspectives and user-centeredness,specify exactly what is going to be measured feasibility and utility. These have been adapted as the general categories for Table 9.(indicators) and units of measurement usedto determine the extent to which desired Table 9: Criteria for selection of outcome measures and toolsoutcomes are attained – e.g., HbA1c level(<7.0%) as an indicator of diabetes control" Category Criterion(Post, Isbell et al. 2005: 4). Utility • Acceptability to stakeholdersMeasurement tools are instruments that • Informational value to stakeholdersenable collection of data pertaining to a set of • Comparability of findingsmeasures. Such tools allow data to be collected • Value added to service deliveryin a standardised, consistent way (MacKeithand Graham 2007: 3), and are typically Relevance and user-centeredness • Relevance to service providedaccompanied by instructions for analysis and • Appropriateness to client diversityinterpretation of the data. • User friendliness • SensitivityThis section examines criteria for selectingoutcome measure and tools, and then looks Psychometric validation • Reliabilityat a range of measures and tools that may • Validitybe applicable in the context of homelessnessservices. Feasibility • Cost • Complexity and user competence6.1 criteria for Selecting MeaSureS • Lengthand toolS • InvasivenessMany authors discuss criteria that can be usedto select outcome measures for particular The text below briefly discusses each of the criteria listed.contexts. In general, the criteria apply to bothindividual measures and to measurement tools, Barr (Barr 2008; Barr n.d.) also provides a range of tips for selecting performance measuresalthough in some cases the interpretation may (including outcome measures) and a rigorous framework for defining and documenting measuresdiffer (for example, feasibility with respect to through her PuMP process.an individual measure may involve considerationof whether the item is too invasive to be Acceptability to stakeholdersappropriate in the service delivery environment; Schalock (2001: 134) notes that it is important that stakeholders "buy into" the measureswith respect to a tool, it may involve selected so that the information produced will be seen as trustworthy and will be well-used.consideration of the number of items included). Stakeholder acceptance in part reflects the suitability of the measures on a range of the otherSome criteria relate to innate characteristics criteria articulated here - i.e. measures are more likely to be acceptable to stakeholders if they areof the measures themselves (e.g. validity, perceived as valid, reliable, cost-effective, relevant to the service provided, and so forth. However,sensitivity) whereas others relate to the tools there may also be other factors that influence stakeholder perceptions of measures. Stakeholdersfit with the context of use (e.g. acceptability to previous experience with particular measures or tools, and association of measures with particularstakeholders, relevance to service provided) sectors, organisations, or theoretical orientations, may influence acceptance.(MacKeith and Graham 2007: 8). 32
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesInformational value to stakeholders Comparability of findings Value added to service deliveryFundamentally, an outcome measure or It is essential that measures selected allow for MacKeith and Graham (2007: 5) distinguishtool should provide additional information aggregation in a meaningful way (Baulderstone between tools that are primarily intendedto stakeholders that is not already available and Talbot 2004: 5). This may have a number to provide evidence of outcomes for(Evans, Greenhalgh et al. 2000: 383). However, of dimensions: purposes of accountability, advocacy anddifferent stakeholders are likely to have different • Aggregation within a single service unit knowledge-building, and tools that bothneeds. For example, service delivery staff are allows for examination of patterns of provide evidence, but also support casework.likely to need detailed information on a clients change within the clients who access that Hudson (1997: 72-73) warns of the problemscurrent level of need and functioning in a range service that arise when introduced measures are ofof areas. Management and funders are more • Aggregation on an organisational little direct use to practitioners or clients. Inlikely to be interested in headline indicators level is required for assessment of the an outcomes monitoring system, measuresthat can provide a picture of overall progress performance of the organisational as a that are of clinical relevance and add valuewithin a cohort of clients. A combination of whole to daily service delivery work are likely tomeasures that meet the informational needs of • Aggregation across a service system (at have higher acceptance and greater benefitsall stakeholders is required. program or total cohort level) allows for to stakeholders. Baulderstone and Talbot an understanding of the overall condition (2004: 37-38) also argue that the needs ofIt is worth noting that informational value of clients served by that system, and may clients and workers be prioritised in outcomealso interacts with other criteria, in particular allow for comparison of the performance of measurement systems.psychometric validity and reliability. Scales service delivery organisations.with relatively low reliability and validity may Tools and measures may add value to servicenot provide useful information about individual The basis for aggregation beyond the level of delivery where they:cases, and may only be useful at a system the individual service is the use of consistent • Provide indicators of risklevel for aggregating outcomes over thousands measures across different services. At an • Facilitate regular assessment of areasof cases (Hudson 1997: 73). In a service intra-agency level, this requires agreement on of need that might otherwise not bedelivery environment, it is preferable to use standard outcomes and measures that are discussedmeasures and tools that are sound for work collected by all service units (Hendrick 2010b). • Provide immediate summary informationwith individuals. At an inter-agency level, it requires negotiation to clients and service delivery staff, for of a general sector framework or approach, example in an immediately available reportAnother issue related to informational value is possibly including an agreed theory of change or visual summary (MacKeith and Grahamwhat level of information a particular result on (MacKeith 2007: 5). 2007: 12)a measure provides. For example, if a client • Can be used to identify and celebratescores 3 on a measure of self-care, what An additional consideration is the availability of achievementsdoes this indicate in terms of the behaviour, normative data for a population or social group, • Are based on an explicit model of changeattitudes and skills the client shows? Scales i.e. availability of information on mean values that can help clients to understand thethat have clearly described anchor points, or and standard deviations for a representative possible stages of their journey of changethat are based on concrete questions, are likely sample (Spence, Donald et al. 2002: 27). This (MacKeith and Graham 2007: 12)to provide clearer answers to these questions potentially allows comparison between the • Have a direct link with action planning.(MacKeith, Graham et al. 2007: 9). data that a given agency or service collects, and a wider population. Normative data tends Relevance to service provided to be available for widely used standardised Measures should reflect the nature of the scales. Use of measures that have been used service provided, the types of outcomes in program evaluations may also enable some sought, and dimensions of importance to comparability (Spence, Donald et al. 2002: 28), clients and of use to staff (Baulderstone and although attention needs to be paid to service, Talbot 2004: 5). Relevance may have a number environmental or process factors that can of levels: measures selected should be relevant reduce comparability. 33
    • Literature Review:Measurement of Client Outcomes in Homelessness Servicesto the organisations mission and strategic equally applicable across cultures, and that do of the multiple and complex difficulties facinggoals, should relate to the desired outcomes not show different psychometric properties homeless people, "what may appear to be veryidentified for particular services, and should across cultures (Spence, Donald et al. 2002: small changes can, for some, still be highlyalso be related logically to what the specific 27). Measures and approaches that do not significant." For this reason, it is vital thatservice does and has control over (Schalock reflect cultural competence are likely to measures are sensitive enough to detect small2001: 32, 134). Stedman, Yellowlees et al. produce information that is of little value changes in a persons situation. Post, Isbell(1997: 15) suggest that measures should cover (Mertens 2008), and may have other negative et al. (2005: 24) note that some standardised"the most important and frequently observed impacts. Issues to be aware of include: instruments designed for use with mainstreamsymptoms, problems, goals, or other domains • Differences may occur in understanding populations may not be sufficiently sensitiveof change for the [client] group(s)." of particular concepts across cultures - for for use with homeless populations – the large example, family may have different majority of homeless clients may fall withinMeasures should also reflect aspects of meanings in different cultures. This the low functionality segment of the scalethe service model including the duration of potentially affects validity of measures. without sufficient detail as to the reasons forengagement, nature of relationship established • The medium may need to be adapted to this result.between client and worker, and expectations suit particular groups (including thoseas to potential interim outcomes that may with sensory disabilities or limited literacy) MacKeith and Graham (2007: 13) note thatoccur during the period of engagement (Mertens 2008: 53) measures with three or five point scales will(Elizabeth 2010). Measures for short-term or • The language may need to be modified to tend to have low sensitivity, and recommendcrisis services may be different to those for suit particular groups (Baulderstone and scales of ten points.long-term support services. Talbot 2004: 7) • Some groups may be suspicious of ReliabilityAppropriateness to client diversity information-gathering processes due to Reliability is a general term for theMeasures and tools may be generic (applicable historical events and recent traumatic consistency of measurements; unreliabilityto a wide range of client groups), or may be experiences (Baulderstone and Talbot refers to inconsistency due to randomintended to be used with particular groups 2004: 7). measurement errors (Bloom, Fischer et al.(people with alcohol or drug addictions. people 2006: 68-69). More reliable measures areover the age of 65, ...). It is important that User friendliness those that demonstrate consistency in resultsoutcomes and measures are relevant to the In a service delivery (as opposed to research) when applied to the same person or thingrange of diversity expected within the client setting, measurement tools are most likely repeatedly and independently, under the samepopulation in terms of socio-demographic to be used regularly and successfully where circumstances.factors such as age, gender, household they are presented in a user friendly format.groupings, functional levels, and other factors MacKeith and Graham (2007: 12) note Reliability of measures and tools has a number(Schalock 2001: 32). In some cases, it may the importance of client friendly language of aspects (Spence, Donald et al. 2002: 24-25;be necessary to have several versions of a rather than technical jargon, attractive clear Bloom, Fischer et al. 2006: 70-73):measurement tool that are tailored for use with presentation of the tool itself, and clear format • Test-retest reliability – the level ofparticular client groups (Wells and Johnson for any report or summary. consistency in results when a measure is2001: 191-192; MacKeith and Graham 2007: 9). applied on different occasions under the Sensitivity same circumstances. If the measure isIn addition to their basic applicability to Sensitivity refers to the measures ability not stable and fluctuates in response todifferent groups, the design of measures to indicate whether a clinically significant random changes in the individual or theand tools also needs to be assessed for its change has occurred for a client over environment, it will have lowered reliability.cultural appropriateness and how it handles consecutive administrations of the measurediversity (Schalock 2001: 135). For aggregation (Stedman, Yellowlees et al. 1997: 14). Eardley,purposes, measures should be used that are Thompson et al. (2008: 7) note that because 34
    • Literature Review:Measurement of Client Outcomes in Homelessness Services• Alternate forms reliability – the extent Reliability of measures depends on a range of • Content validity – the extent to which to which responses to two forms of a factors, including the types of items included the particular questions or items selected measurement tool, which differ only in in the tool and the way that they are worded for measurement are representative the content of the individual items, are and presented. One key issue is the extent and adequately cover the domain being correlated – this indicates the extent to to which users of the measure have different assessed, or are a biased or limited sample which the different measures do measure understandings of the language used. In some of what the tool is intended to measure. the same concept. cases, measures may be inadequately defined • Criterion validity – the extent to which• Internal consistency reliability – the so that users end up providing a rating based an individuals rating on a measure is extent to which parts of a measurement on their best guess rather than with clear predictive of the individuals performance tool (e.g. different items within a tool) guidance (cf. the example of Wells and Johnson or status on certain related outcomes are homogeneous or measure the same (2001: 181-182) in relation to measures of (the criterion against which the measure thing. Tools with high internal consistency re-referral in child protection settings). Training is assessed). The validity of the criterion reliability will contain a set of items that and access to a data dictionary can help to must already be established. The higher are highly correlated with each other, increase inter-observer reliability. the correlation between measure and indicating that the tool measures a single criterion, the better the criterion validity. concept. Validity • Construct validity – the extent to which• Interobserver reliability – the extent to Validity refers to the extent to which a empirically observed relationships among which two or more observers from the measure or tool actually measures what it is measures of concepts agree with the same group are consistent with each intended to measure (Bloom, Fischer et al. inter-relationships among concepts other when they independently observe or 2006: 75). Validity therefore affects whether, predicted by theory. This may include rate a particular thing. This is particularly and to what extent, one can trust the results convergent validity (measures co-vary in relevant to a routine monitoring setting of particular measures, as interpreted for the predicted manner with other variables where multiple staff members (or clients) specific purposes and with reference to specific that they should correlate with in theory), independently complete particular tools. concepts or constructs (Cook and Beckman and discriminant validity (measures do Interobserver reliability may also be 2006). Measures are not valid in the abstract, not correlate with variables that theory affected by the extent of information and but for particular populations and settings. predicts they should not correlate with). training provided to users of the tool.• Cross-informant agreement – the extent to Validity also has a number of aspects (Spence, Some issues that need to be considered which observers from different groups (e.g. Donald et al. 2002: 25-26; Bloom, Fischer in relation to validity are the reactivity of a client, a housing worker and the clients et al. 2006: 76-83), although more recent measures (the extent to which changes may carer) agree in their ratings on a particular views suggest that these could all be grouped be caused by the clients awareness of the measure. Low cross-informant agreement together under the overarching framework measurement process itself rather than by the is common and it can be difficult to of construct validity (e.g. Cook and Beckman intervention), and observers biases – e.g. the tell whether the differences are due to 2006: 166.e110). desire to present oneself in a socially desirable inconsistent application of the measure, • Face validity – the opinion of some manner, or expectations about particular or differences in behaviour in different observer that at face value the measure people based on their socio-demographic environments. appears to measure what it is designed background (Bloom, Fischer et al. 2006: 81). to measure. While face validity can aidReliability of measures or tools can be tested user acceptance of the measure, it is not Both reliability and validity are partly aand rated as a correlation between 0.0 and quantifiable and is subject to differing reflection of the development process of the1.0, where 1.0 represents perfect reliability. judgements of differing observers, hence tool. An iterative process of developmentIn general, correlations of at least 0.70 are some sources argue that it should not be with repeated cycles of testing and revisionrecommended in order for the measure to be considered a form of validity at all (Cook allows for improvement over time of the toolsuseful, however reliability of 0.80 or above is and Beckman 2006: 166.e112). reliability and validity as well as other aspectspreferable (Bloom, Fischer et al. 2006: 74). of its design (MacKeith and Graham 2007: 10). 35
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesCost 6.2 How Many MeaSureS? formal education and training (BaulderstoneOperation of an outcomes monitoring system and Talbot 2004). Some tests require a high In the absence of detailed information onhas a range of costs. While there will always level of training in administration, scoring and/ the service and organisational context, it isbe a baseline investment needed to operate or interpretation (Spence, Donald et al. 2002: not possible to specify an absolute numbersuch a system, the actual cost may vary widely 26-27). In addition to increasing training costs, of measures as the right number to use.depending on the particular measures or tools complex tools are likely to be less reliable However, the literature does provide guidanceselected. It is important that the measures when administered by multiple staff. Whichever on factors to keep in mind in relation to theselected be affordable for the organisation tool is used, the availability of training, support number of measures implemented. Two distinct(Schalock 2001: 32, 134) and represent value and documentation will be important (MacKeith themes emerge from the sources reviewed.for money. Analysis of outcomes in some and Graham 2007: 10).domains may require a depth of information One strand of thinking emphasises thecollection that is unaffordable (Rapp and Some tools are restricted to purchase and use importance of limiting the number ofPoertner 1992: 104). by specific professions, such as registered measures: "select a minimum number of psychologists (Spence, Donald et al. 2002: outcomes that are relevant and obtainable"Specific costs to consider include: 26-27) and would therefore be inappropriate (Schalock 2001: 32). McDaniel (1996) advises• Licensing costs associated with for most homelessness services. “dont measure it unless you plan to change standardised tools - some tools are it”; Berman & Hurt (1997: 87) emphasise available free of charge, whereas others Length that organisations should only measure what are licensed on a per-service or per-client Measurement tools for routine monitoring they will use. A number of authors suggest basis (MacKeith and Graham 2007: 11) need to be feasible in terms of their length keeping the number of measures small• The administrative load (Pleace 2008: 64) (number of items) and the time required to initially and letting the system evolve with on service delivery staff of collecting and complete them (MacKeith and Graham 2007: experience (e.g. NMHWG Information Strategy entering outcomes data, or alternatively 10). However, what is feasible will depend on Committee Performance Indicator Drafting additional staff resources to collect and the nature of the service and the relationship Group 2005: 99). The main benefits cited for enter data between staff member and client: services the "minimum measures" approach are that• The costs of staff time to clean, aggregate, with longer-term engagement are likely to the system is more likely to be manageable, analyse and report on data (Wells and be able to use a lengthier tool. Tools that are will be less confusing for staff, will impose less Johnson 2001: 192-193) overly cumbersome are likely to have a low administrative burden on the organisation, and• Computer equipment and software rate of completion. will therefore be more likely to be sustainable. required to store and process data Rapp and Poertner (1992: 103) suggest that• Extent of training required for staff who Invasiveness having too many measures tends to dilute will be using the measures and tools (Wells The nature of the particular items included in organisational focus and lead to paralysis, and Johnson 2001: 192-193). a tool also needs to be commensurate with because it gives the message that everything the setting and relationship between staff and is equally important.Cost will be affected by a range of factors clients. Invasive or highly personal or sensitiveincluding the number and complexity of the questions may not be appropriate in settings Another strand of thinking points out themeasures collected. where only brief or superficial contact occurs. benefits of having a greater number of Some questions (e.g. in relation to sexuality) measures, and/or more detailed measures.Complexity and user competence may also be perceived as inappropriate or More detail provides greater information onTools that are complex and difficult to use or raising a safety risk in some situations (e.g. the areas where services are succeeding orscore present serious challenges, especially between a female worker and male ex-prisoner failing (Pleace 2008: 64). To minimise errorwhen data collection is to be undertaken by client). In general, questions in relation to and/or bias, Stedman, Yellowlees et al. (1997:service delivery staff, who may have a wide sensitive areas should only be asked if the 20) recommend using multiple assessmentvariety of backgrounds, skills and levels of service has the capacity to follow up with methods, multidimensional assessment support in that area if needed. 36
    • Literature Review:Measurement of Client Outcomes in Homelessness Servicesmethods, and if possible using more than one of potential concerns or aspects of a clients and is assumed to be generalisable torater for each measure or domain (see also condition, can give a more rounded picture of the broader system of care. For example,Bloom, Fischer et al. (2006: 307-308)). To the a clients situation, and may indicate not just in primary care, asthma, diabetes orextent possible, measures should examine whether a particular condition has changed hypertension might be chosen as a tracerthe perspective of all those involved in the but whether the clients overall quality of life condition. Findings concerning processesservice (client as well as service providers). has improved. The Outcomes Star suite is one and outcomes of care for the tracerSimilarly, Schalock (2001: 135) comments that example of a measurement tool that prioritises condition have implications for the entirehuman behaviour is not singular, and that breadth over depth, with only a single measure organisation and are assumed to reflectmeasures should reflect the complexity of in each domain. The disadvantage of this type the functioning of the entire system of carethe human condition and of services desired of approach is that the level of insight into (Booth and Smith 1997: 38-39). By analogyoutcomes. Multiple measures also may make any one domain (and potentially the level of in a homelessness setting it might beit more difficult to fudge the data - i.e. for reliability and validity) is limited. Approaches possible to argue that a particular domainthose collecting data to hide failures or fake that focus on only a few outcomes areas, but or domains were tracer domains thatsuccesses (Pleace 2008: 61). At times, multiple examine them in more depth, can provide reflect the quality and effectiveness of themeasures may be needed to balance each much more useful information in these organisations services overall. Researchother – to avoid the distortion of service domains but may miss changes in other areas would be required to establish the validitydelivery which can follow a narrow focus on a of a clients life (Hatry 1997: 17-18). of the assumption that it is possible tosingle measure of success (Rapp and Poertner generalise from the tracer domain to the1992: 102). It is possible to make an organisational entire system of care. decision to opt for either breadth or depth, butIntegration of the two strands suggests a need there are also a few other possible responses For further research: does the suggestion ofto balance the "specificity and rigour" of the to this issue. a tracer domain model hold up, has anyonemeasurement process with the expense of • Use a modular system of in-depth applied this outside of primary health, anddata collection, to ensure that the usefulness measures for particular domains, in how could the validity of generalisations beof information gained is not sacrificed to false conjunction with a generic suite of core established?economy, while at the same time avoiding indicators covering the full breadth ofmeasures that are too resource intensive to be issues at a shallow level. For example, When opting for a breadth approach, it willimplemented effectively (Wells and Johnson multi-item modules might be developed usually be necessary to select only one or a2001: 192; Pleace 2008: 64). Ultimately the in the areas of housing, mental health, few measures for each domain. One possibilitydecision depends on context (Burns and Cupitt physical health or legal issues if these is to select items from various standardised2003: 27). were seen as relevant to the delivery instruments that load most strongly on the of particular services. However, these dimension being measured. Factor analysisBreadth vs. depth modules might only be implemented in may be required to establish the strengthGiven limits on the number of measures that certain teams, or in respect to certain of loadings. However, Spence, Donald et al.can feasibly be collected, there will usually clients, where the measures were relevant (2002: 28-29) note that the psychometricbe trade-offs between breadth and depth of and feasible. Other services might use only properties of the original instrument/s nomeasurement. These tradeoffs are a feature the generic core indicators. The BT Generic longer apply, and the revised measures shouldof any evaluative activity (Patton 1987: 46-48) Outcomes Scale (Baulderstone and Talbot be piloted and shown to have acceptablebut can be particularly heightened in the 2004: 65-74) provides one example of a reliability and validity. Psychometric expertresource-limited environment of outcomes modular tool where blocks of questions can advice is recommended in this process.monitoring systems. be extracted for use with particular clients. • Use of tracer conditions. In health care,There are advantages to the whole client a tracer condition is a single condition thatapproach, which seeks to cover a wide range is measured throughout an organisation 37
    • Literature Review:Measurement of Client Outcomes in Homelessness Services6.3 tyPeS of data and tyPeS of MeaSureSThere are a variety of different types of measures which can be used to assess outcomes, anda variety of sources of data which can be used as the basis for completing the measure. Table10 summarises a range of measure types which could be used in outcomes monitoring; theinformation is integrated from discussion in Bloom, Fischer et al. (2006: 169-313), Burns and Cupitt(2003: 22), Rapp and Poertner (1992: 106-107) and Baulderstone and Talbot (2004).Underlying all measure types, Bloom, Fischer et al. (2006: 63-64) identify four levels ofmeasurement, which provide increasing levels of information about the thing measured:• Nominal level – assignment to one of a set of mutually exclusive categories (e.g. male / female)• Ordinal level – as per nominal-level measures, plus information about the relative amount of one category compared with another (categories are rank-ordered on a scale – e.g. never, sometimes, often, always)• Interval level – as per ordinal-level measures, plus that adjacent intervals are equal so it is possible to determine how much more one category is compared with another (e.g. Celsius scale for temperatures). It is often reasonable to use standardised scales with the assumption that they are interval-level measures.• Ratio level – as per interval-level measures, plus the categories have a defined zero point with an intrinsic meaning (e.g. age).Table 10: Selected measure types and associated data sources Measure type Locus of change Data source/s Measurement / recording tool 1. Individualised rating scale Any Self-administered Typically paper- – typically rating frequency, questionnaire; client based or electronic extent or severity of some response to interview questionnaire condition, satisfaction with question; rating by a some aspect of a situation, practitioner, relevant ability or agreement with other party, or some statement independent evaluator 2. Standardised scale – Any – depends Client self-report, Typically paper- combines a set of individual on the specific including self- based or electronic items / measures which may scale administered questionnaire be of various types including questionnaire (e.g. rating scales, concrete Rapid Assessment questions etc. Usually have Instrument); rating by known reliability and validity a practitioner, relevant level and may be normed. other party, or May provide an overall score independent evaluator or ‘index’ 38
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesMeasure type Locus of change Data source/s Measurement / recording tool3. Status maintenance and Status, May be client Various – maychange scale – assigning a Resources self-report but be included inclient’s status as one of set usually provided by questionnaire orof designated categories, practitioner or other recorded separatelynormally arranged in a party on basis ofprogression from undesired knowledge of client’sto desired status situation. Data may already be present in routine agency data collection (e.g. housing status)4. Level of functioning Behaviour Client self-report Typically paper-scale – rating client’s (self-administered based or electronicfunctioning or performance in questionnaire, questionnaire;particular domains on a scale response to interview Outcomes Starfrom low level functioning question); practitionerto competent, independent or third party ratingfunctioning5. Goal attainment Any Rating negotiated Paper-based orscale – individualised rating between client and electronic goalscale anchored by a set worker, or may scaling toolof ‘expected’, ‘more than be independentlyexpected’ and ‘less than completed by bothexpected’ outcomes which and then comparedare jointly agreed by clientand practitioner at baseline6. Standardised goal Any Client self-rating or Paper-based orscale – rating scale from ‘no practitioner rating, or electronic goalprogress’ to ‘fully achieved’, independent ratings achievement formwith goals referenced to by bothstandard goal list7. Behavioural count – Behaviour Observation – either Various – paper-counts of frequency and/ self-monitoring (e.g. based checklist,or duration of specific occurrence of self- small objects,behaviours deprecating thoughts) electronic counters, or direct observation etc of overt behaviours 39
    • Literature Review:Measurement of Client Outcomes in Homelessness Services Measure type Locus of change Data source/s Measurement / eventually result in the outcomes tracked by recording tool the lag indicator. A possible application of this 8. Simple observation Any – but Simple observation Paper-based or in the homelessness service context would measure – qualitative inferences e.g. of physical electronic be to establish lead indicators around interim information and/or become more attributes, expressive outcomes (e.g. successful public housing quantitative ratings problematic with movements, physical applications; successful negotiation of arrears ‘internal’ loci location or language repayment agreements) that predict medium- such as affect behaviour. to long-term outcomes. Barr suggests that lead indicators need to be tracked with greater Physical traces frequency than lag indicators. (erosion / accretion) – practitioner or third- Vantage point party observation Vantage point refers to the party or 9. Qualitative information Any Client’s written Client log; paper- parties who are the source of a specific about the nature, extent self-report; client’s based or electronic rating – typically the client, the practitioner, and context of client needs, verbal self-report, case notes and or a relevant third party such as a parent, problems and progress practitioner client files teacher or spouse. It should be noted that observation, third vantage point is not necessarily only that of a party report single person (ratings could be agreed jointly between client and worker, for example). 10. Case status count – e.g. Tends to Organisational records Various Vantage point may also be different from number of re-referrals, be linked to recorder role (for example, a worker may number of clients on at-risk behaviour and complete a measurement tool on the basis of register, etc. status ratings provided by the client). Vantage point is a key consideration inMeasurement tools and packages may include a variety of different types of measures, depending outcomes measurement because it is commonon the purpose for which they are designed and the types of change they aim to measure. Using to find a lack of agreement in ratings bya combination of measure types and data sources can strengthen findings and may also have clients, staff and independent observers onbeneficial effects on practice (Segal 1997: 157; Burns and Cupitt 2003: 28). effectiveness measures (Rapp and Poertner 1992: 105). In the homelessness sector it isQualitative and quantitative measures generally accepted that client involvement inQuantitative measures are typically used as the basis of outcomes monitoring systems, in part outcomes ratings is important, and that thebecause of the ease of aggregation they provide. However, a number of sources also suggest assessment of client outcomes without theirsupplementing these with qualitative information (Spence, Donald et al. 2002: 29; Burns and knowledge may be questionable ethicallyCupitt 2003: 16; Elizabeth 2010). Qualitative data can provide rapid feedback, identify areas of (Baulderstone and Talbot 2004: 7). Anychange that "fall through the gaps" of the quantitative measures, establish the context for change measure that attempts to assess client qualityand interpretation of results, and provide a more authentic view of the clients lived experience. of life, satisfaction, affect or perception mustTo make analysis feasible qualitative data needs to be brief, focused and/or readily searchable for be based on the clients viewpoint (Hudsonkeywords. 1997: 76; Ware 1997: 51-52).Lead vs. lag indicators However, multiple vantage points can provideLead indicators are measures that can be used to predict change in another performance measure a more reliable estimate of change (Rapp and(the lag indicator) (Barr n.d.: 13). They typically measure results of earlier stages of a process that Poertner 1992: 105). In particular, practitioner 40
    • Literature Review:Measurement of Client Outcomes in Homelessness Services Generic vs. condition specific or However, there are also a number of potentialobservation can provide a useful addition program specificto self-report for clients who are under the disadvantages to using standardised scales ininfluence of alcohol or drugs, have mental Generic tools are intended to be used across the homelessness context (Weiss 1972: 36;health issues, have low self-awareness or a wide spectrum of client groups, whereas Baulderstone and Talbot 2004: 10; Post, Isbelllittle insight into their own behaviour. In such condition-specific or program-specific tools et al. 2005: 23-24):circumstances, clients may either over-estimate focus on specific indictors relevant to the • Some standardised scales are costly to useor under-estimate their difficulties (Burns and needs and condition of a particular client due to licensing feesCupitt 2003: 28). group. Generic measures are valuable in • Standardised scales may not cover all the allowing comparisons between different domains of interest to a serviceType of anchor points; theories of change client groups (for example, across services • Use of multiple standardised scales canMeasures using scales require anchor points or organisations), however they may be be time consuming (although in someto allow users of the measure to select the more error prone (Pleace 2008: 60). Specific cases there are shortened versions of theappropriate rating. In general, the more clearly measures may have higher user acceptance scales available that can be used as Rapiddefined the anchor points are, and the more and utility in the service delivery environment Assessment Intstruments)direction that raters have as to the factors to but make comparisons difficult (Stedman, • Standardised scales are often associatedbe considered and their relative importance, Yellowlees et al. 1997: 16). with particular conceptual or servicethe more reliable the measure is likely to be contexts; use of tools outside of the(Weiss 1972: 41). Standardised vs. self-developed tools contexts for which they were intended can There is mixed opinion in the literature on the rely on a series of unproven assumptionsSubjective scales that relate to how the relative merits of using existing standardised which cast doubt on the validity of theclient feels about an area of their life (e.g. scales, as opposed to an organisation measures in the new setting.very satisfied, somewhat satisfied, neither developing their own measures and tools. Key • Standardised scales may be designed forsatisfied or dissatisfied, somewhat dissatisfied, advantages of using standardised tools include use with the general population and mayvery dissatisfied) do not have clearly defined (Rapp and Poertner 1992: 105; MacKeith and not be sensitive enough when used withanchor points. On the other hand, defined Graham 2007: 9-10): clients with complex or severe issues.scales have clear descriptors of behaviours or • Developing and piloting new tools is acircumstances associated with each step of the resource-intensive process MacKeith and Graham (2007: 9-10) point outscale. These scales can be useful in improving • Existing tools have generally already been that developing a new tool (or adapting anreliability, providing more objective criteria for tested and revised to iron out problems existing tool) can help to engage stakeholdersratings and therefore allowing more meaningful • The validity and reliability of most and build an outcomes orientation within ancomparison between the ratings for different standardised scales is known. There agency.clients (Segal 1997: 153; MacKeith and Graham is evidence that agency evolutions of2007: 5-8). tools can sometimes lead to instruments Goal scaling that have poor psychometric properties Goal scaling (Rapp and Poertner 1992: 114;Tools using defined scales may also be based (Culhane, Parker et al. 2007: 12.21) Baulderstone and Talbot 2004) is different toon an explicit theory of change, which includes • Existing tools may come with other approaches to outcome measurementdescriptions of the different points on the documentation, training and/or software to in that instead of attempting to measure thejourney towards the desired end outcome. support data collection clients condition, it measures the client and/MacKeith and Graham (2007: 12) argue that • Standardised tools are more likely to or practitioners assessment of the degree oftools which use an explicit journey of change support comparison across client groups progress or goal attainment that has occurred.are more powerful as casework tools because and with normative data Goal attainment scaling is content-rich withthey "help clients to understand the change • Standardised tools may carry more specific goals and outcome expectations atprocess they are engaged in and the particular credibility with funders. the level of client and worker conversations.challenges and rewards of different stages in However, reported ratings are content freethe journey." 41
    • Literature Review:Measurement of Client Outcomes in Homelessness Servicesin the sense that they summarise the extent The first two sub-headings (quality of life andof progress but do not indicate the domain, wide-spectrum homelessness tools) includefocus of change, or how one clients progress tools which aim to establish a rounded pictureor condition compares to anothers. Standard of a persons situation across multiple domains.goal scaling extends basic goal attainment The remaining areas focus on measures andscaling by linking progress ratings to specific tools targeted to more specific domains and/goals from a standard goal list, thus providing or client groups. However, many of the moreinformation about domain and potentially locus specific tools are also influenced by wholeof change. However, ratings are still relative person models of health or wellbeing, andto the clients starting point, thus making bridge a number of domains.meaningful comparison between clients orservices difficult. While the focus is mainly on complete measurement tools, in a few instancesGoal scaling approaches tend to integrate well individual measures are included where theyinto casework and are popular with service appear to fill a gap where I have not founddelivery staff as they have a natural fit with a suitable existing tool. In general theseaction planning. Their weaknesses in terms measures are drawn from program evaluationsof comparison and lack of defined external or other academic studies.anchors could potentially be reduced bycombining them with other measures such as Several sources provide a comparison orstatus maintenance and change scales (Talbot audit of measurement tools across a range2010). of dimensions (e.g. Stedman, Yellowlees et al. 1997; Spence, Donald et al. 2002; Campbell-6.4 PoSSiBle toolS and MeaSureS Orde, Chamberlin et al. 2005; MacKeith, Graham et al. 2007; Anderson 2008). TheseThis section lists a variety of tools and provide a useful reference when consideringmeasures that are of potential utility to the benefits of particular standardised tools.homelessness service agencies in measuring Another resource is Schalocks (2001: 156) listclient outcomes. It is understood that while the of generic measures which are applicable to alack of permanent safe housing is the common wide variety of contexts.thread that links all people experiencinghomelessness, homelessness services workwith clients on needs, problems and goals ina wide range of life domains: physical health,mental health, relationship difficulties, legalissues, problematic substance use, and soforth. In the SAAP context, self-relianceincludes a wide range of aspects ranging fromaccommodation, income and self-care throughto connectedness and relationships with otherpeople, and access to appropriate forms ofsupport (Eardley, Thompson et al. 2008: 14).The discussion here therefore covers a rangeof domains. 42
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesQuality of lifeThere are a variety of approaches to measuring Quality of Life (QoL), and many differentinstruments have been developed. Table 11 lists several QoL tools of relevance to homelessnessservices.Table 11: Selected Quality of Life tools of relevance to homelessness services Name of tool Domains / areas covered Measures Notes WHOQoL- Physical health, 26 items, 5-point Produced by the World Bréf psychological, social rating scales, Health Organisation. A relationships and self-report condensed version of the environment 100-item WHOQoL tool. Has been validated in many different cultural settings. Australian versions are available. An 8-item version is also available. Registration is required prior to use. (World Health Organisation 1996; University of Melbourne 2010) Lancashire General wellbeing, Mix of quantitative Free to use. Information Quality of work and education, concrete questions available from Institute of Life Profile leisure and community (e.g. current Psychiatry, Kings College, (LQoLP) participation, religion, accommodation) and London. finances, living 7-point satisfaction situation, legal and rating scales safety, family relations, social relations, health, self-conflict 43
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesName of tool Domains / areas covered Measures NotesQuality of Health, health care Includes an impact Can be used in modular form.Life for system, place where survey (mix of (Hubley, Russell et al. 2009)Homeless you live or stay, living category selectionand Hard conditions, financial and 7-point ratingto House situation, employment scales) and MDT toolIndividuals situation, social and (4-point and 7-point(QoLHHI) support services, rating scales) for each recreational and leisure major domain activities, spiritual life, romantic situation, family, and friendsManchester Life in general, health, 43 items, combination Developed by the InstituteShort work and education, of 7-point rating of Psychiatry, Kings College,Assessment finance, leisure, social, scales and yes/no London. Questions areof Quality safety, living situation, questions framed to be applicable toof Life family a range of living situations(MANSA) including homelessness. (Priebe, Huxley et al. 1999) 44
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesHomelessness (wide spectrum)A number of tools provide a broad overview of a persons situation across multiple domainsrelevant to people experiencing homelessness. Often the focus is on functioning – i.e. how wella person is managing particular areas of their life. The CANSAS tool, while developed for usein mental health services, provides such a broad overview of relevant life areas that it is alsoincluded here.Table 12: Wide-spectrum outcomes tools of relevance to homelessness services Name of tool Domains / areas covered Measures Notes Homelessness Self care and living skills, managing 10-point defined (MacKeith, Burns et al. Star tenancy and accommodation, scale based on 2008a) managing money, social networks explicit journey of and relationships, drug and alcohol change misuse, physical health, emotional and mental health, motivation and taking responsibility, meaningful use of time, offending BT Generic Housing, Govt allowance, Financial 3-point importance (Baulderstone and Outcomes counselling / support, Gambling, rating scales, Talbot 2004) Scale / Education / training, Employment, 5-point change Environmental Incest / sexual assault, DV, Family rating scales Outcomes / relationship, Pregnancy, Family Scale planning, Living skills / personal development, Legal issues, Recreation, Health (General), Health (Mental Health), Emotional support / other counselling, Drug and alcohol issues, Material Goods, Community Resources, Safety Camberwell Accommodation, Food, Looking 22 items, 3-point Developed by the Short after the home, Self-Care, Daytime category choice Institute of Psychiatry, Assessment activities, Physical Health, Psychotic for each item (no Kings College, of Need symptoms, Information on condition need, met need, London. A variety of (CANSAS) and treatment, Psychological unmet need) versions are available distress, Safety to self, Safety to oriented to research, others, Alcohol, Drugs, Company, clinical and forensic Intimate relationships, Sexual settings. CANSAS-P Expression, Child Care, Basic also available - self- Education, Ability to use telephone, administered version Transport, Money, Benefits (Trauer, Tobias et al. 2008) 45
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesName of tool Domains / areas covered Measures NotesDraft service Experience of homelessness, Various - primarily (Pleace 2008: 110-111)outcomes substance misuse, social support, yes / no questionsreturn activity, health and wellbeing, and frequency(substance housing situation rating scalesmisuse andhomelessness)Colorado Activities of daily living, legal status, A few concrete (Post, Isbell et al.Coalition for quality of life, housing quality, questions, plus 2005: Appendix 1)the Homeless employment / school, access to 7-point scalesConsumer benefits, physical health, mental oriented toOutcome health, substance use, social functioning andScales resources, basic needs, family / status, to be household communication, youth completed by risk, childcare practitioner during interview with clientOutcomes field Life skills, mental health, substance 5-point rating (Post, Isbell et al.test - Montana, abuse, family relations, mobility, scales oriented to 2005: Appendix 4)2005 community involvement status, resources and functioningERoSH Tenant Quality of life, health, social 26 questions, (ERoSH 2010)Self- networks and involvement, skills 5-point ratingassessment and hobbies, environment, dealing scales Orientedquestionnaire with finances and administration, to tenants living in cultural and religious needs sheltered housing. 46
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesHousing Phibbs (2002: 4-5) provides a usefulThe literature review has not identified any existing standardised tools that focus on the domain model of the different scopes to considerof a persons experience of and functioning in relation to their housing. However, there is a when assessing housing: tenure, dwelling,substantial body of research and evaluation literature that has measured different aspects of neighbours, area, and community.housing outcomes. The key dimensions that tend to be measured are housing type, tenure,affordability, stability, and quality (adequacy and appropriateness) (Kolar 2004; Blunden andJohnston 2005: 1).Housing status (e.g. primary, secondary or tertiary homeless, interim housed, permanentlyhoused, housed at risk) can be used as an index drawing together a number of these dimensions.Housing status could be measured at a point in time, or as a proportion of time spent in varioushousing status categories during a particular period (Clark and Rich 2003: 79-80).Table 13 provides some sample measures and approaches to measuring these dimensions of apersons housing situation.Table 13: Example measures related to housing Dimension Possible measures Housing type and Typically nominal status categories are used, in some cases an tenure ordinal scale from ‘worst’ to ‘best’ housing options although this can be problematic as perceptions of what is better can vary and are context-dependent Affordability Typically considered as a proportion of a person’s income – 30% is conventionally considered the limit of affordability (Kolar 2004) although for people with complex needs on low incomes, the figure may need to be lower Stability Stability has retrospective, prospective and perceptual components • Retrospective: housing mobility (e.g. number of moves in previous given time period) • Prospective: risk of tenancy breakdown – one approach is to use a list of ‘warning signs’ (e.g. arrears, subject to Notice to Vacate, has received breach notice etc) (Pleace 2008: 70-71) • Perception: rating scale – self-report of how stable a person believes their housing is (Kolar 2004) Quality Can measure the existence of specific negative aspects of housing (e.g. mould, infestation, etc - Pleace (2008: 70)); can measure overall satisfaction with housing (rating scale); or could develop a tool rating satisfaction with a range of aspects of housing (condition of the building, location, health impact, neighbour relations, etc) 47
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesPsychological wellbeingThere is a vast array of standardised scales available in the area of mental health. Spence, Donaldet al. (2002) provide a detailed list of measurement tools and scales covering both generic anddiagnosis-specific aspects of mental health. NMHWG Information Strategy Committee PerformanceIndicator Drafting Group (2005) provides a general framework for performance measurement inthe mental health field in Australia (including measurement of client outcomes).Mental health tools can be divided into two main groups: clinically-oriented tools and recovery-oriented tools. Table 14 lists a number of clinically-oriented tools; the first five are the mostwidely-used in mental health services in Australia. Stedman, Yellowlees et al. (1997) field testedand compared these tools in a variety of settings. See also (Eagar, Buckingham et al. 2001; SigginsMiller Consultants 2003).Within Victoria, certain tools are mandated by the Department of Health for collection of outcomesdata by clinical and PDRSS mental health services. Clinical services are required to use the HoNOSand LSP tools, while BASIS-32, WHOQoL and CANSAS are recommended for use by PDRSSservices (Department of Health 2009).The short form (SF-36) is listed under the general health heading below rather than in this table.Table 14: Selected clinically oriented mental health outcomes tools Name of tool Domains / areas Measures Notes covered Behaviour 5 sub-scales: Relation 32 items, 5-point Not in the public and Symptom to self and others, scales, client-rated domain - copyright Identification Scale daily living and McLean Hospital (BASIS) role functioning, although an depression and arrangement exists anxiety, impulsive and through DHS for use addictive behaviour, of BASIS-32 in PDRSS psychosis services in Victoria. (McLean Hospital 2010). BASIS-24 also available Health of the Nation Behavioural problems, 12 items, 5-point (Buckingham, Burgess Outcome Scales impairment, scale, clinician-rated et al. 1998b: 112-118) (HoNOS) symptomatic HoNOSCA / HoNOS problems, social 65+ also available problems Life Skills Profile (LSP) Self-care, 39 items, 4-point LSP-16 (abbreviated non-turbulence, scales completed by version) also available social contact, clinician or family (Buckingham, Burgess communication, member et al. 1998b: 119-122) responsibility 48
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesName of tool Domains / areas Measures Notes coveredRole Functioning Working, independent (Stedman, YellowleesScale (RFS) living and self care, et al. 1997: 29) immediate social network relationships, extended social network relationshipsKessler 10 Scale Focus on 10 items, 5-point Also 6 item and(K-10) psychological distress rating scales 15 item versions or symptoms, completed by client available. particularly of anxiety (Department of and depression / Health and Ageing dysphoria 2003: 57-60)Depression Anxiety Depression, anxiety 42 items, 4-point (University of Newand Stress Scale and stress rating scales South Wales 2010)(DASS) completed by client Public domain. 21 item version also available.General Health Somatic symptoms, Available in 12-, 28-, Not public domain.Questionnaire (GHQ) anxiety and insomnia, 30- and 60-item Focus on non- social dysfunction, versions; four-point psychotic psychiatric severe depression scales (self-report) disorders. 60-item version covers physical illness as well.Clinical Outcomes in Subjective wellbeing, 34 items, 5-point (Core Systems GroupRoutine Evaluation problems / symptoms, scales, self-rated 1998)(CORE) Outcome life functioning, riskMeasure / harm 49
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesIn recent years a range of recovery-oriented measurement tools have emerged. In contrast (Schalock 2001: 143-144) also considers awith clinically-oriented tools these tend to focus more on the clients experience and may cover range of mental health oriented measures,a broader range of life areas. Campbell-Orde, Chamberlin et al. (2005) is a useful resource on and notes that mental health issues may haverecovery tools. Table 15 lists a few examples. significant impacts on families and carers as well as the primary client. It would be useful toTable 15: Selected recovery-oriented outcomes tools consider how these broader outcomes could be measured. Name of tool Domains / areas covered Measures Notes Recovery- Various 28 items, select REE is a suite of Enhancing all that apply tools designed to Environment assess organisational Measure (REE) performance, includes Consumer a consumer self-report Self-Report Tool tool. Copyright Priscilla Ridgway. (Campbell- Orde, Chamberlin et al. 2005: 75) Recovery Coping, empowerment 41 items, (Siggins Miller Assessment Scale and wellbeing, personal 5-point scales Consultants 2003: 87) relationships Carers’ and Users’ Where you live, finances, 16 items, Plain language, clear Expectations of use of time, family, social combination of presentation. (Rethink Services (CUES) life, information and advice, three-category n.d.) Service User access to services, choice of questions and Questionnaire services, relationships with open-ended workers, consultation and qualitative control, advocacy, stigma and questions discrimination, medication Milestones of Stages of recovery Single item, Underlying concepts Recovery Scale 8 categories of level of risk, level of (MORS) (select one), engagement, and level clinician rated of skills and supports. (Pilon and Ragins 2007) Recovery Internal (cognitive, emotional, 8 dimensions, Recovery Measurement Advisory Group spiritual, physical), external choice of 6 Tool also developed – Recovery Model (activity, self-care, social stages of based on this model. Structure relations, social supports) recovery for (Ralph n.d.) each dimension Stages of Hope, identity, meaning, 50 items, 5-stage model of Recovery responsibility 6-point scales recovery. (Andresen, Instrument Caputi et al. 2006) (STORI) 50
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesGeneral healthGeneral health measurement tools tend to cover a range of dimension such as mental health andsocial relationships in addition to physical health. Table 16 provides information on the SF scale,one of the most commonly used general health scales.Table 16: General health outcomes measurement tool Name of tool Domains / areas covered Measures Notes Short Form (SF) Physical health (physical 36 items, mostly Multiple translations functioning, role-physical, 5-point scales available. Not public bodily pain, general health), domain. Shortened mental health (vitality, versions also social functioning, role- available - SF-12, SF-8. emotional, mental health) (QualityMetric 2010)In addition to generic scales, there are a wide variety of condition-specific scales oriented toparticular acute or chronic health problems such as diabetes, asthma, cardiovascular disease,depression, and cancer (Post, Isbell et al. 2005: 22). However, it is suggested that while thesescales might be of relevance to clinical services working closely with people experiencinghomelessness, they are likely to be too specific and require too much expert knowledge to be usedwithin generalist homelessness services. Health Care for the Homeless (1998) summarises theresults of a range of pilot projects of health-related outcome measures with people experiencinghomelessness.The World Health Organisation framework of pathology, impairment, functional limitation anddisability may provide a useful conceptual model for thinking about outcomes around specificdisabilities. In this model, a functional limitation (such as low intelligence) becomes a disabilityonly when it impacts on or interferes with a persons social role or functional level (Schalock 2001:146). Such impacts are mediated by a range of factors including the persons physical and socialenvironment. One way to measure progress may be in terms of the intensity of support needs,which should gradually decrease over time if services are effective (Schalock 2001: 147). 51
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesAddiction / substance misuseTable 17 lists a number of assessment and outcome measurement tools that have a focuson substance use, and/or are specifically designed for working with people with problematicsubstance use issues.Table 17: Substance misuse outcome measurement tools Name of tool Domains / areas covered Measures Notes Maudsley Substance use, health 60 items, variety of (Marsden, Gossop Addiction Profile risk behaviour, physical scales (largely frequency- et al. 1998) (MAP) and psychological health, related) and personal/social functioning Christo Inventory Social functioning, 10 items, 3 point scales Free to use. for Substance- general health, sexual / (Christo 2008) misuse Services injecting risk behaviour, (CISS) psychological, occupation, criminal involvement, drug/alcohol use, ongoing support, compliance, working relationship Treatment Substance use over 4 23 items, various scales (National Outcomes Profile weeks, legal issues, Treatment Agency (TOP) general and mental for Substance health Misuse 2007; Pleace 2008: 67, 71-74) AUDIT alcohol Alcohol consumption, 10 items, primarily 5-point (Babor, Higgins- dependence scale drinking behaviour scales Biddle et al. 2001) and dependence, consequences or problems related to drinking Severity of Severity of dependence 5 items, 4-point scales (World Health Dependence Scale on opioids Organisation 2010) (SDS) 52
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesParticipation in societyEmployment and educational outcomes are typically assessed using status maintenance andchange scales (e.g. full time employed, part time employed, casual employed, looking for work,not in the job market). Childrens educational outcomes measures might include school attendance/ absenteeism and school retention.Table 18: Measurement tools for services assisting people to obtain employment Name of tool Domains / areas covered Measures Notes Work Star Job-specific skills, aspiration 7 items, 10-point (Burns and MacKeith and motivation, job-search scales 2009b) skills, stability, basic skills, social skills for work, challenges The Motivation, lifestyle and 5 items, 9-point scales Currently only Employability social skills, basic skills, available within Map work related skills OSW network. (OSW 2004)There is a larger picture a persons reintegration into the community, of which employment,education and leisure activities might be considered part (Busch-Geertsema 2005). I have notfound standardised tools that measure degree of community participation, although Burke andHulse (2002) used a range of measures of community engagement and participation.Social wellbeingSocial wellbeing has a number of dimensions, covering for example family functioning and varioustypes of social support (esteem support, informational support, social companionship, instrumentalsupport) (Pleace 2008: 75).Available tools include generic Quality of Life tools (which tend to include domains related tonumber and quality of relationships), or standardised sections of surveys such as the BritishHousehold Panel Survey (Pleace 2008: 76). Calsyn and Winter (2002) looked at the relationshipbetween social support, mental illness and stable housing. The measures of social support thatthey used included: natural support (frequency of contact), perceived support (number of peopleavailable to help), satisfaction with support (feelings about social support), and support fromprofessionals.Legal issuesI did not locate any standardised tools in the area of assessing outcomes with respect to legalissues (whether offending behaviour or other engagement with the legal system). One approachis to ask people to indicate whether they have been subject to particular legal orders or eventswithin the previous given time frame (e.g. Pleace 2008: 77). 53
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesMaterial wellbeingMaterial wellbeing outcomes tend to be measured using concrete and status measures (change ingross income, change in benefit status). However, there are a range of other dimensions that arerelevant including the extent to which people can afford basic necessities, and peoples perceptionof their financial situation, in general or in relation to that of others (Pleace 2008: 79). The QoLHHImodule on income provides a standardised set of items that provide good coverage of thisterritory.Activities of daily livingActivities of daily living can include both self-care (personal hygiene, dressing and undressing,eating, continence, elimination) and instrumental activities of daily living (light housework,preparing meals and clean-up, taking medications, shopping, communication, literacy andnumeracy, use of technology – e.g. telephone, ATM).The literature review did not have time to investigate scales for measuring level of functioning inthese areas. Table 19 provides information on one tool that has been reviewed in the literature onmental health outcomes measurement.Table 19: Activities of daily living assessment tool Name of tool Domains / areas covered Measures Notes Resource Utilisation Groups Bed mobility, toileting, 4 items, (Department of – Activities of Daily Living transfer, eating primarily Health and Ageing Scale (RUG-ADL) 4-point scales 2003: 40-41)For further research: other assessment or outcome tools relating to activities of daily livingSafetySafety is a significant concern of many homeless people, however the literature review did notidentify any outcomes tools that focus solely on assessing a persons level of safety. However,vulnerability indices often include one or more items related to safety (Common Ground n.d.;Downtown Emergency Service Centre n.d.). Measurement of safety could focus on peoplesperceptions of their safety, and/or on a history of violent or abusive incidents.For further research: personal safety assessment tools; consider literature relating to familyviolence, child abuse and neglect 54
    • Literature Review:Measurement of Client Outcomes in Homelessness Services7. Measurement processes7.1 SaMPling organisation. For human service organisations, with some clients remaining in the sample fromOne decision point for organisations the sampling frame (list of clients of the one reporting period to the next, and othersimplementing client outcome measurement organisation) may closely approximate the being removed or added.systems is what approach to take to sampling. population of interest (all clients of theIn general, there are two approaches: to try organisation). The question of whether Due to the need to collect baseline data, it isto measure outcomes for all clients, or to any generalisation is possible to broader necessary to select clients for inclusion in theselect a subsample. Sampling all clients has populations outside of the organisation (for sample at the point of assessment or intakeadvantages: example, to homeless men in the greater to the service. This could be achieved in• It is likely to provide greater precision and Melbourne area) is complex, and relates several ways: validity in the results (Hatry 1997: 18) to the level of representativeness of the • Using a quota system where each service• It can be logistically simpler in terms of organisational client population with respect is always monitoring outcomes for a certain workflow and management practice to to the broader population. The fact that the minimum number or proportion of their survey every client routinely rather than organisations client population is not a random client population – as the number drops (for example) one in five (Booth and Smith sample of the broader population immediately below the quota, more clients are selected 1997: 38) reduces the likelihood of representativeness. into the sample• If outcomes measurement is integrated • Using a variation on a systematic random with casework (e.g. in assessment, case One issue that may arise for organisations sampling approach (Trochim 2006b) where review and action planning) then it is seen with a diverse service portfolio is whether to (for example) every 5th client accepted into as a standard good practice element of sample on a whole-of-organisation basis, or a service is selected service delivery with every client. a service-by-service basis. If services have • Using a time-limited snapshot (Burns and widely varying numbers of clients (for example, Cupitt 2003: 19) where all clients acceptedLarger samples do tend to support more a case management service with a group of into the service within a particular timesophisticated statistical analysis, especially 50 long-term clients as opposed to a drop-in period are selectedwhere a population is divided relatively equally service seeing thousands of clients per year),on points of interest or where differences are simple random sampling from a whole-of- Each of these suggestions affects themore subtle. organisation client list may lead to few clients randomness of the sample to some extent, and being sampled from the smaller services. One may therefore lead to some bias of the sampleHowever, surveying every client also requires possible response is to use stratified random with respect to outcome variables.greater resources and effort. Particularly for sampling, with disproportionate samples drawnservices with high throughput and/or demand, from small services, and to adjust for this There may be times when a nonprobabilityit may be impossible within the given resources over-representation in whole-of-organisation sampling approach might be appropriateto attempt outcomes measurement with every analysis (Trochim 2006b). If there are other (Burns and Cupitt 2003: 19). For example,client. Even where resources do potentially sub-populations of particular interest to purposive sampling with a heterogeneityallow for data collection with the entire client the organisation (for example, survivors of focus (Trochim 2006a) might be used wherepopulation, this may be seen as wasteful domestic violence or recent migrants) it would the organisation is interested in information(Booth and Smith 1997: 38). also be worth considering these groups within about the full range of client experiences the stratified random sampling mechanism. and outcomes (including outlier outcomes),If a sample is used, it is preferable that a as opposed to an understanding of averageprobability sample be used if possible (Booth Sampling techniques are also complicated by or typical outcomes. Regardless of whetherand Smith 1997: 38). Probability samples the fact that organisational client populations random sampling is used, it is importantprovide greater ability to generalise to an are dynamic rather than static, and that in to sample with clear defensible criteria.overall population (Trochim 2006b) – for general it will be desired to monitor clients Convenience sampling should be avoidedexample, to generalise from a sample to the until they exit the service. This means that the where possible.entire group of current clients of a service or sample will progressively change over time, 55
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesIf a sub-sample is used, a decision is also However, it would be expected that clients progress for individual clients over time.necessary on the desired proportion of the would be informed about the use, storage of Assignment of a unique identifier such as atotal population to sample. Berman and Hurt and access to outcome measurement data statistical linkage key may assist in providing(Berman and Hurt 1997: 87) suggest that, if (Baulderstone and Talbot 2004: 41). some level of de-identification during analysisfaced with a choice between a high response processes.rate in a single setting (e.g. a single service) Some clients may wish to receive a service,and a low response rate across an entire but to opt out of outcomes measurement 7.3 tiMing and frequencyorganisation, it is preferable to concentrate processes. There is no clear-cut answeron the single setting. A 20% sample across on how services should respond to this. At a minimum, outcomes data should bean entire organisation is likely to "provide Participation in client self-report measures is collected twice for each client – once at intakedistorted or incomplete information" that is not clearly voluntary, however some outcomes (or at the onset of a particular service), andreliable as an aid to decision making. data may be generated by practitioners. As once at exit or completion of the particular noted above, a non-response rate is to be service (Rapp and Poertner 1992: 107), toRegardless of the sampling method chosen, expected for a variety of reasons; services may give an estimate of total distance travelled.some level of non-response is also to be see opt out as one reason. Where outcomes Outcomes data can also be collected atexpected (de Vaus 2002: 152). Baulderstone measurement is integrated within casework multiple points during the period of serviceand Talbot (2004: 42) note that in the processes, but clients are offered the choice provision. Whether this is appropriate dependshomelessness service context it will never be to opt out, staff will need clear guidance on on the nature of the service and time framepossible to obtain a 100% completion rate for what alternative processes are to be used for of engagement. Crisis services are unlikely tooutcomes measurement. Clients may decline assessment and review. have time to collect more than the pre- andto participate, may exit the service at short post-service measures. Long-term servicesnotice, or their circumstances may mean that Organisations may wish to communicate are oriented to goals that are cumulative andit is "difficult or even inappropriate for workers outcomes information externally, for example: are likely to want regular updates on progressto engage them in formal measurement • Aggregated de-identified reports to funders (Berman and Hurt 1997: 85-86).processes." • Use of aggregated outcomes results in advocacy materials, tenders, annual Burns and Cupitt (2003: 26-29) suggest that7.2 conSent reports etc. regular monitoring is useful as it helps to • Exchange of individual client outcome combat attrition of the sample. If a client exitsConsideration is needed to the nature of information with service delivery partners the service in an unplanned or rapid mannerconsent that would be required to collect and involved with that client. and the service is unable to collect exit data,use outcomes data. If outcomes measurement the most recent set of regular monitoring datais seen as an integral part of service delivery, These forms of use may require additional can be used instead. Burns and Cupitt suggestthen a general consent to collection of personal specific consent from clients. Where clients a frequency of between one and three monthsinformation would cover outcomes data in provide general consent for exchange of for regular data collection, depending on theaddition to all other client-related information information between two services or agencies, service type.held by the organisation. It is unlikely that this may include exchange of outcomesadditional, specific consent is required for an information, however again best practice Follow-uporganisation to collect outcomes data. In the suggests that this should be made explicit as Many sources recommend collecting follow-upcontext of Australian privacy legislation, use part of the informed consent process. data if possible, at some point after the clientof client outcomes data for internal service has exited the service (e.g. Rapp and Poertnermanagement and quality improvement It should be noted that while information 1992: 107; Hatry 1997: 18). There are two keypurposes would be seen as a directly related can be de-identified in aggregate reports, reasons for this:secondary purpose of the data collection and the stored outcomes data itself must remain • There is evidence that positive gains madewould also not require additional consent. identifiable in order to be able to track by clients while engaged with humans services programs may decay over time (Wells and Johnson 2001: 180) 56
    • Literature Review:Measurement of Client Outcomes in Homelessness Services• Some long-term outcomes may only be • The optimal follow-up interval is not measurement involves a conversation between visible or measurable at some point after necessarily obvious. Longer intervals may client and worker, this conversation may lead the completion of service delivery. demonstrate more significant changes to a more balanced picture of an individuals (whether positive or negative), however situation or progress – challenging overlyHowever, there are considerable logistical longer intervals may also increase negative or unrealistically positive views ofdifficulties involved in tracking clients. the chance of attrition and make the outcomes. These conversations can provideThe level of transience of many people connection between intervention and workers with a valuable assessment of aexperiencing homelessness means that it can outcome more tenuous. Hatry (1997: 18) clients current situation, can feed directlybe very difficult to locate them one they have suggests 3, 6, 12 or 24 months as typical into case planning and review, and can alsodisengaged from the service (Post, Isbell et follow-up intervals. be a transformative element of casework inal. 2005: 12). Researchers recommend that themselves (MacKeith and Graham 2007: 12).considerable effort be put into recording as 7.4 gatHering and recording data Finally, having service delivery staff collectmuch contact information about the client outcomes data is an inexpensive option as itas possible, and seeking consent to contact The two key questions in relation to collection does not require employment of additionalothers who may be able to assist in locating of outcomes data are: staff (Pleace 2008: 79).the client (Pleace 2008: 62). • Who will collect the data? • How will they collect the data? Despite these advantages, there is a downsideCollecting follow-up information can be a time to having service delivery staff collectconsuming and costly exercise (Hatry 1997: Who will collect and record the data? outcomes data. One problem is the impact of18). It is questionable whether service delivery Decisions about who should collect data are data collection on the service delivery role;staff would have time to undertake this closely related to what measurement tools are every additional data collection requirementfunction; if not, specialist staffing resources used – client self-administered questionnaires impacts on time available for other aspectswould be required for data collection. Another will obviously be completed by the client, while of service delivery (Rapp and Poertner 1992:option is remote monitoring – receiving some level of functioning scales are designed to 107). Another disadvantage is the potentialinformation from other parties involved with be completed only by the practitioner. However, for data quality to be lower than that collectedthe former client about their progress or many tools are designed to be completed in by specialist personnel. Where measurementabout the occurrence of particular positive or an interview-type situation, with discussion is undertaken by a large number of staffnegative outcomes. For example, a service between the client and another party. with varying levels of skill, this can reduceprovider could be notified by a public housing the reliability of the data; regular thoroughprovider if a former client was allocated In deciding who will be involved in such training is likely to be required to ensure aa property, or abandoned their tenancy. conversations, human services organisations minimum level of consistency in use of theHowever, this approach creates substantial have two basic options. The first is to use measurement tools. There is also the potentialethical and legal issues in relation to privacy service delivery staff such as case managers for validity to be affected by collector bias.and data protection (Pleace 2008: 79). or housing workers; the second is to use Where the measurements recorded can result specialist data collection personnel. in consequences for the individual or serviceThere are also some technical issues in relation collecting the data (for example, more orto the collecting and analysing follow-up data: There are several advantages to having service less favourable treatment of particular staff• The possibility of selective attrition – this delivery staff collect outcomes data. These members or clients; continuation or withdrawal could be in a number of directions. Clients staff have an existing relationships with clients. of funding for a service), there is a risk with worse outcomes are more likely to Within the context of such a relationship, that this may lead to falsified or fraudulent be transient and harder to find. However, clients may be more willing to participate in reporting (Rossi 1997: 31-32; Pleace 2008: 60). clients with good outcomes are less likely to outcomes measurement, and may also be more Even where no overt consequences exist, staff re-contact the service of their own accord willing to discuss more personal or sensitive members may be influenced by the context than clients who are still in need (Post, aspects of their lives. Where outcomes and goals of their relationship with particular Isbell et al. 2005: 12; Pleace 2008: 62) 57
    • Literature Review:Measurement of Client Outcomes in Homelessness Servicesclients, and may see improvement where none collect practitioner-rated outcomes data than Outcomes conversations may occur in personexists, or place high value on relatively minor client self-report data (Love and Scott 2010). or over the phone. Cultural considerationsshifts (Weiss 1972: 41). Completion of practitioner-rated tools is are relevant to how such conversations are typically quicker, more flexible in timing and conducted. For example, with indigenousSome authors suggest the use of audits as more under the control of service-delivery clients it may not be appropriate to sit downa means of monitoring collector bias and staff than completion of client self-report and work through a paper-based or electronicdiscouraging fraudulent reporting (Rossi 1997: tools. However, in the homelessness service survey; data collection may involve an informal32; Pleace 2008: 60). However, auditing and environment, client perceptions of progress conversation which is later recorded on paperinspection require additional resourcing and tend to be highly valued. by the staff member (Hamilton 2010).may only detect bias in a haphazard fashion. Client self-administered questionnaires tend In general efficiency of recording is a keyAnother approach is to use specialist to suffer from low response rates. Many client consideration, and data entry directly into apersonnel for collection of outcomes data. self-report measures can best be completed database (rather than onto paper) can saveFor example, an organisation might employ through an interview or conversation between duplication of effort (Barr n.d.: 19). Someresearch staff whose role includes collection of client and worker. This can address literacy software packages allow for automated dataoutcomes data. This may potentially increase barriers, clarify the meaning of particular entry; these typically rely on specially designedreliability and validity, but it does require items and enhance the casework process. paper-based forms which are completed bysubstantial extra resourcing, especially for However, this requires allocation of sufficient hand and then optically scanned by a scannerlarger organisations who may wish to collect time for the conversation (which may be a or fax (Berman and Hurt 1997: 88-89). If theoutcomes information for thousands of clients. challenge for staff with high caseloads), and forms are customisable, these systems areData collection tools that are more complex requires an additional level of competence potentially quite flexible and efficient, howeveror that require more technical knowledge from the practitioner to be able to facilitate the cost and/or development effort may beto administer may also require specialist conversations about outcomes. beyond the reach of many organisations.personnel for effective use. These challenges point to the need for services 7.5 it SySteMSAnother approach might be to make use of to provide adequate training for staff collectingtrained volunteers or client representatives, data, and to think carefully about how data IT systems are vital to the efficienteither for data collection or data entry (Rapp collection will fit into other service delivery management, analysis and reporting ofand Poertner 1992: 107). Involvement of client processes. Baulderstone and Talbot (2004: outcomes information. It is not uncommon forrepresentatives would require consideration of vii) concluded that outcome measurement organisations to begin implementing outcomeprivacy issues and awareness of pre-existing "is only feasible and practical where it is measurement processes, and then to findrelationships among the services client group. properly integrated with the case management that their client data systems do not have process.” Outcomes conversations have a the capacity to store the information beingAt a broader level, there are service system natural fit with intake, exit and case review generated (Parkinson 2005: 4; Hendrick 2010b).issues around collection of outcomes data processes and should ideally occur at these Parkinson (2005) provides an overview ofwhere clients are involved with multiple points rather than as a separate add-on. options for organisations in managing outcomesservices or agencies. To avoid over-surveying, Similarly, Burns and Cupitt (2003: 19) suggest data, including modifying existing data systems,it may be appropriate to negotiate (or have that outcomes monitoring should be integrated developing new systems or using off-the-shelfprotocols in relation to) which service(s) will into the organisations other monitoring products. Each option has advantages andmonitor outcomes in such situations. systems. Outcomes questions are more likely disadvantages. to be completed regularly and well whereHow will data be collected and recorded? they are part of the overall service monitoring The nature of the outcomes and other data toAnecdotal evidence suggests that it is likely system rather than an additional parallel be collected will have a major impact on theto be easier to regularly and systematically system. design of data system. Ideally, clarity should be 58
    • Literature Review:Measurement of Client Outcomes in Homelessness Servicesachieved on reporting requirements, measuresto be used and data collection processes, priorto commencement of development (Parkinson2005: 15-16).Gaining an understanding of stakeholderrequirements is a crucial early step. Hudson(1997: 74-77) argues that many data systemsare designed with insufficient attention to theday-to-day realities of practice and the needsof clients and service delivery staff. In theworst case this may result in a "DINO" (data in,nothing out) system that provides little usefulinformation for staff. Staff buy-in is crucial todata quality. Availability of real-time reportingon outcomes or other data for individual clientsor groups of clients greatly enhances the valueof the system to staff and clients (Bermanand Hurt 1997: 87). Early consultation withstakeholders is likely to increase buy-in and leadto a set of specifications that more fully meetsthe organisations needs (Parkinson 2005:18-19).Berman and Hurt (1997: 87) articulate severalother key criteria for client data systems,including data entry mechanisms that minimiseimpact on staff time and maximise accuracy andthroughput; management of data capture withchecks and reminders; and storage in a waythat allows aggregate analysis and reporting.Parkinson (2005: 30-31) provides basicspreadsheet designs for capturing hard and softoutcomes data. There are a number of existingsoftware packages designed for the humanservices which are oriented to capture of clientoutcomes data. For example:• "Efforts to Outcomes" from U.S. company Social Solutions integrates case management and outcomes recording functionality (Social Solutions 2010)• The Outcomes Star System is the online version of the Outcomes Star and includes a free web-based database for capturing and reporting on Outcomes Star data (Homeless Link 2010). 59
    • Literature Review:Measurement of Client Outcomes in Homelessness Services8. Use of outcomes data8.1 rePorting Figure 2: Function-based and geography-based aggregation chains for client outcomes data.Client outcomes may be aggregated and Arrows point in direction of increasing aggregationmonitored at a variety of levels includingthe individual client or household, servicedelivery team, service, program (which may be Service system Countrydelivered across a number of organisations),organisation, and geographic region (NMHWGInformation Strategy Committee PerformanceIndicator Drafting Group 2005: 19). Separate Sectoraggregation chains can be identified basedon geographic areas and on the functional Stateorganisation of the service system; these arediagrammed in Figure 2. Population outcomes Programare generally referenced to geographicaggregation at the levels of Region or above. OrganisationService provider organisations will generally DHS Regionbe unable to collect population data (althoughthey may be able to monitor census data DHS Regionaggregated by other organisations), and willbe unable to aggregate beyond the level ofservice or organisation on the function-based Local Governmentaggregation chain. Service Area Service delivery Neighbourhood team Function-based Individual client / Geography-based aggregation household aggregation A variety of stakeholders will also be interested in the findings from outcomes measurement, including clients, service delivery staff, organisational management, Boards, service delivery partners, funders, and the broader public (Office of Housing n.d.: 6). Systems of feedback and/ or reporting may need to be tailored to the different needs of these stakeholder groups; different report formats and levels of detail will be appropriate for different groups (Wells and Johnson 2001: 193). Many sources suggest that accessibility of outcomes results to staff is essential to maintain commitment to data collection and to improve practice (e.g. Hudson 1997; Clements 2010; Elizabeth 2010). 60
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesAnalysis and reporting of outcomes data Use of standardised formats for reports will make both the production and use of reports moretypically relies on specialist staff (Schalock efficient. Two typical standardised formats are report cards (e.g. for inclusion in an annual report),2001: 35), and can be a resource-intensive and longitudinal comparisons (Schalock 2001: 35).activity in itself. Trade-offs may be requiredbetween the number of different ways in which Hatry (1997: 14-15) provides an example of a tabular report format comparing actual vs. targetdata is de-aggregated, the number of different outcomes on a range of measures for a single service or organisation; this is adapted as Table 20.reports produced, and the level of detail ofeach report. Table 20: Sample tabular format for comparison of actual and target outcomes (adapted from Hatry (1997: 15)). + indicates actual performance better than target, - indicates actual performance worse Many sources support the importance of than target regular outcomes reporting, with timing ofreports linked to the organisations planning Outcome indicator Last Thisand performance management timetable period periodso that up-to-date outcomes information isavailable to support decision-making (Schalock Target Actual Difference Target Actual Difference2001: 35; Wells and Johnson 2001: 193; Burnsand Cupitt 2003: 34). MacKeith and Graham Percentage of children returned to 35 25 -10 35 30 -5(2007: 16) suggest that after a number of home within 12 monthsperiodic reports have been produced, it maybe possible to establish benchmarks for Percentage of children with more 20 20 0 15 12 +3performance. They also recommend that data than two placements within thebe analysed more comprehensively on a less past 12 monthsregular basis (e.g. every 1 – 3 years) to providedeeper insight into how outcomes related to Percentage of clients reporting 80 85 +5 80 75 -5interventions, and how outcomes compare satisfaction with their livingacross client groups and service types. arrangementsReport formatsOutcomes reports should be designed for The Outcomes Star chart provides an example of a graphical report format for individual clientmaximum clarity of communication. Guidelines outcomes data, enabling comparison of results for one client at different time periods (MacKeith,for report design include (Rapp and Poertner Burns et al. 2008a: 13).1992: 103; Burns and Cupitt 2003: 33; Barrn.d.: 27-31): Client access to outcomes information• Keep it simple – highlight the key Outcomes information can be shared with a client in two ways: discussion of their individual information (particularly key trends) and outcomes, and discussion or provision of de-identified aggregated outcomes data (for a service or avoid irrelevant detail the organisation). Both approaches may have benefits in motivating clients to continue providing• Use well-designed graphs to enable outcomes information (Burns and Cupitt 2003: 27). comparison of results over time and across groups Under Victorian privacy legislation (Information Privacy Act 2000 - Information Privacy Principle• Place outcomes in context of the client 6), clients have a right to access and correct personal information that the organisation holds group and service environment (especially about them. This would be expected to include their outcomes data, and it is unlikely that any for reports intended for external of the exceptions to the access right under IPP 6 would apply to outcomes data. However, in audiences) some circumstances service delivery staff may be reluctant to share outcomes data with clients,• Explain the meaning of numerical scores particularly in the case of practitioner-rated measures. Staff may perceive that sharing the or particular categories when reporting results on particular scales• Comment on the reliability and validity of the data and note any factors which might affect this. 61
    • Literature Review:Measurement of Client Outcomes in Homelessness Servicesratings would constitute negative feedback to achieve a particular outcome in a given after the period of engagement, ratherto the client that may impact on the clients period, by the total number of clients of the than the details of the many ups andself-esteem or motivation, and may jeopardise service who were in the "target population" downs that may occur during the period ofthe therapeutic relationship. Sharing outcomes for that outcome during that period (Friedman, engagement. It may be useful to undertakeratings, particularly where they differ from DeLapp et al. 2001b: 3.7-3.8; Spellman and both analyses.a clients perceptions of themself, may also Abbenante 2008: 24-26). The target populationinvolve a lengthy conversation requiring may be the entire client group of the service, There is some debate in the literatureconsiderable skill on the part of the practitioner or a sub-group. For example, if the desired about the reliability of simple change scores(including for example, motivational outcome was that an individual maintain stable (assessment rating at time 2 minus assessmentinterviewing skills). Some staff may feel a housing for at least 6 months, clients who had rating at time 1) as measures of change. Somepressure to rate more positively than they been engaged with the service for less than sources recommend the use of more complexotherwise would, if they know that ratings will six months would not be included in the target methods including analysis of variance andbe shared with clients (Love and Scott 2010). population for calculation of the percentage of covariance and regression techniques, however clients who achieved the outcome. Stedman, Yellowlees et al. note a growing bodyGiven the right of clients to access their of literature suggesting that simple changeoutcomes information, it appears important Even with simple percentage-based analysis, a scores are valid indicators of change (1997:that organisations provide clear guidance and number of decisions may need to be made: 19-20).training for staff about how outcomes ratings • Are point-in-time or improvement-over-are to be generated and shared with clients. time findings more useful? Point-in-time Where a series of data points are available measures reflect the proportion of the over time (for example, a series of outcome8.2 analySiS of aggregate data client population who meet or exceed ratings for an individual client, or a series of some target in the period – for example, percentage values for a client group), moreMethods for analysis of outcomes data can the proportion of clients in permanent sophisticated analyses become possiblerange from basic to sophisticated, depending housing in a particular 3-month period. including calculation of mean, median, rangeon the questions that the organisation wishes Improvement-in-time measures reflect the and standard deviation, and identificationto answer, and the resources available. proportion of clients whose outcomes data of trends over time (Bloom, Fischer et al. for a given period showed improvement 2006: 540, 555-559). Barr (n.d.: 26) notesAn initial consideration for many organisations compared to some previous point in time that calculation of mean values becomesis reconciliation of data from multiple data (for example, compared to their outcomes more reliable with a greater number of datacollection systems, including multiple client data for the previous period, or their points, and recommends a minimum of 20databases. Some form of unique identifier baseline data) (Friedman, DeLapp et al. data points if possible. However, even with asuch as a statistical linkage key is essential to 2001b: 3.7) smaller number of data points, where statisticslinking together data on a single client held in • Is the target population drawn from all are triangulated with other forms of evidencemultiple repositories. Often a client may have clients of the service during the period, (including the knowledge of practitioners) theycontact with more than one service within an or only those exiting? Basing it on the may provide useful infomation (Friedman,organisation, and conventions are needed for total client population (including clients DeLapp et al. 2001b: 2.11).the process by which data is combined and who continue with the service as well asdiscrepancies are resolved. those who have exited) provides a much Identification of desired, typical and broader picture of the types of changes undesired ranges for particular measuresOnce individual client data has been cleaned, that are occurring for clients of the service. may be useful, and there are a variety ofaggregate analysis can take place. For many However, analysing outcomes for clients techniques for generating these, includingpurposes, simple percentage-based analyses who have exited the service helps to statistical process control methods such as theare sufficient. Percentages are typically focus on the "net effect" of the service, X-moving-Range Chart (Bloom, Fischer et al.calculated by dividing the number of clients comparing the clients situation before and 2006: 582-583). 62
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesTests of statistical significance may be of One issue that may arise is the extent to qualitative information is available from staffrelevance to outcomes analysis in several which diverse measures can be combined into or clients, this may provide a much richerways: a composite index giving an overall rating of understanding of the process of change and• In comparing a series of outcome results change or current situation. Indexes might be of barriers and enablers to positive outcomes. over time, tests of statistical significance used in a variety of situations: Qualitative information will need to be themed can help to establish whether a particular • Some measurement tools calculate an which can be a complex process in itself (Burns data value is likely to represent a genuine overall score (e.g. an overall risk, severity and Cupitt 2003: 32). change in the mean of the results, or may or complexity rating) based on combining be due to chance (Bloom, Fischer et al. values from a variety of measures Recognising client and service differences 2006: 582) • Organisations might wish to combine a In addition to the type and quality of the• When aggregate outcomes results are number of measures within a single domain service intervention itself, a wide variety of based on a probability sample from the into an overall index for that domain – for other factors influence client outcomes. These total client group, tests of statistical example, to combine measures relating to outcome predictors include factors related to significance can help to establish whether housing tenure, affordability, stability and the client (e.g. socio-demographic variables, the aggregate results can be generalised quality into an overall proportion of clients co-morbidities and personal biography) and to the broader client population, or may be whose housing situation improved factors relating to the environment (e.g. level affected by sampling error (de Vaus 2002: • It might also be possible to combine of resourcing of the service itself, accessibility 169-171). measures across different domains to of other resources and supports in the local give a sense of overall improvement or community) (Berman and Hurt 1997: 86).It is important to balance consideration of deterioration in the situation of a personstatistical significance with clinical or practical or group, or overall level of success of a Inclusion of these variables in outcomessignificance. Clinical significance refers to the service in achieving positive change. analysis is important for several reasons:extent to which the parties involved in the • When comparing outcomes results fromservice delivery situation (including client, The advantage of an index is its communicative services of the same type, it is importantservice delivery staff, carers) believe that power – its ability to summarise complex to be able to adjust for differences inmeaningful change has occurred (Bloom, information in a form that can be quickly the client groups served by the services.Fischer et al. 2006: 516). Clinical significance communicated and understood. However, In particular, it is useful to adjust foris usually referenced to the expectations and indexes can also be problematic. Their validity the severity and complexity of clientsgoals of clients, and/or to comparison of the depends on a range of design issues including presenting issues. This enables fairclients functioning against some standard the choice of measures to include, avoiding comparisons across services and also(Stedman, Yellowlees et al. 1997: 19-20). repetition of the same factor or dimension, avoids creating a disincentive for servicesOutcomes results may show statistically decisions on the relative importance of who work with clients who are less likelysignificant change but not clinically significant different measures, and statistical procedures to achieve desired outcomes (Hatry 1997:change, or vice versa. to represent the relative values of measures 14-16; Spellman and Abbenante 2008: that use different scales. Indexes may also 41). Adjustment of outcomes results withWhere samples are used, it will usually be over-simplify complex situations and mask the respect to client need, complexity and/orvaluable to test for and adjust for sample upward and downward movement of separate severity of issues is usually referred to asbias (de Vaus 2002: 152-165). In particular, indicators (Weiss 1972: 37; Ware 1997: 63-64). casemix adjustment or risk adjustmentin the homelessness setting it will be useful (Booth and Smith 1997: 41-42; Spellmanto look at non-response groups, including It is useful to consider what other information and Abbenante 2008: 41).those who declined to participate in outcomes may be combined with quantitative analysis of • Comparing outcomes across differentmeasurement and those where it was not client outcome measures. Process measures sub-groups of the client population of afeasible to collect the data for other reasons. may illuminate some of the service delivery service or organisation can help to identifyFurther investigation might show particular factors that impact on client outcomes. Where differences in the effectiveness of servicespatterns of outcomes for these groups. 63
    • Literature Review:Measurement of Client Outcomes in Homelessness Services for particular groups (Johnson 2010). Some itself may also provide the required data for client groups may show unusually positive casemix categorisation. or negative results on a particular outcome indicator; this can provide a prompt for Hatry and Lampkin (2003: 18) suggest that further investigation and for development it will typically be useful for human service of new service approaches (Berman organisations to analyse outcomes data with and Hurt 1997: 86; Hatry and Lampkin respect to client factors including: 2003: 25). • gender • age groupWhile casemix adjustment is important, • race/ethnicitycasemix methodologies are still in development • income group(Booth and Smith 1997: 42). Casemix • type of disabilityadjustment depends on a complex set of • educational leveldecisions regarding which characteristics are • housing status.chosen as the basis for adjustment, and howthey are weighted (Spellman and Abbenante Personal biography, specifically a persons2008: 43). As these decisions affect how the pathway into homelessness, has alsoperformance of individual services is viewed been shown to have a profound impact onrelative to others, they can become a point of experiences while homeless and the type ofconflict (Rossi 1997: 26). outcomes that may be expected (Johnson, Gronda et al. 2008).Buckingham, Burgess et al. (1998a) undertooka major study that developed the first version It is also relevant to consider service factorsof a casemix classification system for specialist such as:mental health services in Australia. The study • the specific team, office or facilityfocused on the relationship between client • the specific types of service provided (didneed and cost of service provision. While the client receive support plus transitionalthe study demonstrated that it was possible housing, or support only?)to identify a set of empirically validated • the amount of service provided (e.g.casemix categories, both the process and number of hours or visits).the resulting set of categories were complex,and it was not always easy to separate Hatry and Lampkin (2003: 18) also suggestclient from practitioner variation (p.264). that reports for use by Coordinators orNo single measure was found suitable to Managers within individual teams could bedetermine need, and the classification used de-aggregated by the specific caseworker,a combination of instruments and measures however this may have potential disbenefitsthat assessed demographic, clinical and level including increasing the temptation for staff toof functioning characteristics (p.271). The falsify data.study concluded that accurate measurementof client need would require the introductionof new measurement tools. However, one ofthe advantages of casemix adjustment foroutcomes is that (depending on the outcomemeasures chosen) the baseline outcomes data 64
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesHatry and Lampkin (2003: 25) provide a sample tabular format for de-aggregating data by client Triangulating with other data sourcescharacteristics; this is adapted as Table 21. Hatry (1997: 16) provides a sample tabular format forcomparing outcomes by service unit and difficulty of problems at intake; see Table 22. It will often be useful to combine quantitative outcomes data with other informationTable 21: Sample format for summarising results on a specific outcome indicator by demographic or including data about processes, resources andservice characteristics (adapted from Hatry and Lampkin 2003: 25) the broader context. Triangulation helps to enhance the credibility of findings, and to build Respondent Clients Reporting Change in Their Problem Condition from the Time They a much deeper understanding of underlying characteristics Entered Service to Time of Measurement processes and causal factors. Sources of Total additional data may range from anecdote and Extent of Improvement (%) opinion of staff and clients, to outputs from responding formal program evaluations. (n = 625) None A Little Somewhat Considerable (n = 50) (n = 83) (n = 429) (n = 63) 8.3 uSing tHe findingS Sex and race White male 265 7 11 71 10 One of the least explored topics in the literature is how outcomes information can White female 284 8 13 68 11 best be used by organisations. Non-white male 36 11 30 53 6 Non-white female 40 10 17 65 8 Service development and planning An important use of outcomes information Age is in service development. Both positive and 18-34 272 13 16 58 13 negative results can provide valuable prompts 35-49 125 6 11 75 8 for the evolution of particular interventions (Hatry and Lampkin 2003: 27). Some 50-64 105 3 11 80 5 organisations routinely use an outcomes- 65 and over 123 6 11 74 9 focused approach to planning, including Total 625 8 13 69 10 reflecting during strategic or business planning sessions on client outcomes data (Burns and Cupitt 2003: 33; Talbot 2010). However, theTable 22: Sample format for comparing outcomes by organisational unit and difficulty of pre-service loop of feedback from outcomes informationproblems (adapted from Hatry 1997: 16) to organisational and service change can also be part of a broader, ongoing process of Difficulty of Percentage of clients who maintained stable housing for at least 6 months after continuous quality improvement (Burns and problems at housing allocation Cupitt 2003: 34) as management and staff intake identify opportunities for improvement. Service Delivery Service Delivery Service Delivery Total Unit 1 Unit 2 Unit 3 Hatry and Lampkin (2003: 28) suggest holding Minor 52 35 56 47 “How Are We Doing?” sessions with managers and staff using outcome reports as a basis for Moderate 35 30 54 39 discussion. The group can explore why certain Major 58 69 61 63 functions have been going well and how Total 48 44 57 50 successful strategies can be extended to other areas. The group can also attempt to identify reasons for poorer outcomes and suggest ways 65
    • Literature Review:Measurement of Client Outcomes in Homelessness Services Dealing with unexpected or negative findings such situations. However, Burns and Cupittto improve the situation. Team meetings couldbe one forum for such discussions. Aggregated data sets are likely to show (2003: 32) note that negative outcomes do not unexpectedly high or low outcomes for some necessarily reflect the level of effort and skillSchalock (2001: 35-37) notes that many factors indicators, either across an organisation or for of staff, but may be connected to a varietyimpact on the extent to which organisations a particular service or client group. Some of of organisational and external factors. Theactually implement improvements in response the most useful learning can stem from these following questions may help to identify someto findings of evaluation or outcomes outlying values. Hatry and Lampkin (2003: of these factors:monitoring. These include the organisation’s 23) advise that organisations should establish • Did the level of resources, for example, the“personality”, operating environment and a routine process for examining the findings level of money or staffing devoted to theother priorities. Success factors for internal from outcomes reports and identifying unusual project, affect the outcome?utilisation of outcomes findings include level values. Further investigation will often be • Was the outcome affected by the wayof stakeholder engagement, credibility of the required to understand whether these values you delivered your service (for example,results, authority and leadership, and level of represent inconsistencies in the measurement by service policy, procedures or particularresourcing for implementing changes. process, or in fact represent unusual outcome interventions selected)? patterns experienced by clients. If it does • Did external factors, such as lack ofOutcomes data may also be used for advocacy appear that the data represents unusually housing stock, prevent you achieving yourand tendering purposes. Burns and Cupitt positive or negative outcomes, organisations outcome?(2003: 35) suggest that apart from making will generally be keen to understand the • Were the aims as originally stated realisticfunding applications more effective, sharing factors that may have contributed to these or appropriate, given the nature of theoutcomes results with funders can improve outcomes. client group or the services delivered?their understanding of the nature of the clientgroup and the process of change, and help to Hatry and Lampkin (2003: 23) suggest the It may also be useful to consider the distinctionbuild realistic expectations about what can be following strategies for investigating unusual between program failure and theory failureachieved with the available resources. outcomes results: (Weiss 1972: 38). Program failure occurs • Discussing the results with staff and asking when the program activities did not achieveUsing measures in individual casework for their interpretation of why outcomes the proximate goals or conditions that areRelatively little has been written on the way in outperformed or under performed understood to lead to the desired outcomeswhich outcomes data can be used to support expectations (for example, program activities did not lead toindividual casework. Literature in the area of • Forming a staff working group to examine clients being linked with health services, andmental health suggests that it may be useful the reasons for the observed outcome therefore did not improve clients’ health statusto compare current with previous episodes levels in the longer term). Theory failure occursof service delivery to identify patterns of • Consulting with clients about the outcomes when the program did achieve the expectedchange: improvement (greater positive change results, for example via a focus group proximate goals or conditions, but these didfrom admission to discharge than in previous • Conducting a more formal research or not lead to the desired final outcomes (e.g. theepisodes), stability (same level of change from evaluation process in conjunction with an health service linkages established did not inadmission to discharge), deterioration (smaller appropriate academic partner organisation fact lead to any improvement in client health).level of improvement per admission), and to or consultant. Distinguishing these two explanations forunderstand patterns of relapse (Department negative outcomes requires that interventionsof Human Services 2009: 60-61). This model It is possible that outcomes results may be are in fact based on a clear theory of change,could be extended to understand patterns of perceived by some stakeholders to reflect and that data is available to establish whetherchange for clients with high levels of transience poorly on the effectiveness of the service interim conditions were met or not.who frequently present within the homeless or organisation. This may potentially impactservice system. negatively on staff motivation. The literature offers little advice on processes for handling 66
    • Literature Review:Measurement of Client Outcomes in Homelessness Services9. Process of introducing outcome measurementIntroduction of outcomes measurement informational needs can be met. (Wells and specific in the process. A key advantage ofwithin an organisation can mean significant Johnson 2001: 182) top-down approaches is that they can providechanges for many areas of the organisation. • Outcomes-based accountability can involve a consistent language and conception ofMacKeith and Graham (2007: 14) comment shifts in agency culture at all levels (Wells high-level outcomes, with the assurance thatthat organisations typically underestimate the and Johnson 2001: 177); stakeholder outcomes information gathered from individualmagnitude of impact of outcome measurement involvement helps facilitate this cultural service units will then be able to be easilyon service delivery (both in changes to shift and allows space for concerns and aggregated and fitted into this framework.practice, and in benefits). It can be a major issues to be raised and worked through. Burns and Cupitt (2003: 15) note that largecultural shift for an organisation. Outcomes organisations often find it useful to identifymeasurement also needs to integrate with There can be a tension, however, between a set of high-level outcome areas commonservice planning and review activities, and with inclusive process and speed of implementation: to all services, to provide this type of overallIT systems. "Many states and localities are using an framework. inclusive process in the development andThere is no definitive roadmap for introduction implementation of outcome initiatives. For The two approaches are not mutuallyof outcomes measurement, and it is likely that example, they begin by organising work groups exclusive, and tend to be combined in varyingevery organisations journey will be unique, or engaging a broad advisory committee. The degrees in implementations of outcomeswith much learning along the way (Hendrick more people involved, the longer the process systems. The approach used may also2010b). However, there are also common will take. Conversely, the fewer the people, fluctuate over time in response to the stagethemes in the literature around key processes the more quickly the process may proceed. of implementation. For example, Melbournethat support implementation, and some of However, without strong commitment and Citymission found it useful to begin in 2008these are summarised here. understanding, the process may be derailed by with a bottom-up approach, to introduce the lack of agreement" (McDaniel 1996). concepts of outcome measurement to frontline9.1 StakeHolder involveMent staff and generate statements of key outcomes Bottom-up vs. top-down implementation for particular services. The balance has nowMost authors emphasise the importance of It is evident that both bottom-up and top-down shifted to incorporate more top-down work,involving a range of stakeholders throughout approaches can be used to the development looking to standardise a core set of outcomesthe process of implementation of outcomes of desired outcome statements and measures. and measures to enable aggregation of datameasurement, including the early stages of Bottom-up approaches start with individual across the organisation (Hendrick 2010b).development of the system. This is seen as teams or service units discussing their goalshaving a number of benefits: and valued outcomes, and how they might 9.2 key StageS in iMPleMentation• Involvement of stakeholder increases "buy- measure achievement of these (Hendrick in", i.e. level of stakeholder commitment 2010a). A key advantage of bottom-up A number of authors outline a series of and understanding, and therefore approaches is that they tend to generate a lot key steps or stages in the selection and increases the chances of successful of enthusiasm and commitment from service implementation of outcome measures. In implementation (Wells and Johnson 2001: delivery staff, who have a strong sense of some cases, these appear to be intended as 177-178). In particular, Burns and Cupitt ownership of the outcomes being measured. a linear sequence to follow (e.g. Wells and (2003: 36) advocate that staff should be Johnson (2001)). In other cases, they appear "brought on board early on, rather than Top-down approaches tend to be driven to be intended more as key areas that need be presented with a monitoring tool and centrally by management, defining overarching to be covered off during the development required to gather information." organisational outcomes and indicators and implementation process, although not• Stakeholder perspectives are vital to (perhaps with reference to desired population necessarily in a linear progression. defining the types of outcomes that should outcomes) and then looking at how these can be measured. Stakeholder involvement be devolved to individual service streams, and helps to ensure that stakeholders then to individual teams, often becoming more 67
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesWhile different sources vary on the details, the approaches tend to have many common elements.These are summarised in Table 23, which draws on five sources: Wells and Johnson (2001:182-193), McDaniel (1996), MacKeith and Graham (2007), Hatry and Lampkin (2003) and Burns andCupitt (2003).Table 23: Key stages in development and implementation of an outcomes measurement system Stage Elements Define purpose • Identify the primary reasons why outcome measurement is being introduced (decision-making, accountability, and audience improving effectiveness, as a casework tool, ...) • Identify the intended audiences for outcomes information (including both internal and external audiences) • Identify which programs are to be included Identify desired • Review and clarify program aims and areas of responsibility, drawing on program documentation, observation of outcomes practice, and stakeholders’ views • Involve a wide range of stakeholders (clients, staff, management, funders, sector partners, the community) in discussing the key results that the program aims to produce. Explore concerns and questions • Identify areas of agreement and seek a resolution to areas of disagreement regarding desired results (it is unlikely to be possible to satisfy everyone) • Define a set of outcome domains and within these, statements of key desired outcomes, linked to the program aims • Prioritise among multiple outcomes: determine those that are most important Select measures • Determine the types of measures that will be used to assess each outcome. Add indicators systematically based and measurement on priority of the outcome and feasibility and cost of collection tools • For each outcome, examine how setting performance standards may cause unintended consequences. Create checks and balances as necessary • Identify key client and service characteristics to be linked to outcome information • Identify whether there are pre-existing measurement tools which cover the types of outcomes and measures under consideration, and whether these are viable in the service context • Where pre-existing tools are not available or cannot be used, identify required sources of data for each measure • Examine currently available data to determine whether any of it is directly relevant to desired outcomes. Begin to use this data for performance monitoring • Where required, develop measurement instruments that will enable collection of data for the desired outcomes • Pilot and revise measurement tools to ensure they are user-friendly and are gathering in the required information • To the extent possible, examine the validity, reliability, sensitivity and cultural appropriateness of proposed measures and tools 68
    • Literature Review:Measurement of Client Outcomes in Homelessness Services Stage Elements Implement the • Develop an implementation plan that is flexible to adapt to ongoing learning measurement • Allocate resources and responsibility to undertake development, training, data collection, data analysis and process reporting, and review of the measurement processes • Pilot the measurement processes on a limited scale before rolling out on an agency-wide basis • Ensure that measurement processes are integrated into the other processes of the organisation (especially service delivery) • Provide training to those collecting and using the data • Revise pre-existing forms and data collection instruments where required to increase consistency and minimise duplication • Revise electronic data systems to support data capture Implement • Develop reporting templates and produce reports feedback systems • Seek explanations for unusual or unexpected findings • Ensure that data is fed back to teams (e.g. through team meetings or in supervision) • Ensure that data is used as an input to organisational planning and quality improvement activities Create an • Demonstrate clear and ongoing executive commitment to outcomes measurement outcomes-oriented • Encourage all staff to see outcomes (and their measurement) as central to their work culture within the • Celebrate successful outcomes at an individual, team and organisational level organisation • Reflect on opportunities for improvement and areas where the organisation wishes to “turn the curve” – to make a substantial difference to its effectiveness9.3 tiMefraMeAs every organisational context is different, it is clearly impossible to give a definitive estimateof timeframe for introduction of an outcome measurement system. However, a number ofauthors agree that it tends to be a long-term process, due to the many philosophical, technicaland political challenges associated with the work. It could be hypothesised that timeframe ofimplementation will be dependent partly on level of resourcing for the developmental work, andpartly on the size of the organisation and the diversity of its operations.McDaniel (1996) provides a ballpark estimate for the initial outcomes development stage,stating that "many states and local agencies are taking up to two years in the beginning stagesof outcome development. The challenge is to balance the necessary process with the need tomaintain momentum and interest." Hatry and Lampkin (2003: 4) also map out a two-year processfor initial introduction of measures, including a pilot phase of around 12 months. Burns and Cupitt(2003: 38) suggest between 6 and 18 months for implementation depending on the size andcomplexity of the organisation. 69
    • Literature Review:Measurement of Client Outcomes in Homelessness Services9.4 defining and organiSing The following suggestions indicate someMeaSureS elements that help to sustain outcomeOrganisations introducing a package of client measurement systems over time:outcome measures (or other performance • Understand the embedding of outcomesmeasures) need to be able to clearly define the measurement as an ongoing process thatmeasures, organise the definitions and update will require ongoing attention (Burns andthem as required. Barr (n.d.: 15) suggests Cupitt 2003: 37)adopting a single, standard organisation-wide • Strong commitment from seniortemplate for defining performance measures. management to outcomes measurementShe also suggests that organisations maintain over the long term, and cleara dictionary summarising their performance communication of that commitment andmeasures, which will provide users of the vision to the rest of the organisation (Wellsmeasures (including those coordinating and Johnson 2001: 177-178)measurement systems and those analysing • A preparedness to respond appropriatelythe data) with up-to-date information on the to potential "bad news" (including apparentmeasures. A database could be used for this short-term program failure) (Wells andpurpose. Johnson 2001: 178) • A commitment to recurrent allocation of9.5 SuStaining outcoMe resources to support the measurementMeaSureMent SySteMS system • Staff performance expectations tied toAnecdotally, there is evidence that sustaining completion of outcomes measurementa measurement system is one of the key for an appropriate proportion of clientschallenges in the outcome measurement field (although not expectations tied to the(Talbot 2010). Often the initial implementation outcomes themselves).of a measurement system receives appropriateresourcing and is accompanied by a burst Outcomes measurement systems, like allof enthusiasm as staff focus on the goals of performance measurement systems, needtheir work with clients and begin recording to recognise and respond to the changingoutcomes. However, maintaining staff informational needs of stakeholders over timemotivation over time amidst the pressure (McDavid and Hawthorn 2006: 328-). A regularand stress of service delivery can be difficult. process of review of the measures will helpAs other organisational priorities and quality to keep the outputs relevant and aligned withimprovement initiatives are introduced, desired client outcomes.outcomes systems may be put on thebackburner and lose impetus. This canbecome a self-reinforcing process as dataquality drops and the information producedbecomes less meaningful.Sustainable measurement systems are thosethat are embedded so that they becomea normal and routine part of operations.However, few authors have concretesuggestions about how this can be achieved. 70
    • Literature Review:Measurement of Client Outcomes in Homelessness Services10. Key questions to resolve in developing an outcome measurement framework Whose outcomes will be measured?A wide range of questions need to be • How do desired outcomes, and agreedanswered in developing a comprehensive outcome measures, fit with program logic • Aim for entire client population, or aframework for measuring outcomes within models of the organisations services? sample? What sampling procedure will bea human service organisation. The list • What theory of change underlies the used?of questions below are informed by the organisation’s approach to casework and to • Individuals or households / families?considerations outlined by Berman and Hurt outcome measurement? • What processes will be used to seek(1997: 84-85), but also by the issues identified consent to collect, use and exchangethroughout this literature review. Which measurement approaches, tools outcomes data? and measures will be used to monitor these • How will service delivery be adapted (if atThe list appears daunting, and does in fact outcomes? all) for clients who opt out?represent a large amount of work which will • Goal attainment scaling? Standardisedbe required in developing a well-thought-out scales? Status maintenance and change At what times will outcomes be measured?measurement system. However, there is some scales? • Ongoingly or episodically (snapshot)?logical flow through the questions so that • Whose vantage point should be prioritised? • Linked to key service events for individualanswering earlier ones will help to answer later • Are there pre-existing measurement tools clients (e.g. entry, exit, housing transition)ones. For example, identifying which domains available that are suited to the outcomes or regular cyclical measurement (e.g.and loci of change are of interest will help that the organisation wishes to monitor, monthly, quarterly)?to answer questions about the appropriate or does the organisation need to develop • How will outcome measurement be timedmeasures and tools, the required sources of its own? for services with brief or unpredictabledata, and who will be in the best position to • Which measures should be applied across client contact (e.g. Initial Assessment andgather this data. all services and which measures should Planning)? apply only to selected services, functions • Will follow-up data be collected, and if so,What are the key outcomes that the or client groups? at what duration(s) after exit?organisation wishes to monitor? • Should the measures and tools be designed• Who are the key stakeholders for outcomes in modular formats to allow use of sections How will outcomes data be gathered measurement and what are their needs? of measures for particular contexts? and stored?• Which domains are of interest? • What process, activity or output measures • What are the key sources of data for the• Which loci of change are of interest? will be used in conjunction with the measures selected (self-administered• Which time frames (short, medium, and/or outcome measures? questionnaire, observation, records, ...)? long term)? • What breadth and depth of measures are • Who should undertake data collection• Which interim outcomes are important? cost-effective for the organisation to apply? (service delivery staff, service development• Which outcomes are consistent across • What breadth and depth of measures are staff, volunteers, ...)? the organisation and which are specific to feasible to collect in particular services? • How (if at all) will outcomes measurement particular services or client groups? • How can additional measures be added for processes be integrated into casework• What unintended outcomes (if any) should particular purposes (e.g. for a snapshot processes such as assessment, action the organisation seek to monitor? during a particular time period)? planning and review? • How can the organisation maximise the • Where will data be collected (in the office,How do outcomes and measures fit with the potential for consistency of approach with on outreach to clients accommodation,organisations theory of casework? sector partners? ...)?• What is the connection between outcomes • In what format will data be collected measures, assessment information, action (paper, electronic, Rickter scale, verbal plan goals and action plan progress then later recorded, ...)? reviews? • In what format will data be stored for analysis (database, spreadsheet, ...)? • What modifications or developments need to occur to IT systems to allow this? • Who will enter the data? 71
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesHow will outcomes data be used?• What aggregate reports are required, for which stakeholders, in what formats, and at what times?• What analysis techniques and tools will be used to generate these reports?• How (if at all) will outcomes be adjusted for casemix (severity and complexity of presenting issues)?• How can limitations on internal and external validity be made clear to users of these reports?• How will aggregate reports be used in team, business and strategic planning?• How will outcomes data for individual clients be made available to those clients and the staff working with them, in an accessible and user-friendly format?What resources will be allocated to sustain outcome measurement within the organisation?• What resources will be allocated for development and review of the system (IT development, development of measures and tools, review of pilots...)?• What resources will be allocated for operation of the system (data collection, data entry, data analysis and reporting, ...)?• What documentation and training will be provided to staff in order for them to operate the system to a high standard?• What incentives (if any) will be provided ? 72
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesReferencesAnderson, C. (2008). An Overview of Soft Outcome / Informal Learning Measurement Tools. Norwich, The Research Centre, City College Norwich.Andresen, R., P. Caputi, et al. (2006). "The Stages of Recovery Instrument: Development of a measure of recovery from serious mental illness." Australian and New Zealand Journal of Psychiatry 40: 972-980.Babor, T. F., J. C. Higgins-Biddle, et al. (2001). AUDIT: The Alcohol Use Disorders Identification Test. Guidelines for Use in Primary Care. Geneva, World Health Organization, Department of Mental Health and Substance Dependence.Barr, S. (2008). PuMP® Implementer E-Workbook: active ready-to-use templates to design and bring to life your performance measures.Barr, S. (2009). Case Study Notes: Measuring Outcomes to End Homelessness: Meaningful Measures for Program Outcomes from the National Alliance for Ending Homelessness. Stacey Barr’s Measures & More Mastermind.Barr, S. (n.d.). 202 Tips for Performance Measurement: Practical ideas to choose, create and use meaningful performance measures with lots of buy-in!Baulderstone, J. and C. Talbot (2004). Outcome Measurement in SAAP Funded Services: Final Report, Flinders Institute of Public Policy and Management.Berman, W. H. and S. W. Hurt (1997). Developing Clinical Outcomes Systems: Conceptual and Practical Issues. Outcomes Measurement in the Human Services: Cross-Cutting Issues and Methods. E. J. Mullen and J. L. Magnabosco. Washington, DC, NASW Press: 81-97.Bloom, M., J. Fischer, et al. (2006). Evaluating practice: Guidelines for the accountable professional. Boston, Pearson / Allyn & Bacon.Blunden, H. and C. Johnston (2005). Public housing and nonhousing outcomes – a background paper. Shelter Brief. Sydney, Shelter NSW.Booth, B. M. and G. R. Smith (1997). Outcomes Measurement: Where We Are. Outcomes Measurement in the Human Services: Cross-Cutting Issues and Methods. E. J. Mullen and J. L. Magnabosco. Washington, DC, NASW Press: 35-43.Buckingham, B., P. Burgess, et al. (1998a). Developing a Casemix Classification for Mental Health Services. Volume 1: Main Report. Canberra, Commonwealth Department of Health and Family Services.Buckingham, B., P. Burgess, et al. (1998b). Developing a Casemix Classification for Mental Health Services. Volume 2: Resource Materials. Canberra, Commonwealth Department of Health and Family Services.Burke, T. and K. Hulse (2002). Sole parents, social wellbeing and housing assistance. Final Report. Melbourne, Australian Housing and Urban Research Institute.Burns, S. and S. Cupitt (2003). Managing outcomes: a guide for homelessness organisations. London, Charities Evaluation Service / London Housing Foundation.Burns, S. and J. MacKeith (2009a). Alcohol Star: the Outcomes Star for alcohol misuse. User Guide., Alcohol Concern and Triangle Consulting.Burns, S. and J. MacKeith (2009b). Work Star: The Outcomes Star for work and learning. Organisation Guide. London, Triangle Consulting. 73
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesBusch-Geertsema, V. (2005). "Does Re-Housing lead to re-integration? Follow-up studies of re-housed homeless people." Innovation: The European Journal of Social Science Research 18(2): 205-226.Butcher, B. and L. Marsden (2004). Measuring Soft Outcomes: A Review of the Literature. Norwich, The Research Centre, City College Norwich.Calsyn, R. and J. Winter (2002). "Social Support, Psychiatric Symptoms, and Housing: A Causal Analysis." Journal of Community Psychology 30(3): 247-259.Campbell-Orde, T., J. Chamberlin, et al. (2005). Measuring the Promise: A Compendium of Recovery Measures Volume II. Cambridge MA, Human Services Research Institute.Chamberlain, C., G. Johnson, et al. (2007). Homelessness in Melbourne: Confronting the Challenge. Melbourne, RMIT University Press.Christo, G. (2008). "Christo Research Systems." Retrieved 17 June, 2010, from http://users.breathe.com/drgeorgechristo/.Clark, C. and A. R. Rich (2003). "Outcomes of Homeless Adults With Mental Illness in a Housing Program and in Case Management Only." Psychiatric Services 54(1): 78-83.Clements, D. (2010). Personal communication: Meeting at HomeGround Services, 26/03/2010.Common Ground. (n.d.). "Vulnerability Index: Prioritizing the Street Homeless Population by Mortality Risk." Retrieved 16 April, 2010, from http:// www.commonground.org/wp-content/uploads/2008/04/Vulnerability%20Index%20101.pdf.Commonwealth of Australia (2008). The Road Home: A National Approach to Reducing Homelessness. Canberra, Department of Families, Housing, Community Services and Indigenous Affairs.Commonwealth of Australia, State of New South Wales, et al. (2005). MULTILATERAL AGREEMENT IN RELATION TO THE SUPPORTED ACCOMMODATION ASSISTANCE PROGRAM.Cook, D. A. and T. J. Beckman (2006). "Current Concepts in Validity and Reliability for Psychometric Instruments: Theory and Application." The American Journal of Medicine 119: 166.e167-166.e116.Cook, T. D. and D. T. Campbell (1979). Quasi-Experimentation: Design & Analysis Issues for Field Settings. Boston, Houghton Mifflin.Core Systems Group (1998). CORE System User Manual. Leeds, Core Systems Group.Council of Australian Governments (2009a). National Affordable Housing Agreement.Council of Australian Governments (2009b). National Partnership Agreement on Homelessness.Culhane, D. P., W. D. Parker, et al. (2007). Accountability, Cost-Effectiveness, and Program Performance: Progress Since 1998. 2007 National Symposium on Homelessness Research: 12-11 to 12-42.Cunningham-Smith, V. (2010). A journey keeping our focus on making sure people are better off as a results of our services. The search for consumer outcomes in health and welfare services. Powerpoint presentation. Melbourne, Inner East Community Health Service. 74
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesDavies, R. and J. Dart (2005). The ‘Most Significant Change’ (MSC) Technique: A Guide to Its Use.de Vaus, D. (2002). Analyzing Social Science Data: 50 Key Problems in Data Analysis. London, SAGE Publications.DeCoster, J. (1998). "Overview of Factor Analysis." Retrieved 6 April, 2010, from http://www.stat-help.com/factor.pdf.Department of Health. (2009). "About Outcome Measurement: Victorian Collection of OM in PDRSS." Retrieved 19 May, 2010, from http://www.health. vic.gov.au/mentalhealth/outcomes/about2.htm.Department of Health and Ageing (2003). Mental Health National Outcomes and Casemix Collection: Overview of clinician-rated and consumer self-report measures, Version 1.50. Canberra, Department of Health and Ageing, Mental Health & Suicide Prevention Branch.Department of Human Services (2006). Homelessness Assistance Service Standards. Melbourne, Victorian Department of Human Services.Department of Human Services (2009). Opportunities in practice: Outcome measurement in mental health. Revised edition, March 2009. Melbourne, Victorian Department of Human Services Mental Health and Drugs Division.DHS Evaluation Support Unit (n.d.). Evaluation frameworks: A development guide. Melbourne, Department of Human Services.Downtown Emergency Service Centre. (n.d.). "Vulnerability Assessment Tool." Retrieved 16 April, 2010, from http://elearning.ahc.org/files/ DESC_Vulnerability%20Assessment_Tool%202003.pdf.Duignan, P. (2005). "Outcomes hierarchy definitions." Retrieved 21 April, 2010, from http://www.parkerduignan.com/ot/untitled_text_page.html.Eagar, K., W. Buckingham, et al. (2001). The Victorian Mental Health Outcomes Measurement Strategy: Final Report on the Implementation of Outcome Measurement In Adult Area Mental Health Services. Melbourne, Victorian Department of Human Services.Eardley, T., D. Thompson, et al. (2008). Measuring the Impact of SAAP-funded Homelessness Services on Client Self-reliance: Final Report. Sydney, Social Policy Research Centre, University of New South Wales.Elizabeth, S. (2010). Personal communication: Meeting between Sally Elizabeth and Mark Planigale to discuss Youth Refuges outcome measurement pilot. Meeting at HomeGround Collingwood, 14/04/2010.ERoSH. (2010). "Publications." Retrieved 17 June, 2010, from http://www.shelteredhousing.org/information/publications.Evans, S., J. Greenhalgh, et al. (2000). "Selecting a mental health needs assessment scale: guidance on the critical appraisal of standardized measures." Journal of Evaluation in Clinical Practice 6(4): 379-393.Felce, D. and J. Perry (1995). "Quality of life: Its definition and measurement." Research in Developmental Disabilities 16(1): 51-74.Fitzpatrick, J. L., J. R. Sanders, et al. (2004). Program Evaluation: Alternative Approaches and Practical Guidelines. 3rd edition. Boston, Pearson.Friedman, M. (2005). Trying Hard Is Not Good Enough: How to Produce Measurable Improvements for Customers and Communities, FPSI Publishing / Booksurge. 75
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesFriedman, M., L. DeLapp, et al. (2001a). The Results and Performance Accountability Implementation Guide: Case Studies.Friedman, M., L. DeLapp, et al. (2001b). The Results and Performance Accountability Implementation Guide: Questions and Answers about How to Do the Work.Gray, J. (2008). Outcome measures for preventing homelessness through mediation, Framework.Hamilton, L. (2010). Personal communication: Meeting between Lorrinda Hamilton and Mark Planigale, 68 Oxford St Collingwood, to discuss outcomes measurement in the context of the local Mental Health Alliance. Meeting date 31/03/2010.Hatry, H. and L. Lampkin (2003). Key Steps in Outcome Management. Series on Outcome Management for Nonprofit Organizations. H. Hatry and L. Lampkin, The Urban Institute.Hatry, H. P. (1997). Outcomes Measurement and Social Services: Public and Private Sector Perspectives. Outcomes Measurement in the Human Services: Cross-Cutting Issues and Methods. E. J. Mullen and J. L. Magnabosco. Washongton, DC, NASW Press: 3-19.Health Care for the Homeless (1998). Health Care for the Homeless Outcome Measures... A Chronicle of Twenty Pilot Studies. October 1998, Health Care for the Homeless.Hendrick, E. (2010a). MORFing @ MCM: The MORF: Measuring Outcomes & Results Framework. Powerpoint presentation, 31/03/2010. Melbourne, Melbourne Citymission.Hendrick, E. (2010b). Personal communication: Meeting between Elaine Hendrick and Mark Planigale to discuss MORF. Meeting at 79 Little Oxford St Collingwood, 16 April 2010.Homeless Link. (2010). "The Outcomes Star System." Retrieved 12 May, 2010, from http://www.outcomesstarsystem.org.uk/.Hubley, A. M., L. B. Russell, et al. (2009). Quality of Life for Homeless and Hard-to-House Individuals (QoLHHI) Inventory: Administration and Scoring Manual. Vancouver, University of British Columbia.Hudson, W. W. (1997). Assessment Tool as Outcomes Measures in Social Work. Outcomes Measurement in the Human Services: Cross-Cutting Issues and Methods. E. J. Mullen and J. L. Magnabosco. Washington, DC, NASW Press: 68-80.Hulse, K. and L. Saugeres (2008). Housing insecurity and precarious living: an Australian exploration. AHURI Final Report No. 124. Melbourne, Australian Housing and Urban Research Institute.Johnson, G. (2010). Personal communication: Comments by Guy Johnson at Outcomes Project Reference Group Meeting, 3 May 2010.Johnson, G., H. Gronda, et al. (2008). On the outside: pathways in and out of homelessness. North Melbourne, Vic., Australian Scholarly Publishing.Kaufman, M. C. (2007). Durham System Of Care: Life Domain Guide for Child and Family Teams, The Durham Centre.Kimberley, S. (2009). Monitoring and Evaluation: Definitions. Melbourne, HomeGround Services. 76
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesKolar, V. (2004). Home First: A longitudinal study of outcomes for families who have experienced homelessness - Final Report. Melbourne, Hanover Welfare Services.Love, A. and M. Scott (2010). Personal communication: Meeting between Mark Planigale, Anna Love and Matthew Scott at Clarendon Clinic, 19/03/2010.MacKeith, J. (2007). ‘User-focussed’ – what does it really mean? London, London Housing Foundation.MacKeith, J. and S. Burns (2008). Mental Health Recovery Star. User Guide. London, Mental Health Providers Forum and Triangle Consulting.MacKeith, J., S. Burns, et al. (2008a). The outcomes star: Supporting change in homelessness and related services, 2nd edition. Organisation Guide. London, London Housing Foundation and Triangle Consulting.MacKeith, J., S. Burns, et al. (2008b). The outcomes star: Supporting change in homelessness and related services, 2nd edition. User Guide. London, London Housing Foundation and Triangle Consulting.MacKeith, J. and K. Graham (2007). A Practical Guide to Outcomes Tools. London, London Housing Foundation.MacKeith, J., K. Graham, et al. (2007). Review of Outcomes Tools for Homelessness Sector, London Housing Foundation.Marsden, J., M. Gossop, et al. (1998). The Maudsley Addiction Profile: A brief instrument for treatment outcome research. Development and User manual. London, National Addiction Centre / Institute of Psychiatry.McDaniel, N. C. (1996). "Key Stages and Common Themes in Outcome Measurement." The Evaluation Exchange 2(3) Retrieved 29 March, 2010, from http://www.hfrp.org/evaluation/the-evaluation-exchange/issue-archive/results-based-accountability-2/key-stages-and-common-themes-in- outcome-measurement.McDavid, J. C. and L. R. L. Hawthorn (2006). Program Evaluation & Performance Measurement: An Introduction to Practice. Thousand Oaks, California, Sage Publications.McLean Hospital. (2010). "Measuring Patient Self-Reported Outcomes." from http://www.basissurvey.org/basis32/.Mertens, D. M. (2008). Stakeholder Representation in Culturally Complex Communities: Insights from the Transformative Paradigm. Fundamental Issues in Evaluation. N. L. Smith and P. R. Brandon. New York, The Guilford Press: 41-60.Mission Australia (2005). NSW/ACT Community Services Outcomes Measurement Tool Kit. Version 1. November 2005. Sydney, Mission Australia, NSW/ ACT Community Services.Mission Australia (2006). Internal Service Evaluation. Template Version 4. June 2006. Sydney, Mission Australia, NSW/ACT Community Services.National Alliance to End Homelessness. (2006). "Frequently Asked Questions about Housing First for Individuals and Families." Retrieved 22 April, 2010, from http://www.endhomelessness.org/content/article/detail/1424.National Treatment Agency for Substance Misuse (2007). Treatment Outcomes Profile v1.0, National Treatment Agency for Substance Misuse. 77
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesNMHWG Information Strategy Committee Performance Indicator Drafting Group (2005). Key Performance Indicators for Australian Public Mental Health Services: Report of Performance Indicator Drafting Group. ISC Discussion Paper No.6. Canberra, Australian Government Department of Health and Ageing.Office of Housing (n.d.). Measuring outcomes: Rationale, ideas and resources. A resource kit for Victorian agencies implementing the Homelessness Assistance Service Standards and the National Community Housing Standards. Melbourne, Victorian Department of Human Services, Office of Housing.OSW (2004). The Employability Map: A tool for measuring progress to employability, Off the Streets and Into Work.Parkinson, D. (2005). Putting outcomes into practice: A guide to developing IT-based outcome data management. London Housing Foundation IMPACT Programme briefing paper, London Housing Foundation & Charities Evaluation Service.Patton, M. Q. (1987). How to Use Qualitative Methods in Evaluation. Newbury Park, California, Sage Publications.Phibbs, P. (2002). Housing assistance and non-shelter outcomes. Sydney, AHURI.Pilon, D. and M. Ragins (2007). Milestones of Recovery Scale (MORS).Planigale, M. (2010a). Discussion paper: A consistent set of casework domains for HomeGround. Melbourne, HomeGround Services.Planigale, M. (2010b). Measuring Service Performance – the Big Picture. Melbourne, HomeGround Services.Planigale, M., S. Kimberley, et al. (2009). HomeGround Services – Program Logic Model – version: 18 Feb 2009. Melbourne, HomeGround Services.Pleace, N. (2008). Effective Services for Substance Misuse and Homelessness in Scotland: Evidence from an International Review. York, The Centre for Housing Policy, University of York / Scottish Government Social Research.Post, P. A., F. E. Isbell, et al. (2005). Developing Outcome Measures to Evaluate Health Care for the Homeless Services. Nashville, National Health Care for the Homeless Council.Priebe, S., P. Huxley, et al. (1999). "Application and results of the Manchester Short Assessment of Quality of Life (MANSA)." Int J Soc Psychiatry 45(1): 7-12.QualityMetric. (2010). "Generic Health Surveys." Retrieved 17 June, 2010, from http://www.qualitymetric.com/WhatWeDo/GenericHealthSurveys/ tabid/184/Default.aspx.Ralph, R. O. (n.d.). "At the Individual Level: A Personal Measure of Recovery." Retrieved 17 June, 2010, from http://www.nasmhpd.org/spec_e- report_fall04measures.cfm.Rapp, C. A. and J. Poertner (1992). Social administration: A client-centered approach. New York, Longman.Rethink (n.d.). CUES: Carers’ and Users’ Expectations of Services. London, Rethink. 78
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesRossi, P. H. (1997). Program Outcomes: Conceptual and Measurement Issues. Outcomes Measurement in the Human Services: Cross-Cutting Issues and Methods. E. J. Mullen and J. L. Magnabosco. Washington, DC, NASW Press: 20-34.Rossi, P. H., H. E. Freeman, et al. (1999). Evaluation: A Systematic Approach. 6th edition. Thousand Oaks, California, Sage Publications.Schalock, R. L. (2001). Outcome-Based Evaluation. New York, Kluwer Academic / Plenum Publishers.Segal, S. P. (1997). Outcomes Measurement Systems in Mental Health: A Program Perspective. Outcomes Measurement in the Human Services: Cross-Cutting Issues and Methods. E. J. Mullen and J. L. Magnabosco. Washington, DC, NASW Press: 149-159.Segone, M. (2008). Bridging the gap: the role of monitoring & evaluation in evidence-based policy making. Geneva, Unicef.Siggins Miller Consultants (2003). A report to the Mental Health Branch, Department of Human Services: Consumer self-rated outcome measures in mental health.Smith, G. R., K. M. Rost, et al. (1997). Assessing Effectiveness of Mental Health Care in Routine Clinical Practice. Outcomes Measurement in the Human Services: Cross-Cutting Issues and Methods. E. J. Mullen and J. L. Magnabosco. Washington, DC, NASW Press: 124-136.Social Solutions. (2010). "Human Services Software." Retrieved 12 May, 2010, from http://www.socialsolutions.com/Human-Services-Software/ Performance-Management.aspx#.Spellman, B. and M. Abbenante (2008). What Gets Measured, Gets Done: A Toolkit on Performance Measurement for Ending Homelessness, National Alliance to End Homelessness - Homelessness Research Institute.Spence, S. H., M. Donald, et al. (2002). Outcomes and indicators, measurement tools and databases for the National Action Plan for Promotion, Prevention and Early Intervention for Mental Health 2000. Brisbane, School of Psychology and Centre for Primary Health Care, University of Queensland.Stedman, T., P. Yellowlees, et al. (1997). Measuring Consumer Outcomes In Mental Health: Field Testing of Selected Measures of Consumer Outcome in Mental Health. St Lucia, Qld, University of Queensland.Stevens, B. (2009). Personal communication: Email from Briannan Stevens re: Pilot of Outcomes Star at Micah Projects, Brisbane. Email dated 4 April 2009.Talbot, C. (2010). Personal communication: Meeting between Chris Talbot and Mark Planigale, Fitzroy, Friday 9 April 2010.Trauer, T., G. Tobias, et al. (2008). "Development and evaluation of a patient-rated version of the Camberwell Assessment of Need Short Appraisal Schedule (CANSAS-P)." Community Mental Health Journal 44(2): 113-124.Trinh, Q.-T. (2010). Personal communication: Meeting between Mark Planigale, Quynh-Tram Trinh and Trish Watson at Department of Human Services, Melbourne, 17/03/2010.Trochim, W. M. K. (2006a). "Nonprobability Sampling." Retrieved 10 May, 2010, from http://www.socialresearchmethods.net/kb/sampnon.php. 79
    • Literature Review:Measurement of Client Outcomes in Homelessness ServicesTrochim, W. M. K. (2006b). "Probability Sampling." Retrieved 10 May, 2010, from http://www.socialresearchmethods.net/kb/sampprob.php.University of Melbourne. (2010). "WHOQoL Instruments downloads." Retrieved 17 June, 2010, from http://www.psychiatry.unimelb.edu.au/qol/ whoqol/whoqol-insruments.html.University of New South Wales. (2010). "Depression Anxiety Stress Scale." Retrieved 17 June, 2010, from http://www2.psy.unsw.edu.au/groups/dass/.University of Toronto Centre for Health Promotion. (2010). "QOL Concepts: The Quality of Life Model." Retrieved 6 April, 2010, from http://www. utoronto.ca/qol/concepts.htm.Wadsworth, Y. (1997). Everyday Evaluation on the Run. 2nd edition. Crows Nest, NSW, Allen & Unwin.Ware, J. E. (1997). Health Care Outcomes from the Patients Point of View. Outcomes Measurement in the Human Services: Cross-Cutting Issues and Methods. E. J. Mullen and J. L. Magnabosco. Washington, DC, NASW Press: 44-67.Weiss, C. (1972). Evaluation research: Methods for assessing program effectiveness. Englewood Cliffs, New Jersey, Prentice-Hall, Inc.Weiss, H. B. (1997). Results-Based Accountability for Child and Family Services. Outcomes Measurement and Social Services: Public and Private Sector Perspectives. E. J. Mullen and J. L. Magnabosco. Washington, DC, NASW Press: 173-180.Wells, S. J. and M. A. Johnson (2001). "Selecting Outcome Measures for Child Welfare Settings: Lessons for Use in Performance Management." Children and Youth Services Review 23(2): 169-199.World Health Organisation (1996). WHOQOL-BREF: Introduction, administration, scoring and generic version of the assessment. Field trial version. Geneva, World Health Organisation Programme on Mental Health.World Health Organisation. (2010). "Severity of Dependence Scale (SDS)." Retrieved 17 June, 2010, from http://www.who.int/substance_abuse/ research_tools/severitydependencescale/en/index.html. 80
    • ContaCt uswww.HoMeground.org.auinfo@HoMeground.org.auCollingwood City68 Oxford Street 660 Elizabeth StreetCollingwood 3066 Melbourne 3000Housing/Support 9288 9611 Supportive Housing 8359 0200Freecall 1800 048 325 Fax 8359 0201Outreach 9288 9650Fax 9288 9602 PraHranCorporate enquiries 9288 9600 PO Box 354Fax 9288 9601 Prahran 3181 Tenancy & Property 9537 7999st Kilda Housing Options 9932 8368122 Chapel Street Fax 9537 7797St Kilda 3182Housing/Outreach 9537 7711 PrestonFax 9537 7722 600 Plenty RoadSupport 9537 7888 Preston 3072Fax 9537 7991 Support 9471 4522 Freecall 1300 844 125 Fax 9471 4533