Literature Review:MeasureMent of Client outCoMesin HoMelessness serviCes                           january 2011Mark Planig...
Mark Planigale                              Research & Consultancy                                     results by design  ...
Literature Review:Measurement of Client Outcomes in Homelessness ServicesContents1. introduction                          ...
Literature Review:Measurement of Client Outcomes in Homelessness Services1. Introduction1.1 aBout HoMeground ServiceS     ...
Literature Review:Measurement of Client Outcomes in Homelessness Services                                                 ...
Literature Review:Measurement of Client Outcomes in Homelessness Services2. Benefits and challenges of client outcome meas...
Literature Review:Measurement of Client Outcomes in Homelessness Services2.2 diSBenefitS,                                 ...
Literature Review:Measurement of Client Outcomes in Homelessness ServicesThe	information	produced	may	be	used	poorly      ...
Literature Review:Measurement of Client Outcomes in Homelessness ServicesA question of particular interest for communityse...
Literature Review:Measurement of Client Outcomes in Homelessness Services3. Homelessness sector:   approaches to defining ...
Literature Review:Measurement of Client Outcomes in Homelessness ServicesTable	2:	National	Partnership	Agreement	on	Homele...
Literature Review:Measurement of Client Outcomes in Homelessness Services                                                 ...
Literature Review:Measurement of Client Outcomes in Homelessness Servicesdata per quarter will be gathered. In addition   ...
Literature Review:Measurement of Client Outcomes in Homelessness Services                                                 ...
Literature Review:Measurement of Client Outcomes in Homelessness Services                                                 ...
Literature Review:Measurement of Client Outcomes in Homelessness Servicesgoals. While there are certainly philosophical   ...
Literature Review:Measurement of Client Outcomes in Homelessness Services4. Measurement of client outcomes as a research a...
Literature Review:Measurement of Client Outcomes in Homelessness Services                                                 ...
Literature Review:Measurement of Client Outcomes in Homelessness ServicesAt times these approaches and their              ...
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Literature review: measurement of client outcomes in homelessness services
Upcoming SlideShare
Loading in...5
×

Literature review: measurement of client outcomes in homelessness services

6,317

Published on

Explores a wide range of practical and theoretical issues relating to introduction of client outcomes measures in welfare / human service organisations, with a particular focus on the housing and homelessness assistance sector.

Published in: Business, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
6,317
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
61
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Literature review: measurement of client outcomes in homelessness services

  1. 1. Literature Review:MeasureMent of Client outCoMesin HoMelessness serviCes january 2011Mark Planigale Service Development and Research
  2. 2. Mark Planigale Research & Consultancy results by design PO Box 754 Macleod VIC 3085 Australia Tel: 0429 136 596 Email: results@planigale.com Web: www.planigale.com HoMeground ServiceS 1a/68 Oxford St Collingwood VIC 3066 Australia Tel: (+61) 3 9288 9600 Email: research@homeground.org.au Web: www.homeground.org.auCopyright © HomeGround Services & Mark Planigale 2010
  3. 3. Literature Review:Measurement of Client Outcomes in Homelessness ServicesContents1. introduction 1 1.1 About HomeGround Services 1 1.2 About HomeGround’s Client Outcome Measures Project 1 1.3 Literature review: aim and key questions 1 1.4 Literature review: methodology and scope 2 1.5 Note on language 2 1.6 Acknowledgements 22. BenefitS and cHallengeS ofclient outcoMe MeaSureMent 3 2.1 Benefits 3 2.2 Disbenefits, challenges and risks 4 2.3 Balancing benefits and risks 53. HoMeleSSneSS Sector: aPProacHeSto defining and MeaSuring client outcoMeS 7 3.1 Australia – national outcomes and indicators 7 3.2 Victorian Department of Human Services 9 3.3 SAAP pilots: Baulderstone and Talbot (2004) 10 3.4 Results-based accountability 10 3.5 Outcomes Star 11 3.6 National Alliance to End Homelessness 12 3.7 Benefits and challenges of a sector-based approach 124. MeaSureMent of client outcoMeSaS a reSearcH activity 14 4.1 Monitoring and evaluation 14 4.2 Naturalistic and experimental approaches to research 15 4.3 Monitoring systems as single-system research designs 16 4.4 Limitations of monitoring systems 175. concePtualiSing client outcoMeS 22 5.1 General definition 22 5.2 Stakeholder perspectives 25 5.3 Outcomes as an element of program logic 25 5.4 Domains 26 5.5 Locus of change 28 5.6 Point of intervention (prevention vs. amelioration) 30 5.7 Housing and non-housing outcomes 316. MeaSureS and MeaSureMent toolS 32 6.1 Criteria for selecting measures and tools 32 6.2 How many measures? 36 6.3 Types of data and types of measures 38 6.4 Possible tools and measures 427. MeaSureMent ProceSSeS 55 7.1 Sampling 55 7.2 Consent 56 7.3 Timing and frequency 56 7.4 Gathering and recording data 57 7.5 IT systems 588. uSe of outcoMeS data 60 8.1 Reporting 60 8.2 Analysis of aggregate data 62 8.3 Using the findings 659. ProceSS of introducing outcoMe MeaSureMent 67 9.1 Stakeholder involvement 67 9.2 Key stages in implementation 67 9.3 Timeframe 69 9.4 Defining and organising measures 70 9.5 Sustaining outcome measurement systems 7010. key queStionS to reSolve in develoPingan outcoMe MeaSureMent fraMework 71referenceS 73
  4. 4. Literature Review:Measurement of Client Outcomes in Homelessness Services1. Introduction1.1 aBout HoMeground ServiceS HomeGround’s Client Outcome Measures literature review document is intended toHomeGround Services is one of Melbourne’s Project aims to establish the foundations of serve as a technical reference during theleading housing and homelessness the systems needed to monitor outcomes for process of implementing an outcomeorganisations. HomeGround’s vision is people engaged with HomeGround’s services. measurement system.to end homelessness in Melbourne, and The project activities include:HomeGround’s mission is to get people housed • Researching the types of indicators that The literature review was guided by theand keep people housed. HomeGround has a can be used to measure housing and following key questions, grouped under theunique combination of experience delivering wellbeing outcomes headings of context, system design, andhigh volume housing and homelessness • Defining outcome measures for each of implementation:services to people in crisis, providing property HomeGround’s service streamsand tenancy management services, and • Identifying or developing tools to collect Context:providing short and long-term support services client outcomes data • What are the known benefits andto people who have experienced homelessness • Providing input to the development of disbenefits or risks of introducing clientin the past. electronic data systems that can store and outcome measurement systems within a report on client outcomes data human service organisation?HomeGround is a strong supporter of ‘housing • Identifying strategies to embed outcome • What current approaches to outcomefirst’ approaches to ending homelessness. measurement in practice. measurement provide guidance or directionHomeGround has played a leading role in to HomeGrounds efforts?introducing new housing and support models The project runs from January–July 2010 and isin Victoria, including the Private Rental expected to be followed by a phase of piloting System design:Access Program and a range of Supportive and implementation of outcome measures • What types of information can be producedHousing models. HomeGround has a strong within the organisation, influenced by the by outcome measurement systems andreputation for quality service provision, findings of the project. what conclusions can this informationeffective advocacy, and for bringing private support?and community sector partners to the table This Literature Review and the accompanying • What client outcomes are of relevance toto achieve change for people experiencing discussion paper A consistent set of casework homelessness services and how can theyhomelessness. domains for HomeGround (Planigale 2010a) be conceptualised and categorised? are the key outputs of Phase 2 (background • What specific measures and measurementHomeGround has a long-standing commitment research) of the project. tools may be of relevance to homelessnessto sector development and a track record of services?involvement in significant research projects 1.3 literature review: • What are the options for data collection(see e.g. Chamberlain, Johnson et al. 2007; aiM and key queStionS processes (when should data be collected,Johnson, Gronda et al. 2008). from who, by who, and in what format)? The aim of the literature review is to gain an • How can results be presented to be1.2 aBout HoMeground’S client overview of political, theoretical and logistical of maximum benefit to clients and theoutcoMe MeaSureS Project considerations related to the introduction of organisation? client outcome measurement systems withinHomeGround is committed to ensuring a homelessness organisation in Australia. The Implementation:that its approach to ending homelessness expected benefit of the literature review is • What are the best processes foris underpinned by strong evidence. The that it will allow the process of developing and introducing outcome measurementorganisation has identified that one of the implementing measures to be well-informed, systems within organisations?most important sources of evidence is well planned, and more likely to be successfulinformation on the outcomes achieved in (avoiding known pitfalls, and enabling more The literature review document is organisedpartnership with clients. useful measures to be developed). The into sections mirroring the above questions. 1
  5. 5. Literature Review:Measurement of Client Outcomes in Homelessness Services 1.6 acknowledgeMentSThe document ends with a section For example, the area of evaluation theorysummarising the main questions that an was identified as an important input to I wish to thank a number of individuals whoorganisation may need to answer in developing the literature review, and the following have generously and insightfully shared theiran outcome measurement system. sources were selected to provide a range experience and knowledge, and have therefore of perspectives on this topic: Weiss (1972), made important contributions to the directionSome elements of the literature review are Schalock (2001), Bloom et al. (2006), and content of this literature review. (Inspecific to the context of the homelessness Wadsworth (1997), Patton (Patton 1987) and alphabetical order): Deb Batterham, Danielservice system in Victoria. However, many Rossi, Freeman et al. (1999). Clements, Yann Decourt, Sally Elizabeth,of the issues discussed appear to be general Sherrill Evans, Sue Grigg, Hellene Gronda,themes within the outcomes measurement There is a plethora of information available in Lorrinda Hamilton, George Hatvani, Elaineliterature and are of wide applicability. the area of client outcomes measurement. Due Hendrick, Heather Holst, Guy Johnson, Deb to the limited time available for the literature Keys, Sue Kimberley, Anna Love, Matthew1.4 literature review: review, many relevant areas were either visited Scott, Theresa Swanborough, Chris Talbot,MetHodology and ScoPe only briefly, or not at all. Topics of interest that Quynh-Tram Trinh, Trish Watson and Zoe Vale. were not able to be adequately pursued withinThe literature review was conducted by Mark the literature review are labelled for further HomeGround also wishes to thank thePlanigale. The research process drew on two research in this document. following partner organisations as a whole formajor sets of sources: their contribution to this project: the Australian• Review of a range of source documents Early in the literature review process, the topic Housing and Urban Research Institute, including books, journal articles, reports, of life domains was identified as a priority to Department of Human Services, Hanover government publications, web pages, and both the Client Outcome Measures Project and Welfare Services, Melbourne Citymission, measurement tools; and to other work in progress at HomeGround. This RMIT University, Royal District Nursing Service• Meetings with selected stakeholders topic was therefore given a stronger emphasis Homeless Persons Program, and Sacred Heart including HomeGround staff and within the project. The discussion paper on Mission. representatives of sector and government casework domains (Planigale 2010a) should be partner organisations. read in conjunction with this literature review document.The document review was ‘seeded’ usingthree publications already known to be of high 1.5 note on languagerelevance: Baulderstone and Talbot (2004),Spellman and Abbenante (2008) and MacKeith In this paper the term client is used toand Graham (2007). These publications, along refer to a person who uses the services of awith initial meetings with stakeholders, were community or welfare organisation. This isused to derive a set of key themes and issues in line with HomeGround Services standardto be explored. These themes form the basis of terminology. It is acknowledged that in otherthe key questions noted above. contexts, a range of other terms may be preferred including consumer, service user,From the seed publications, other relevant patient or participant.sources were identified using a snowballapproach. As the research process identifiedareas where further depth was needed,other relevant sources were found either byinternet search or through library catalogues. 2
  6. 6. Literature Review:Measurement of Client Outcomes in Homelessness Services2. Benefits and challenges of client outcome measurementThe literature identifies a range of benefits what they are doing is working, to what achieved)" (MacKeith 2007: 2). While theassociated with client outcome measurement, extent, and for which clients. It helps motivational benefits may vary acrosshowever it is also worth considering potential the organisation (or service or team) different groups of staff, organisations thatdisbenefits, challenges and risks associated to answer the question of whether it is involve staff in defining desired outcomeswith outcome measurement systems. This being successful in its mission, and can and measures often report an enthusiasticallows a well-informed decision about whether, therefore play a role in guiding decision response (e.g. Hendrick 2010b).and how, to proceed. making. While outcomes measurement by • Advocacy benefits: outcomes measurement itself cannot answer questions related to can assist in demonstrating the successful2.1 BenefitS attribution, in combination with other data results of a program or intervention, as it can potentially provide some information well as potentially demonstrating levelsThe principal benefit that tends to be cited on service effectiveness. Outcomes data of client need. This information mayin favour of client outcome measurement can also at times provide a useful input to help in the task of generating supportis that it focuses staff, organisations and formal program evaluations. (partnerships, public perception, funding).service systems on the needs, goals and • Evaluative benefits (individual level): • Knowledge building: outcomesachievements of clients. Rapp and Poertner information on individual client outcomes measurement can contribute to research(1992: 16) articulate the central tenet that can provide useful data for reflection by and evaluation, can generate hypotheses“the raison d’être of the social administrator individual staff members and clients on and questions for further research, andis client wellbeing and that the principal task individual progress and the effectiveness can contribute to drawing together learningof the manager is to facilitate that wellbeing”, of the services being provided to that from across many organisations to assistand from this derive four principles of individual. the development of evidence based serviceclient-centred management, including Creating • Assessment benefits: outcomes delivery.and maintaining the focus on the clients and measurement can provide clients, serviceclient outcomes. MacKeith (2007) notes that delivery staff, managers and Boards with One note of caution is that while these benefitsa strong focus on the desired outcomes of a an overview of how the clients’ situations are often asserted, there is little documentedservice tends to positively change the way that and needs are changing over time. This research that objectively demonstratesstaff and clients work together. Having a clear is important information that can help them (Booth and Smith 1997: 42). There isshared understanding of what the goals are to drive future service delivery at the certainly anecdotal evidence that outcomes(and a shared language for talking about them) individual and program level. measurement can have positive effects on staffcan be an important basis for working together • Quality improvement benefits: outcomes morale (Clements 2010; Hendrick 2010b), and(Spellman and Abbenante 2008: 4). measurement can drive quality there is some evidence that introduction of improvement, both by identifying what outcome measures by funders can positivelyA focus on client outcomes, together with the works, and by identifying interventions or affect service effectiveness (e.g. Wells andability to measure them in a meaningful way, approaches that are less successful and Johnson 2001: 194). Friedman et al. (Friedman,can have the following benefits (Burns and are in need of review. DeLapp et al. 2001a) cite a number of caseCupitt 2003; MacKeith 2007; MacKeith and • Motivational benefits: outcomes studies in which outcomes measurement wasGraham 2007; Spellman and Abbenante 2008): measurement can help both staff important to demonstrating the population-• Evaluative benefits (system level): and clients to recognise progress and level changes resulting from partnerships outcomes measurement assists funders to celebrate achievements. This can be a to improve wellbeing. It is unclear whether assess the effectiveness of use of public significant shift for organisations whose these change efforts were better implemented funds and to consider how it may be staff are constantly attending to the or more successful as a result of the use of targeted to maximise cost-effectiveness. hard work of service delivery. "It can be outcome measurement, although it is clear• Evaluative benefits (organisational de-motivating to always be travelling (i.e. that the use of baseline data was an important level): outcomes measurement assists focussed on delivery) and never arriving motivational tool in initiating change. organisations to understand whether (i.e. recognising that a goal has been There is clearly scope for further research on the outcomes of client outcome measurement itself. 3
  7. 7. Literature Review:Measurement of Client Outcomes in Homelessness Services2.2 diSBenefitS, The information produced may be of poor including its cost: "One needs to be cautiouscHallengeS and riSkS quality that the outcomes measurement systemPotential adverse aspects of outcomes does not consume in resources more than There may be a variety of reasons whymeasurement are grouped here under its information is worth." Hudson (1997: 78) outcomes measurement systems do notheadings that deal broadly with four categories makes a related point, noting the distortion produce useful information. Outcome measuresof impacts: that can arise if the resources put into (and measurement tools) are subject to• Resourcing impacts managing the performance of a program the same range of threats to reliability and• Staff impacts (including monitoring outcomes) exceed the validity as any other psychometric measures.• Impacts relating to the value of the resources actually provided for service delivery. Some potential limitations and sources of information produced, and bias or error that are particularly related to• Impacts on service delivery. Outcomes measurement can be difficult to the organisational context may include low sustain over time response rates (Hatry 1997: 18), administrationSee also Post, Isbell et al. (2005: 6-13). Anecdotal evidence (Clements 2010; Talbot of complex measures by staff with limited 2010) suggests that one of the greatest training or knowledge (Berman and HurtOutcomes measurement can be expensive challenges in outcomes measurement is 1997: 88), collector bias (especially whereMost authors agree that implementing sustaining the measurement systems over those responsible for ratings are the sameoutcomes measurement can be an expensive time. While appropriate resources and as those delivering the service – cf. Rossiand time-consuming process. Resources organisational focus may be provided initially, (1997: 31)), and use of ratings to achieveare required for researching and developing these may be impacted by the pressures of an instrumental purpose related to serviceoutcome statements, measures and tools; service delivery and the introduction of other delivery (e.g. to demonstrate client eligibilitytraining; developing or modifying data systems; organisational initiatives. Ongoing commitment for certain resources or services) rathercollecting and entering data; supporting staff and resourcing is required to ensure the than as an accurate reflection of the clientsand trouble-shooting; analysing and reporting systems function well. status (Hudson 1997: 77). It is also possibleon data; and ongoingly reviewing the outcomes that the selection of measures themselvessystem itself. Rapp and Poertner (1992: 107) Staff may feel threatened can be subject to dumbing down due to thecaution that managers typically underestimate It is possible that some service delivery political purposes to be served by the results.the level of resources needed for data staff may feel threatened by outcomes Segal (1997: 154-155) gives the example thatcollection and data entry, while Berman and measurement systems (Rapp and Poertner stakeholders may shy away from the use ofHurt (1997: 87) note that outcomes data 1992: 101). Staff can feel they are being hard outcomes measures as they are lesssystems are more likely to contribute valuable scrutinised. Where services are delivered to likely to show change than soft measures.information if they are adequately resourced, clients with complex needs in resource-poorand operated by trained staff. Implementing environments, there is a risk that managers Another key challenge related to informationoutcomes measurement superficially in an or frontline staff will feel they are being quality is the adequacy of electronic systemsattempt to minimise cost and avoid changes held accountable for things that are outside for storing outcomes data. Most homelessnessin organisational culture may actually be less of their control (Schalock 2001: 4, 39). It is organisations operate with a suite ofcost-effective – MacKeith (2007: 4) suggests essential that both those collecting and those inconsistent and only partially functional clientthat organisations that approach outcomes interpreting the data understand the range of data systems that are not designed to capturemeasurement in this way will not achieve the factors and constraints that affect outcomes, or analyse outcomes data or integrate it withbenefits of increased focus, motivation and including the fundamental observation that service delivery. Even with the best measures,effectiveness. outcomes are largely controlled by the client. if data systems are not available to process It is possible that introduction of outcomes the data this can lead to a breakdown in theSchalock (2001: 39) argues that despite the measurement may lead to the departure of measurement process (Hendrick 2010b).value of outcome measurement, it also needs a small number of staff who are not willingto be balanced with other considerations, to make practice changes (Smith, Rost et al. 1997: 132). 4
  8. 8. Literature Review:Measurement of Client Outcomes in Homelessness ServicesThe information produced may be used poorly staff morale implications of data indicating argue that it is important to hold serviceEven where the information produced by low levels of success (Segal 1997: 155). providers accountable for both outcomesoutcomes measurement is of high quality and Stakeholders may become dissatisfied with and process (quality assurance) – "Measuringrelevance, it is only beneficial to the extent the length of time required to achieve change one to the exclusion of the other can leadthat it is actually used to improve services for in outcomes; Wells and Johnson (2001: 195) to inappropriate practices that focus on theclients (Schalock 2001: 39). Berman and Hurt suggest that achieving change tends to take measure rather than the experience of the(1997: 87) comment that there is "no greater much longer than expected. child and family" (180). A related issue is theburden" than the collection of information potential for outcomes measurement to leadthat is not used. One danger is that the One possible response is to use risk to soft targeting or to services focusing oninformational needs of some stakeholders adjustment or casemix style approaches countable outcomes to the exclusion of less(often service delivery staff) will be ignored to balance outcomes against complexity and easily measured goals (Wells and Johnsonin outcome measurement systems (Hudson severity of presenting issues (e.g. Spellman 2001: 195).1997: 73, 76). A different problem is the use and Abbenante 2008: 41-43). However, this inof outcomes data to motivate conclusions or itself is complex and adjustment formulae tend Another potential detriment to service deliverydecisions without understanding the limitations to be controversial. is that outcome measurement results mayof the data. Booth and Smith (1997: 40) note become just another way of categorising orthat the end users of the results (often agency Adverse outcomes information can also have labelling clients, and that this can becomeor governmental decision makers) are not an effect in individual casework. Outcomes disempowering for clients. Pilon and Raginsnecessarily familiar with technical aspects of tools can show regression as well as progress (2007: 11) discuss an example of a mentalresearch or evaluation design. This can lead to (MacKeith, Graham et al. 2007: 13), and health clinic that had been rating clientsan inflated view of the reliability and validity of negative ratings by a case manager of a using the MORS recovery indicators and hadoutcomes data in demonstrating causal links clients progress, if known to the client, may begun using the milestones as a shorthandbetween programs and results. For this reason, impact on the worker-client relationship (Love way to describe consumers: “She’s a three.”among others, Wells and Johnson (2001: 193) and Scott 2010). Careful thought should be “He thinks he’s a 7, but he’s really a 5.” Theadvise caution in using outcome information to given to the way that outcomes are discussed authors express concern at the dehumanisationallocate resources. in these contexts. implied by this use of language, and suggest that attention to agency culture is importantThe information produced may reflect adversely Measurement may have adverse impacts on to ensuring that outcomes tools are used in aon services service delivery strengths-based perspective.Organisations typically hope that outcome The literature identifies two types of possiblemeasurement will demonstrate the success adverse consequences to service delivery of 2.3 Balancing BenefitS and riSkSof their work. However, as Rossi (1997: 24) outcomes measurement. One is a distortionnotes, program designers and operators often of the types of intervention provided by a Organisations considering introducinghave exaggerated expectations in terms of measurement focus on certain outcomes. outcomes measurement systems may find itthe outcomes of their services. Outcome For example, in a child protection setting, useful to weigh up the potential benefits andmeasurement may in fact suggest that a a measurement focus on length of time to risks of such systems. Although the list of risksprogram is neither effective nor efficient - family reunification (with shorted durations may appear daunting, the majority of thesewhether because the environment imposes understood as better) might lead to a pressure impacts can be mitigated through a well-severe constraints on what it is possible to on staff to return children to their families planned and well-executed implementationachieve, or because the intervention itself is while significant risks or instability still existed. process, including clear communicationmisguided. It is important that organisations Therefore, a counterbalancing measure processes. This speaks to the need forentering into outcome measurement are of return to foster care within a specified adequate resourcing for implementation,prepared to face this possibility, and have period of time should also be used (Wells and for timeframes that allow for meaningfula strategy for dealing with the political and and Johnson 2001: 191). Wells and Johnson consultation with a range of stakeholders. 5
  9. 9. Literature Review:Measurement of Client Outcomes in Homelessness ServicesA question of particular interest for communitysector organisations is whether the benefitsof outcomes measurement systems areworth the resources required. The answerto this question will depend on the contextand informational needs of each individualorganisation. However, it is useful to rememberthat:• Organisations can choose the scope of the outcomes that they choose to measure, and find a scope that represents best informational value for money• A staged approach to implementation can distribute the resource burden over time and also has other benefits, including enabling assessment of the usefulness of the initial stages prior to roll-out of subsequent parts of a package.Baulderstone and Talbot (2004: vii) weighed uppractical aspects of outcome measurement inSupported Accommodation Assistance Program(SAAP) services, concluding: “Of those thatparticipated in the pilot instrument trials,some reported positive experiences and somenegative. The project team found that wherethere was negative experience, the reasons forthis were identifiable and could be dealt with,and that there was no fundamental barrier tooutcome measurement implementation in mostinstances.” 6
  10. 10. Literature Review:Measurement of Client Outcomes in Homelessness Services3. Homelessness sector: approaches to defining and measuring client outcomes3.1 auStralia – national outcoMeS The NAHA includes population-level outcomes, outputs and performance measures; selected itemsand indicatorS are summarised in Table 1.The overall framework for tacklinghomelessness in Australia is established by Table 1: the National Affordable Housing Agreement National Affordable Housing Agreement: (NAHA) (Council of Australian Governments excerpts from outcomes, outputs and performance indicators2009a), which aims to ensure that allAustralians have access to affordable, safe Outcomes • people who are homeless or at risk of homelessness achieveand sustainable housing that contributes to sustainable housing and social inclusionsocial and economic participation. The NAHA • people are able to rent housing that meets their needsis supported by three National PartnershipAgreements: one on social housing, one Outputs • number of people who are homeless or at risk of homelessnesson Indigenous Australians living in remote who are assisted to secure and sustain their tenanciesareas, and most relevant to the current • number of people who are assisted to move from crisispaper, the National Partnership Agreement on accommodation or primary homelessness to sustainableHomelessness (NPAH) (Council of Australian accommodationGovernments 2009b). Performance indicators • proportion of low income households in rental stressThese government documents may be • proportion of Australians who are homelessrelevant to measurement of client outcomes in • proportion of people experiencing repeat periods of homelessnessindividual homelessness services in two ways:• They provide an indication of government policy focus in terms of homelessness, The NPAH is designed to contribute to the NAHA outcome People who are homeless or at risk of which is one source of guidance in terms homelessness achieve sustainable housing and social inclusion. Table 2 summarises the outcomes of the types of outcomes that services explicitly stated in the agreement; other outcomes implied in the agreements outputs; and may pursue, and how outcomes could be excerpts from the performance indicators specified in the agreement (the performance indicators framed for advocacy purposes. included here are those focused on service effectiveness rather than service quality or effort). The• They provide an indication of government performance indicators also have associated baselines and performance benchmarks: see Council interest in particular data items, and of Australian Governments (2009b: 7-8). therefore of potential future reporting requirements. While some of these are population outcomes (and therefore likely to be measured through census-type approaches rather than from agency data), others may need to be built into agency data collection. 7
  11. 11. Literature Review:Measurement of Client Outcomes in Homelessness ServicesTable 2: National Partnership Agreement on Homelessness: stated outcomes, implied outcomes and selected performance indicators Stated outcomes • Fewer people will become homeless and fewer of these will sleep rough; • Fewer people will become homeless more than once; • People at risk of or experiencing homelessness will maintain or improve connections with their families and communities, and maintain or improve their education, training or employment participation; and • People at risk of or experiencing homelessness will be supported by quality services, with improved access to sustainable housing Implied outcomes • Public and private tenants sustain their tenancies • Homeless people secure or maintain stable accommodation • Homeless people (including families) ‘stabilise their situation’ Performance indicators • Proportion of Australians who are homeless • Proportion of Australians who are experiencing primary homelessness (rough sleeping) • The number of families who maintain or secure safe and sustainable housing following family violence • Increase in the number of people exiting care and custodial settings into secure and affordable housing • Reduce the number of people exiting social housing and private rental into homelessness • The proportion of people experiencing repeat periods of homelessness • Number of young people (12 to 18 years) who are homeless or at risk of homelessness who are re-engaged with family, school and workThe Supported Accommodation Assistance Program (SAAP) has long maintained a data collectionsystem. This system does include some data items which provide an indication of statusmaintenance and change, pre- and post- engagement, in areas such as accommodation, mainsource of income, labour force status, student status and living situation of clients. However, manyof the changes achieved by clients are not reflected in SAAP data reporting (Baulderstone andTalbot 2004: 1, 5). 8
  12. 12. Literature Review:Measurement of Client Outcomes in Homelessness Services Youth refuges: Support Assessment FrameworkThe SAAP V Multilateral Agreement (Commonwealth of Australia, State of New South Wales etal. 2005: 36-40) included a set of performance indicators, many of which are concerned with A pilot is currently underway of an outcomesaccess, cost, and activities or outputs. Two of the current performance indicators and three of the measurement system for youth refuges inindicators "being considered for future implementation" can fairly be seen as outcomes indicators. Victoria. The measures, recording tool andThese are summarised in Table 3. implementation process are being developed and coordinated by Sally Elizabeth forTable 3: SAAP V Multilateral Agreement: selected performance indicators related to client outcomes DHS, with considerable input from sector stakeholders. The tool is to be piloted from Current indicators Indicators being considered for future implementation May 2010. 20. The extent to which 26. Number and proportion of clients at risk of homelessness The Support Assessment Framework (SAF) clients’ case management who were assisted by SAAP to maintain their accommodation is intended to meet a range of needs as an goals are achieved at case integrated assessment, planning, review, closure 27. Number and proportion of clients at risk of homelessness who communication and outcomes measurement were assisted by SAAP to maintain family links where their health tool. The draft tool covers a range of domains 37. Number and proportion and safety will not be endangered including income, living skills, health, mental of clients not returning to health, housing / homelessness and a series SAAP within 6 months 28. Number and proportion of people at risk of homelessness of others. For each domain, the tool has who were assisted by SAAP to obtain appropriate accommodation essentially a 5-point status maintenance and upon their exit from an institution change scale. The young person is given a rating on each scale at intake, and again at exit. Each domain also has space for brief3.2 victorian dePartMent of HuMan ServiceS qualitative comments in relation to assessmentThe Victorian Department of Human Services (DHS) has an interest in the development of of support required, support provided (actionoutcome measures for homelessness and other services. Prior to the 2009 re-organisation of taken), and any action in progress at pointthe former DHS into the current Departments of Health and Human Services, the Department of exit. The qualitative items are importanthad engaged Allen Consulting to develop a set of draft Departmental, Program and aspirational because they help to validate the ratings andoutcomes (aspirational outcomes being those requiring the input of other stakeholders, such to unpack the meaning of particular ratingsas other branches of government). Subsequent to the departmental re-organisation, these at analysis stage. Some basic demographicproposed outcomes are being revised to make them more suitable to the current focus of DHS. data is also included. The tool allows for anThe process is being driven by the DHS Central policy team. However, these outcomes are not yet index of the young persons overall situationpublicly available (Trinh 2010). There is also considerable work occurring within the Department (total of ratings at entry or exit), although thein relation to development of outputs and outcomes for the new Victorian Homelessness Strategy details of how this will be calculated are still(Homelessness 2020). Again the draft outcomes are not available for public comment at this stage, being finalised. The tool is designed to reflectalthough they will be based partly on consultation that has already occurred. the short stays (average 6 weeks) and interim outcomes expected in a youth refuge serviceProgram evaluations of homelessness services contracted by the Department also provide environment.an indication of the Departments focus of interest in terms of outcomes. Recent and currentevaluation projects (including e.g. those of Elizabeth St Common Ground Supportive Housing, SAF data for all Victorian youth refugeKensington Redevelopment, J2SI, YHAP2) have focused on housing stability, social inclusion clients will be collected by agencies in an(including participation in Employment, Education and Training), and physical and mental health. Excel spreadsheet template. The data willWhere children are involved (for example in evaluations of neighbourhood renewal programs), be deidentified, aggregated and analysed oneducational continuity, retention and attainment are also of interest (Trinh 2010). a quarterly basis by DHS. At this stage the estimate is that on average 1000 records of 9
  13. 13. Literature Review:Measurement of Client Outcomes in Homelessness Servicesdata per quarter will be gathered. In addition The main outcome measurement approaches 3.4 reSultS-BaSed accountaBilityto the sector-level statistics, it is intended trialled were Goal Attainment Scaling and Results-based accountability is a high-profilethat SAF will be useful at a variety of levels Standard Goal Scaling. Several services trialled approach to managing for results thatfrom individual client self-assessment, through level of functioning scales and the BT Generic originated in the United States. In its broadworker use in case planning and review, to Outcomes Scale. Other approaches and tools sense the term implies at a minimum thatsupervision and agency-level reflection on were considered (a set of specific Status “expected results are clearly articulated andoutputs and outcomes. In this way the data Maintenance and Change scales; pre-existing that data are regularly collected and reportedgathered may also be used by organisations to standardised scales), however these were not to determine whether results have beenfacilitate practice change. pursued further as consultation suggested they achieved” (Weiss 1997: 174). Results-based did not meet the needs of the stakeholders accountability has a particular emphasis on3.3 SaaP PilotS: BaulderStone and or provide a good fit with the service delivery using outcomes definitions and measurementstalBot (2004) context (Baulderstone and Talbot 2004: 9-10). to focus service provision and to leverage collaboration among human services agenciesAs noted above, SAAP performance Mission Australia and a broad range of partners who have themeasurement has primarily concentrated on During and following the Baulderstone and potential to impact on a problem.equity and efficiency rather than on client Talbot study, a range of Mission Australiaoutcomes. However, Baulderstone and Talbot services were involved in outcomes Friedman’s approach(2004) conducted and reported on a pilot measurement pilots (Clements 2010; One of the best known proponents ofproject that tested the applicability of a variety Talbot 2010). All Mission Australia NSW/ results-based accountability is Mark Friedman,of outcome measurement tools across a range ACT Community Services were expected to who along with collaborators has developed aof SAAP service types. The intention was to complete an outcomes measurement project in particular approach to applying results-baseddevelop a system of outcome measures that 2004/5; services in other parts of Australia also accountability (Friedman, DeLapp et al. 2001b;would be acceptable to SAAP agencies and that participated in pilots. A Tool Kit was developed Friedman 2005). Results are understood aswould allow aggregation of outcomes data to to support these projects (Mission Australia “population conditions of wellbeing for children,state and national levels. The project examined 2005). Routine outcomes measurement was adults, families and communities”; results databoth housing and non-housing outcomes. also seen as feeding into formal service provides information about whether the efforts evaluation processes and reflection on practice being made to achieve results are succeedingThe project found that outcomes measurement (Mission Australia 2006: 11). (Friedman 2005: 11-13).is useful at a number of levels (client and workeruse in case management; service management; The measurement approaches and tools The core principles of results-basedprogram planning). However, the authors promoted through the Tool Kit were Goal accountability are (Friedman, DeLapp et al.concluded that due to the great diversity in Attainment Scaling, Standard Goal Scaling and 2001b: 1.1):SAAP service types and service delivery across the BT Generic Outcomes Scale. The Tool Kit 1. Start with ends, work backward to means.Australia, no one outcomes tool was applicable includes an extensive list of goals related to What do we want? How will we recogniseto all SAAP services (or even to all services SAAP support areas but tailored to Mission it? What will it take to get there?within a service type) (2004: 37). Given this, Australia’s services. 2. Be clear and disciplined about language.the authors suggested that the needs of client 3. Use plain language, not exclusionaryservice workers and managers need to be given For further research: insights of Mission jargon.priority in decisions on which forms of outcome Australia from their experience of the outcome 4. Keep accountability for populationsmeasurement are undertaken. Another key measurement pilots and subsequent work in separate from accountability for programsfinding was that staff skills and training were performance measurement and agencies.critical to the use of outcomes tools and the a. Results are end conditions of wellbeingcollection of reliable and valid data. for populations: children, adults, families and communities. 10
  14. 14. Literature Review:Measurement of Client Outcomes in Homelessness Services Melbourne Citymission: MORF 3.5 outcoMeS Star b. Customer or client results are end conditions of wellbeing for customers of a Melbourne Citymission (MCm) began The Outcomes Star approach was developed program or agency. introducing an outcomes measurement in the United Kingdom. The first version of the5. Use data (indicators and performance system in 2008. The organisation had already star tool was developed by Triangle Consulting measures) to gauge success or failure identified results-based accountability as the in 2003 for St Mungo’s, a London-based against a baseline. framework within which it wanted to develop homelessness agency (MacKeith, Burns et6. Use data to drive a disciplined business-like its outcomes systems. Initially, Friedman’s al. 2008a: 7). Subsequently the star tool has decision making process to get better. approach was piloted with seven services evolved into a suite of related tools designed7. Involve a broad set of partners. from MCm’s Homelessness and Children & for use in different sectors: the Homelessness8. Get from talk to action as quickly as Disability Services portfolios. The pilot was Star (MacKeith, Burns et al. 2008b), the Mental possible highly successful in engaging staff and led to Health Recovery Star (MacKeith and Burns a decision to roll out the Measuring Outcomes 2008), the Alcohol Star (Burns and MacKeithOne of the key concepts of results-based and Results Framework (MORF) across the 2009a), and the Work Star (Burns andaccountability is that of baselines and organisation from mid-2009 (Hendrick 2010a). MacKeith 2009b).of “turning the curve”. A baseline is arepresentation of the current state of affairs - During the pilot MCm identified the need to All of the tools are based on a commonboth a historical picture (“where we’ve been”), augment Friedman’s framework in several ways approach which includes:and a forecast (“where we’re heading if we (Hendrick 2010b): • An explicit ‘journey of change’ modeldon’t do something different”) (Friedman 2005: • Friedman’s work tends to assume that the with ten steps (ranging from “stuck” to28). The expression ‘turning the curve’ is a way aims and objectives of services are already “self-reliance”)of describing the desired change in a particular clear and can provide a starting point for • A set of life domains, which vary from toolcondition with comparison to the baseline. defining client results – in fact, this is not to tool but with substantial similaritiesTurning the curve involves doing better than always the case. The process of discussing • A single measure in each domain,the baseline forecast. One of the strengths of outcomes with services needs to include a reflecting a global measure of the person’sFriedman’s approach is that it can be strongly stage of clarifying program aims “relationship” with that domain, includingmotivating in bringing people together to • Processes for collecting, storing and how well they are managing needs oridentify which curves are the highest priority to analysing data need to be defined. Due to problems in that domainturn, and then in working in partnership to “beat the complexity and diversity of the data • A common visual presentation for thethe baseline” on these curves (Cunningham- systems in operation within MCm (as within measurement tool and supportingSmith 2010). most large human services organisations), information, using a star-shaped this is a major challenge. arrangement of axes representing differentFriedman’s approach involves clear processes domains, and a ladder representing stagesfor “getting from talk to action”, and has a clear MCm continue to work on many areas of the of changeframework for identifying the most important implementation of MORF, including working • The intention that the tool be integratedtypes of performance measures that may be with individual services on defining desired with casework processes and that ratingsused to monitor progress (Friedman 2005: 72). outcomes, standardising outcome statements be jointly agreed through discussionHowever, it does not dictate any particular set within portfolios, and developing data systems. between client and worker.of outcome measures and for this reason could At this point, the expectation is that a set ofbe flexibly combined with other theoretical standardised core outcomes will be defined The various versions of the outcomes starand practical approaches to outcomes for each portfolio; services within the portfolio have been piloted and are currently beingmeasurement. For example, it would be will be expected to work towards one or more used in a range of services, mainly in the U.K.possible for an organisation to monitor a range of these core outcomes, but may also have A web-based electronic data system has alsoof outcomes, but to use Friedman’s processes additional service-specific outcomes that are of been developed to capture and analyse dataduring planning, drawing on the monitoring data importance in their context (Hendrick 2010b). produced from the star tool.to drive a focus on particular results areas. 11
  15. 15. Literature Review:Measurement of Client Outcomes in Homelessness Services 3.7 BenefitS and cHallengeS of aMajor strengths of the outcomes star context of the problem or need that the Sector-BaSed aPProacHinclude its thorough piloting process program is designed to address, and theand appropriateness to the context of activities and outputs of the service One of the questions that hovers aroundhomelessness services. Potential limitations • A clear distinction between outputs and outcomes measurement in the homeless sectorare its focus on ‘soft’ rather than ‘hard’ outcomes is the extent to which it is possible to have anoutcomes (for example, it would not provide • Analysis of outcomes within three distinct integrated approach to outcome measurementa count of status change in respect to clients’ time frames (short-term, medium-term and across the sector. There are potential benefitshousing situations), and the limited depth of long-term) to an integrated approach, specifically:measurement in any one domain. However, • A clear definition of the target population • The ability to aggregate data to servicethere may be potential to complement the for a particular desired outcome (certain system levels and to reflect on the‘soft’ measures included in the tool with goals may only apply to a subpopulation effectiveness of the service system as aadditional measures that help to fill out the such as persons with disabilities, or to wholepicture of change for individuals. those who have achieved earlier goals) • The ability to compare the performance • A specific formula for calculating an actual of services and agencies, and to therebyMicah Projects, Brisbane outcome: the number of persons who gain an insight into factors influencingMicah Projects in Brisbane began piloting the achieved the desired goal, divided by the effectivenessHomelessness Star in 2009 (Stevens 2009). total number of persons in the target • Minimising data collection impositions onInitially the tool was trialled with one small population (i.e. percentage of population individual agencies and clients.team, with positive reactions from staff. achieving the goal)Training in the use of the tool was provided • Setting meaningful outcome targets. However, there are also complexitiesfor most of Micah’s staff in the second half of associated with implementing a sector-wide2009. Micah also intended to trial adapting the The NAEH approach is designed for use approach; these are briefly discussed here.tool for use with homeless parents, adding an by funders and communities as well as byadditional domain around parenting. service delivery organisations; it includes Developing common outcomes and measures suggested methods for comparing outcomes It is unclear to what extent desired outcomesFor further research: benefits and challenges across services (including risk adjustment are common to all stakeholders (services,of implementation of tool at Micah Projects approaches), and for examining population- funders, clients) within the homelessness level changes. sector, and to what extent they differ across3.6 national alliance to end service types and client groups. BaulderstoneHoMeleSSneSS A strength of the NAEH approach is that it and Talbot (2004: 37) found that no one provides clear mechanisms for aggregation outcomes tool was applicable to all SAAPThe United States-based National Alliance to and comparison of results; a weakness is services, due to the diversity of serviceEnd Homelessness (NAEH) strongly support that measuring outcomes solely in terms of type and contexts across Australia. Evenoutcomes measurement and have produced a percentage of target population achieving a within a single organisation there can beToolkit to support government and community goal can miss progress that clients make in tensions between standardisation versusorganisations in measuring effectiveness other areas, and can obscure differences in customisation of tools and measures. Theseoutcomes (Spellman and Abbenante 2008). the degree of change on particular measures. can reflect tensions between the needs of theThe approach is not prescriptive and does not In addition, setting targets and comparing organisation as a whole and the needs of itsprovide a particular measurement tool. outcomes across services are often highly sub-units. political processes, which may have unintendedSome features of the NAEH approach include side-effects. Putting aside the tools themselves, it is unclear(Spellman and Abbenante 2008; Barr 2009): to what extent different expressions of desired• Use of a program logic model to place outcomes reflect differences in language, as desired and actual outcomes within the opposed to differences in conception of service 12
  16. 16. Literature Review:Measurement of Client Outcomes in Homelessness Servicesgoals. While there are certainly philosophical wastage of resources within the service system. While this proposal has merit, one of theand political implications to the way outcomes difficulties with it is that the robustness ofare described, practice wisdom also suggests Ideally, it would be possible for outcomes generalisations about interventions in thethat at a practical level, service providers data to be collected by services who are best homelessness field tend to be lower thanwould be able to find areas of agreement placed to obtain the information (i.e. services those in the health field, owing to the typesabout what constitute desirable as opposed to who have the expertise, the resources and of research designs that can reasonably beundesirable outcomes. One of the problems the appropriate quality of relationship with pursued in homelessness settings.with such a process, however, may be that the client), and shared appropriately with‘consensual’ definitions of desired outcomes other services who are working in partnershipmay be high-level and generic, making with that client. This would fit with joint casethem difficult to operationalise. Another planning and review processes (Hamiltonproblem is that services may have different 2010), and would also potentially allowunderstandings of the process of change, responsibility for outcome measurement toleading to conflicting ways of describing the be shared between services. For example, asteps involved in progress (MacKeith 2007: 5). clinical mental health service might collect one set of measures while a Psychiatric DisabilityA separate but related question is the Rehabilitation and Support provider wouldfeasibility of collecting particular core data collect a different set.items across all homelessness services, orservice types (therefore providing a set of Exchange of information in such a systemcommon core outcome measures). The more would need to be governed by the informedof a ‘stretch’ it is for service providers to collect consent of the client. Clear processes fordata, the lower the reliability of the data is seeking and recording this consent would needlikely to be. to be developed.Exchange of outcomes information Burden of proofOne of the problems that can arise as more MacKeith (2007: 5-6) differentiates the useservices begin to collect outcomes data is of outcome measurement to evaluate thethe duplication of data collection by multiple effectiveness of service provision modelsservices. This particularly impacts on clients or interventions from its use by individuallinked to a range of support services – for agencies to monitor their success. She arguesexample, a client may have an outreach support that the “burden of proof” currently falls onworker, a clinical case manager, be linked to a each individual service to demonstrate thatcommunity health centre, use a day program their intervention type is worth funding.and receive occasional assistance from a crisis Instead, she proposes following the healthaccommodation provider. If each of these service model of clinical trials to assess theservices is gathering outcomes data, with no effectiveness of an intervention. Once the trialscoordination between services, the client may are complete, the value of the intervention isbe ‘bombarded’ by measurement requests established and there is no requirement forusing the same or different instruments (Love individual services to prove the value of theand Scott 2010). In addition to the annoyance intervention. Instead, agencies can focus oncaused to the client, this may well undermine monitoring their own success compared withthe reliability of the data collected. The established benchmarks.duplication of data collection also represents 13
  17. 17. Literature Review:Measurement of Client Outcomes in Homelessness Services4. Measurement of client outcomes as a research activityClient outcomes measurement systems are selected criteria (Rossi, Freeman et al. 1999: monitor. Evaluations are typically customisedusually intended as practical performance 20-21). Some commentators distinguish to the context of individual services and themeasurement and program management tools, evaluation from research on a range of time-specific needs of particular stakeholders.rather than as vehicles for theoretical research. considerations including the purpose of the Evaluations can explore how and why certainNevertheless, they do involve gathering and study, the relationships of stakeholders to effects occur and is suitable for exploringanalysing data to create new knowledge or the study, and the types of standards used to issues such as long-term impacts, causalinsight, and therefore can be viewed as a form judge the adequacy of the study (Fitzpatrick, attribution, cost-effectiveness and comparativeof research. Sanders et al. 2004: 6-7). Within the field effectiveness (McDavid and Hawthorn 2006: of program evaluation, a wide spectrum of 293; Segone 2008: 101-103).It is useful to consider outcomes measurement approaches can be used (e.g. Wadsworthsystems through the lens of research design 1997: 77-109). The borderline between evaluation andfor several reasons: monitoring is not clearly defined and outlying• To identify choices about the types of Relationship between monitoring and evaluation forms of each approach may resemble outcomes data that are desired, and how Measurement of client outcomes can be the other. A detailed and sophisticated these may be gathered and analysed considered as an evaluative activity (Kimberley monitoring system that includes a wide range• To identify ways in which measurement 2009) – that is, an activity that is designed of performance measures and supplements systems can be designed so as to provide to provide information that may assist in this with a significant amount of qualitative the most reliable and valid data assessing the value of an intervention or data may in effect be an ongoing evaluation• To identify the types of knowledge or program. However, there can be major process. However, for most homelessness insight that outcomes measurement differences in the way that this evaluative organisations such a system would be systems can typically produce, and the activity takes place in an organisational unaffordable. On the other hand, an level of certainty of the conclusions drawn. context. The key distinction that is drawn effectiveness-focused evaluation with limited This may include understanding the within the literature is between monitoring of resources and a quantitative methodology limitations on the types of inferences that client outcomes (as part of an organisational may resemble a time-limited performance can reasonably be supported by the data. performance measurement system) and measurement system. evaluation of client outcomes as part of aOutcomes measurement also has a relationship formal program evaluation. Some sources see both routine outcomeswith program evaluation. This section monitoring and in depth ad hoc evaluationconsiders these relationships and explores Monitoring typically involves systematic, as subtypes of program evaluation (e.g.the types of findings typically produced by periodic collection and analysis of data to Hatry 1997: 3-4). Some have argued thatoutcomes measurement systems. assess performance in relation to an agreed performance measurement can fulfil many of standard set of indicators. Monitoring the same purposes that program evaluations4.1 Monitoring and evaluation systems are usually designed to be ongoing have served, and that in-depth program rather than time-limited. Monitoring provides evaluation is therefore an “expensive luxury”.In general terms, evaluation refers to “the succinct, regular feedback that can assist However, there are important differencesidentification, clarification, and application of with accountability, quality improvement between the typical forms that monitoringdefensible criteria to determine an evaluation and responding to evolving trends in the systems and program evaluations take, andobject’s value (worth or merit) in relation to environment. the level of information that they are able tothose criteria” (Fitzpatrick, Sanders et al. 2004: provide (McDavid and Hawthorn 2006: 293).5). Program evaluation typically involves the Evaluation involves episodic, in-depth collection The term evaluation is used in this paper forapplication of social research procedures to and analysis of information. Evaluation can in-depth, episodic program evaluations insystematically develop a valid description of draw on a broad range of data sources and contrast to ongoing monitoring systems.particular aspects of program performance, methods, and can examine factors thatand the comparison of the performance to are too costly or difficult to continuously 14
  18. 18. Literature Review:Measurement of Client Outcomes in Homelessness Services 4.2 naturaliStic and exPeriMentalIt is worth noting that the term monitoring evaluation is also used in the literature, however it aPProacHeS to reSearcHtypically refers to process-oriented program evaluations that focus on how the program is beingdelivered, rather than on outcomes (Fitzpatrick, Sanders et al. 2004: 21). There is a long-standing and sometimes fierce debate in the social sciences (and in scienceAdvantages and disadvantages more generally) about which approachesHatry (1997: 4) provides a useful summary of the relative advantages and disadvantages of to research are the best for studying andevaluation and monitoring, which is adapted as Table 4 below. Monitoring is used to track understanding the world. This “paradigmsperformance against a limited set of measures, and for early identification of trends in the external debate” is linked to philosophical positionsor internal environment. Evaluation is used to guide significant organisational decisions or gain a about the nature of reality, truth, and of whatdeeper understanding of phenomena identified through monitoring. constitutes credible evidence (Patton 1987: 165).Table 4: Advantages and disadvantages of monitoring compared to evaluation (adapted from Hatry (1997: 4) Table 1-1) Experimental approaches to research are typically associated with the positivist tradition, Type Advantages Disadvantages which emerged from the physical sciences. These approaches tend to aim for objectivity, In depth, ad hoc evaluation • Identifies causes of • High cost emphasise separation of the researcher from outcomes (to some • Covers few services that being researched, and assume value- extent) • May take extended time to neutrality. They often rely on and valorise • Can provide relatively get results quantitative data (Patton 1987; Wadsworth strong evidence on 1997: 165). Within the evaluation literature outcomes and effect there are significant critiques of this paradigm, • Can provide guidance as which argue that subjective and objective to improvement action meanings are socially constructed, are central to understanding human phenomena, and that Regular outcomes • Covers many or most • Provides little information all research is value-driven (Wadsworth 1997: measurement agency programs on causes of outcomes 101). • Provides information on a • Provides little guidance on regular basis improvement actions In contrast, naturalistic approaches to research • Lower cost per program • Subject to a variety of are typically associated with phenomenological covered interpretations and constructivist philosophical positions. • Hints at improvement These approaches do not attempt to actions artificially manipulate the environment or the phenomenon being studied, and typically proceed inductively from practice to theory.For greatest benefit, organisations use a combination of ongoing monitoring and episodic Naturalistic approaches aim to describe andevaluation. Data from monitoring can contribute to evaluations, by providing historical data understand naturally unfolding processes,streams (Hatry 1997: 4). Program evaluations provide a much richer understanding of context and including variations in experience, and tendof the factors that may be impacting on client outcomes. to emphasise qualitative data (Patton 1987: 13-15; Wadsworth 1997: 95-96). NaturalisticEvaluation designs may also influence monitoring systems – for example, an evaluation might approaches have been criticised at times forcollect measures on particular domains, which are then partially carried on by the organisation a perceived lack of objectivity (Patton 1987:in routine outcomes monitoring. Evaluations may also identify particular areas of strength or 166).weakness which an organisation may wish to monitor in an ongoing way. 15
  19. 19. Literature Review:Measurement of Client Outcomes in Homelessness ServicesAt times these approaches and their possible intended and unintended outcomes caused change are inferred. Instead theassociated methods have been characterised are being explored. The Most Significant comparisons depend on individual oras fundamentally opposed and irreconcilable. Change approach (Davies and Dart 2005) is groups for which equivalence either is notHowever, more recently authors have argued particularly relevant to this context. However, established, or is known to be absent.that both approaches are useful and that qualitative approaches will be less appropriate • Single-system designs are those in whichthe key challenge is to match appropriate where the informational need is a broad the comparisons are for one individual,approaches to particular research or evaluation understanding of overall patterns of outcomes group or collectivity, at different pointsquestions and issues (e.g. Patton 1987: 169). across a large group of people. in time (at a minimum, before and duringMost evaluations use several designs or application of the treatment).combinations of designs to address different Given the focus of the outcomes measurementquestions. Mixed method designs that combine literature, the bulk of this literature review It is often not feasible to use true experimentalqualitative and quantitative approaches often focuses on quantitative methodologies for designs in human services evaluations, for ayield richer insight and can increase validity monitoring outcomes of large groups of variety of reasons. In particular, there may be(Fitzpatrick, Sanders et al. 2004: 263, 305). people. ethical issues and risks related to withholding treatment from a ‘control’ group, althoughThe large majority of literature on client For further research: explore possible ways in in certain situations where demand outstripsoutcomes measurement assumes a which qualitative and naturalistic approaches supply a control group may be possible (Cookquantitative methodology, and by association could be used as the basis of an outcomes and Campbell 1979: 347-350; 374). In aimports much of the focus of the positivist measurement system monitoring system, with limited resources, ittradition on reliability and validity. A few is very unlikely that an experimental designsources do acknowledge the usefulness 4.3 Monitoring SySteMS aS would be attempted.of qualitative data (e.g. Burns and Cupitt Single-SySteM reSearcH deSignS2003), although generally as an adjunct to Classic outcomes measurement systemsquantitative measures. This overwhelming The literature distinguishes ‘classical’ are most closely aligned with single-systemfocus on quantitative methods may reflect experimental designs from quasi-experimental research designs. They share the definingthe informational needs that tend to drive designs (including single system designs) (Cook characteristic of single system models: themonitoring systems (e.g. senior management and Campbell 1979: 4-6; Bloom, Fischer et al. planned comparison of a noninterventiondesire for oversight of results for an entire 2006: 44-49): (“baseline”) period with observations ofcohort of clients), as well as the genesis of • Experimental designs involve at least intervention period(s) or in some cases, aoutcomes monitoring in governmental and two groups of participants (a ‘treatment’ post-intervention period, for a single client orbusiness performance measurement systems group and a ‘control’ group), with system (Bloom, Fischer et al. 2006: 322).(McDavid and Hawthorn 2006: 282-288). different treatment provided to the two groups. Random selection and random Within the area of single system designs,Qualitative methods clearly have the assignment are used to attempt to obtain again there are a wide variety of differentpotential to offer considerable insight into equivalence (on average) between the designs (Bloom, Fischer et al. 2006: 352).client outcomes, especially in programs in two groups on all relevant characteristics. The basic single-system design is A-B (wherewhich outcomes are expected to be highly Data is aggregated within these groups A represents the baseline phase and B theindividualised rather than standardised. and analysis of comparisons between intervention phase). More complex designsQualitative methods have strengths in the average scores of the groups are involve replication of original conditions and/capturing diversity and subtlety of experience undertaken. or treatment periods (e.g. A-B-A-B designs);and of outcomes (Patton 1987: 24-30). ‘Open • Quasi-experimental designs depart from others may involve successive or alternatinginquiry’ approaches may be of particular value the experimental model in that they do interventions (e.g. A-B-A-C or A-B1-B2-B3in the early stages of developing outcomes not use random assignment to create designs, where the subscripts representmeasurement systems, when the range of the comparisons from which treatment- varying intensity of the same intervention). 16

×