The aim of our presentation is to explore evaluation approaches to determining outcomes in monetary values, with a particular emphasis on Social Return On Investment, a relatively new approach to cost benefit analysis. In the first part of the presentation, Helen will give a brief overview of the broader spectrum of cost benefit analysis models, and explain why we felt that Social Return on Investment is an evaluation approach that every evaluator should have in their tool box. Fran will outline the essence of SROI, including its purpose, principles and design. Cynthia will then demonstrate its application to evaluation practice by presenting a case study in which SROI was used to answer questions of value and impact in an evaluation of an agricultural and environmental aid programs in Africa. We finish with an exercise for you and our on line colleagues that we hope will enhance our collective appreciation of the value of this tool for evaluators.In acknowledgement of Amy’s edict to be creative in this assignment, our choice of communication tool for the first part is an Ignite presentation. Ignite presentations are the equivalent of a power point peptide or caffeine hit. It is a five minute presentation with 20 slides in which the slides automatically turn over every 15 minutes. So could you please hold any questions til the end of this part of the presentation. You won’t have to wait long!
There are many evaluation theorists and practitioners who put the conversion of benefits into dollar figures in the too hard or too dodgy basket. Julian King, when he presented to our class in session 3 was clear that many intangibles can’t or shouldn’t be valued in dollars: Zappala asserts that ‘many NFPs are falling prey to a ‘Wall Street Syndrome’ when it comes to measuring program effectiveness by applying complicated, time consuming and expensive measurement approaches to straightforward programs (Zapala: 2011) – yes there is risk and I will have more to say on that later, but the real danger is not so much that evaluators seek to monetise the value of non financial program outcomes but that in practicethe framework of evaluation is generally conceived far too narrowly. (Shergold:2012)
The question of how evaluators should value programs that cost funders money but create financial value AND at the same time build social benefit is in fact a noble quest, but where is our fellowship of evaluation cost-benefit theory ? Where is the fierce and comprehensive debate by evaluation theorists and academics on this topic? Even more simply, where is the extensive and considered analysis of the merit of using a SROI approach? Our search of literature found very little critical research on the technological and methodological aspects of SROI, and of what we found, hardly any came from evaluation theorists.
Attributing cost to intangible benefits is not tilting at windmills. Remember, governments and funders appear to ‘lose money’ on most programs in a financial sense, so there is a critical imperative for evaluators to get on with this, and do it well. It is about staying true to the essential evaluation tasks of assessing value and merit (Davidson:2005)
Judgment about the value of a program is an explicit feature of almost every impact or outcome evaluation. Regardless of whether the evaluation specifications have a requirement to do cost benefit analysis, as an evaluator, you won’t be able to reliably ascertain impact if you ignore the elephant in the room - the evaluation cost and benefit questions of economic impact, value and merit.
This slide says it all. Evaluation commissioners often feel short changed when evaluation of efficiency and effectiveness does not extend to consideration of cost effectiveness or cost benefit analysis.. Cartoon sourced from http://www.econosseur.com/economic-jokes.html
You can interpret this cartoon (sourced from http://www.econosseur.com/economic-jokes.html) in many ways, but to me its intention is to highlight how your ability to explain the link between a specific action and its outcome has long been a fundamental requirement for gaining support.
Judgment on the merit of a program or intervention means understanding cost and benefit related to the program. The evaluator will need skills and tools to capture cost and identify benefit. This is where, in general, the literature becomes very thin. If you are an evaluator looking for the right tools to allow you to provide ‘explicitly evaluative conclusions’ (Davidson:2005) and identify and place a value on all relevant outcomes , there has actually been very little development of the cost benefit models beyond those developed by and for economists. Fran and Cynthia will explain more on how SROI captures cost
Cost benefit approaches vary significantly, but in essence all attempt to define in monetary terms the relationship between cost and benefit for a given initiative. Of the two aspects of cost benefit analysis, cost is usually considered to be more easily measured particularly for program outcomes rather than products. But there are risks if only costs with an agreed market value, the diamonds, are measured. An evaluator needs to move beyond ‘lip service’ in defining benefits for the purpose of establishing thefull costs of investment in the program.
There are many forms and models for evaluating cost and benefit. Economists have donated these. Can you see any that were developed by evaluation theorists? This enormous gap in evaluation theory Cost benefit analysis often refers to determining cost and benefit to the whole of society; cost effectiveness (insert definitions for each term in the slide)
As you can see Galileo is the godfather of SROI. He was not only a renaissance man, he was a SROI man. Unfortunately none of his drawings on cost benefit models survived so the task of working out how you ‘make measurable what is not so’ has been left to contemporary evaluation theorists and practitioners. In a way, SROI attempts to pick up where Galileo left off
One reason we wanted to explore Social Return on Investment was because of its orientation towards measuring value from the bottom up and according to the perspective of different stakeholders. If you did the on line survey you will have some idea of how stakeholders might be engaged in defining, and putting a value on, outcomes.
Brief outline of why SROI is a significant tool for evaluators who use participatory approaches and/or qualitative methods, and its close relationship to program logic development.
A frequent dihlemma for an evaluator is that no market value exists for measurement of a program’s most significant outcomes. A defining aspect of SROI is that it provides a framework for measuring and accounting for both tangible and intangible values, and uses monetary values to represent these values. This is used to prepare ratios of benefits to costs, again expressed in dollar figures.
Hand over to Fran to explain what happens in Step 2 – the ‘miracle’ of SROI
Economist joke to finish A man walking along a road in the countryside comes across a shepherd and a huge flock of sheep. He tells the shepherd, "I will bet you $100 against one of your sheep that I can tell you the exact number in this flock." The shepherd thinks it over. It's a large flock, so he accepts the bet. "There are 973 sheep," says the man. The shepherd is astonished, because the man is exactly right. "O.K., I'm a man of my word. Take one." The man picks one up and begins to walk away. "Wait," cries the shepherd. "Let me have a chance to get even. Double or nothing that I can guess your occupation." The man agrees. "You are an economist for a government think tank," says the shepherd. "Amazing!" responds the man. "You are exactly right! But tell me, how did you deduce that?" "Well," says the shepherd, "Put down my dog and I'll tell you.”Joke sourced from http://www.econosseur.com/economic-jokes.html
What IS SROI?Brief history and contextPurposeSeven PrinciplesDifferences with CBA
Please note that all names and figures in this example are false. Also this case study is exclusively for the RARE class of 2013 only and must not be shared beyond this class. These are the conditions from the organisation to use this case study as the study is not yet finalised and one to the public.
Four steps involved in this stage
SROI presentation PPT
SOCIAL RETURN ONINVESTMENT(SROI)Helen CaseyCynthia MulengaFrancisca Reutter
SROI - Just another attempt toturn social programs intocommercial transactions?
“Dear Mr Ghandi,We regret we cannotfund your projectbecause the linkbetween spinningcloth and the fall ofthe British empirewas not clear to us”
Presentation outline1. What is SROI?2. Contexts3. How is SROI Implemented• Stages4. Uses (inappropriate)5. Weaknesses6. Strengths
1. What is SROI?• Approach to program, project and policy evaluation thataims to account for non-financial outcomes using monetaryvalues to represent them.• A way of reporting on value creation measuringsocial, environmental and economic results.• Includes a consistent approach with standard steps.• Strong emphasis on involving stakeholders.(Nicholls, Lawlor, Neitzert, & Goodspeed, 2012)
SROI measures the value of social benefits created byan organisation, in relation to the relative cost ofachieving those benefits, expressed in a SROI ratio:SROI ratio = present valuevalue of inputs(Rotheroe & Richards, 2007)
Two types of SROIEvaluative• Conductedretrospectively• Based on outcomes thathave already takenplace.• Preferred use in ongoingevaluation and not as afinal outcome measure.Forecast• Conducted before hand.• Predicts how much socialvalue will be created if theactivities meet theirintended outcomes(Nicholls, Lawlor, Neitzert, & Goodspeed, 2012)
Brief history and context• In 1997, REDF (Roberts Enterprise Development Fund, USA) launched aninitiative to asses impact of non for profits. In 2000 SROI was firstdocumented.• A Network of practitioners was formed in 2006: SROI Network (UK and USA).• New Economics Foundation in the UK edited a DIY Guide to Social Return oninvestment in 2007.• Office for the Third Sector (UK) developed a Measuring Social Value projectin 2008, aiming to develop SROI.(Flockhart, 2005)(Lingane & Olsen, 2004)(Arvidson, Lyon, Mc Kay & Moro, 2010)
PurposeIts fundamental purpose is to provide a model for allocatingmonetary expression of the value of outcomes for which noagreed market value exists.It can de used for a range of evaluation purposes:• Assess projects. (Forecast)• Demonstrate achievements (Evaluative).• Help improve organisational operations.(Nicholls, Lawlor, Neitzert, & Goodspeed, 2012)
7 Principles of SROIInvolve stakeholders.Understand what changes.Value the things that matter.Only include what is material.Do not over-claim.Be transparent.Verify the result.
How is it different to Cost Benefit Analysis?SROI• Used by managers to inform thepractical decision-makingoptimizing their social andenvironmental impacts.• Strong explicit emphasis onstakeholders and the types ofinvolvement they can have.• Comparison is not recommended,unless certain precautions aretaken.CBA• Used by funders outside anorganization to determine whethertheir investment or grant iseconomically efficient.• Does not necessarily includestakeholders.• Aimed at comparison.• (Arvidson, Lyon, Mc Kay & Moro, 2010)
2. Context• Designed originally to be used among NGOs and not for profits(“Third Sector”).• Growing interest in social value measures in the contexts ofincreased outsourcing of the delivery of public services, and theincreased need of funders to secure real value for money.(Wood & Leighton. 2010)• Can be used by:• Private businesses.• Non for profit and social organizations.• Government departments (Public Service Commissioners)• Funders.(Nicholls, Lawlor, Neitzert, & Goodspeed, 2012)
3. How is SROI ImplementedReporting, using and embeddingCalculating the SROIEstablishing impactEvidencing outcomes and giving them a valueMapping outcomesEstablishing scope and identifying key stakeholders
4. Inappropriate usesIt is NOT appropriate to compare the socialreturn on investment ratios alone.Also NOT appropriate when:• A strategic planning process has already been undertaken and is alreadybeing implemented and there is no chance of modifying;• Stakeholders are not interested in the results;• It is being undertaken only to prove the value of a service and there is noopportunity for changing the way things are done as a result of theanalysis;• Resources are scarce.
5. Weaknesses• Social Impact can be a personal or political measurement. (Lingane & Olsen.2004)• Needs considerable resources to be implemented. (Flockhart, 2005)• SROI “readiness” mainly involves being able to identify and measureorganisational outcomes adequately in a quantitative way. (Wood &Leighton, 2010)• Can easily be misused focusing solely on SROI Ratio (Arvidson, Lyon, Mc Kay& Moro, 2010)• The “if it can not be measured it can not be managed” trap.(Arvidson, Lyon, Mc Kay & Moro, 2010)• Quantifying inputs can be very tricky. (Arvidson, Lyon, Mc Kay & Moro, 2010)
“ An SROI analysis is only as good as the data thatis put in. In addition to properly resourcingorganisations to collect outcomes data, SROIanalyses can be strengthened by shared researchon outcomes, proxies, and indicators”(New Economics foundation, 2008. in Wood, Leighton. 2010. p 28)
6. Strengths• Fosters a commitment towards transparency and accountability (Rotheroe &Richards, 2007)• Promotes better communication and engagement between different stakeholders• Expected to foster improvement of quality data• Evaluative process promoted by SROI includes making organisations aware oftheir own values(Arvidson, Lyon, Mc Kay & Moro, 2010)• SROI principles have widespread approval, provide a benchmark fororganizations to set their goals and review their activities. ( Wood &Leighton, 2010)• Method includes specific guidelines that refer both to technical aspects as socialinteraction / political aspects, allowing relatively consistent procedures.
6 steps to implement SROIReporting, using and embeddingCalculating the SROIEstablishing impactEvidencing outcomes and giving them a valueMapping outcomesEstablishing scope and identifying key stakeholders
Project Objectives• Project goal: To improve the livelihoods of the people ofKalomo area.• Project Outcomes: Farmers adopt sound naturalresource management practices
Project interventions1. Community mobilisation around FMNR2. Intensively training of community memberson FMNR practices3. Promotion of complementary naturalresource management (NRM) techniques4. Strengthening of community structures
Rationale• 1. To find out whatproject outcomesimpacted on keystakeholders
2. what theseimpacts areworth to thekeyStakeholders
All as a way ofinterpreting theproject‟s value as aresult of theinvestment made
Data collection methods•Focus Group discussions of primarystakeholders•Key Informant interviews of selected•Household survey stakeholders•Visual data of geographical area•Shadow pricing
SROI stage 1: Scope and stakeholder•Measurement over the 3 yrimplementation period•Primary stakeholders – Leadfarmers•Neighbouring farmer households•Comparison group
• Stakeholders validated and identified the followingoutcomes1. Increased household and communal assets inthe form of trees and livestock2. Increased household consumables sourced from naturalresources3. Increased household income4. Improved health5. Psychological Benefits – increased hope, aesthetics6. Economic assets7. Environmental BenefitsSROI stage 2: Map Outcomes
SROI stage 3:Evidencing and Valuing Outcomes• Step 1: Develop outcome indicators:• Remember outcome: Increased household and communalassets in the form of trees and livestock. Indicator is• Nº of Households reported increased availability ofand accessibility to the resources (rafters forre/construction, firewood for cooking, thatch forroofing, and herbal medicines for basic treatment)• Amount of trees in the area
• Step 2: Collect outcomes data• How many experienced this change? 52 Households• How many trees in the area had regenerated? 1 000 000• Step 3: Establish how long the outcomes last• 6 years (2 years of project + 4 years post-project)• Step 4: Put a value on the outcome.• Market value of rafters, firewood, thatch, and herbal medicines (notfor trade)• Communities thought about how they used to collect it before and therisks involved• They valued this particular outcome at $100,000
• Impact is only what is a result of theintervention• DeadweightWhat would have happened anyway?• 9% of those not accessing FMNR said access to more wildresources had increased• Displacement• 0% displacementSROI stage 4: Establishing Impact
• AttributionHow much of the outcome is because of otherorganisations or interventions.10% - One community was already partly organisedaround tree protection• Drop off0% drop off - Community commitment unlikely to drop inthe short period of analysis due to the extent ofbenefits, therefore trees will continue to be there oreven increase in number
To calculate the impact of this outcome:= (Financial proxy x qty of outcome) minus dead weightminus attribution(100,000 x 52) - 9%5,200,000 – 468,0004,732,000 - 10%4,732,000 - 473,200= $4,258,800 in that year
• Five steps involved:1. Projecting into the future – drop off rate.2. Calculating the net present value – discount rate (timevalue of money)3. Calculating the SROI ratio = Present value/value ofinputs4. Sensitivity analysis – Which assumptions have thegreatest effect on your model?5. Payback period- At what point does return value >investment.SROI stage 5: Calculating the SROI
• Communicate meaningfully• Short Report• Transparent and concise• ConsistentSROI stage 6:Reporting Using and Embedding
Challenges• SROI methodology is silent on whether to define „value‟ interms of money funds‟ origins or the recipient community. E.g. a benefit in Choolwe, worth $500, is the equivalent of 50% ofthe average per capita income in Australia.Should it be expressed like that? Or should it be expressed as theequivalent in the financier economy?• SROI literature is weak on providing guidance on how tofacilitate stakeholder identification of meaningful valuesfor non-marketable benefits.
Challenges (continued)• Evaluators found it difficult and time-consuming to explainto stakeholders groups the notion of proxy financial valuesfor social, environmental and cultural returns.• Interviews and focus groups took a lot longer or coveredfewer topics in the allotted time due to the culturaldisconnect of trying to elicit proxy market/financial valuesfrom people who have an almost entirely non-economicculture, livelihood and value system.
References• Arvidon, M., Lyon, F., McKay, S., & Moro, D. (2010). The ambitions and challenges of SROI . UK: Third Sector Research Centre, University of Birmingham.Retrieved from http://www.tsrc.ac.uk/LinkClick.aspx?fileticket=QwHhaC%2br88Y%3d&tabid=500• Davidson, J. (2005) Evaluation Methodology Basics – The nuts and bolts of sound evaluation Thousand Oaks California: Sage Publications• Flockhart, A. (2005). Raising the profile of social enterprises: The use of social return on investment (SROI) and investment ready tools (IRT) to bridge thefinancial credibility gap. Social Enterprise Journal, 1(1), 29.• Lingane, A., & Olsen, S. (2004). Guidelines for social return on investment. California Management Review, 46(3), 116-135.• London Business School, New Economics Foundation and Small Business Foundation (2004). Measuring social impact: the foundation of social return oninvestment (SROI). Retrieved from http://sroi.london.edu/Measuring-Social-Impact.pdf• New Economics Foundation (2008) Investing for Social Value: Measuring social return on investment for the Adventure Capital Fund. London, UK: NEF.•• Nicholls, J., Lawlor, E., Neitzert, E., & Goodspeed, T. (2012). A guide to social return on investment (2nd ed.). UK: The SROI Network. Retrieved fromhttp://www.thesroinetwork.org/publications/doc_details/241-a-guide-to-social-return-on-investment-2012• Rotheroe, N., & Richards, A. (2007). Social return on investment and social enterprise: Transparent accountability for sustainable development. SocialEnterprise Journal, 3(1), 31.• SROI Network. (2011). The seven principles of SROI The SROI Network. Retrieved from http://www.thesroinetwork.org/publications/doc_details/140-the-seven-principles-of-sroi• Shergold, P., (2012) The Social Return on Universities Retrieved from http://www.onlineopinion.com.au/view.asp?article=13605• Wood, C., & Leighton, D. (2010). Measuring social value. London, UK: Demos• World Vision Australia. (2012). Social Return on Investment. unpublished manuscript• Zappala, G. (2011). CSI Briefing Paper no. 5. Solving social problems & demonstrating impact. A tale of two typologies. Centre for Social Impact, Universityof New South Wales, Sydney, Australia. Retrieved on 20/04/13 from http://www.csi.edu.au/assets/assetdoc/145e2b8d68c4a0b6/CSI_Briefing_5_Paper_-_Solving_Social_Problems_and_Demonstrating_Impact_2011.pdf