Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

2012SFF - Made to measure

524 views

Published on

Workshop on different impact measurement tools

  • Be the first to comment

  • Be the first to like this

2012SFF - Made to measure

  1. 1. Theory of changeImpact Reporting and Investment StandardsGlobal Impact Investing Ratings SystemSocial Return on InvestmentDemonstrating ValueRandomized Control Trials
  2. 2. Measuring Social Value Dr. Tessa Hebb Measuring Up, Social Finance Forum Centre for Impact Investing, MaRS, Toronto November 8th 2012
  3. 3. What is Blended Value“The Blended Value Proposition states is that all organizations, whether for-profit or not, create value that consists of economic, social and environmental value components—and that investors (whether market-rate, charitable or some mix of the two) simultaneously generate all three forms of value through providing capital to organizations.” Jed Emerson
  4. 4. Metrics and Measurement Blended value requires measurement of all three forms of value economic, social and environmental. “Measurement should be viewed as a process whereby the greatest value is achieved through organizations building up and learning from data and evidence over time.” (Measuring the Value of Corporate Philanthropy).”
  5. 5. What are social metrics? Social metrics are measurement tools that can be used to define and articulate social value, social outcomes and the results generated by investment and activities in the social sector.
  6. 6. You manage what you measure“A review of measurement methodologies did not turn up a “silver bullet” or single numeric against which performance can be universally gauged. Rather, this reading reinforced the notion that, to an extent, measurement is its own reward. It encourages improvement, management, and the explicit formulation of assumptions and expectations.” (Measuring the Value of Corporate Philanthropy)
  7. 7. Start with your mission1. What are the results for which you will hold yourselves accountable?2. How will you achieve them?3. What will they really cost?4. How do you build the organization you need to deliver these results?
  8. 8. What to Measure (Measuring the Value of CorporatePhilanthropy). • Links among the mission, programs, and measures must be clearly defined and articulated in order to narrow the number of required indicators. • The measures should be easily collectible and communicable. • The measures should be strategically designed and applicable across the organization at all levels, while also encouraging of operating units to focus on high- level strategies. • Above all, the measures must address progress toward the mission and illustrate whether and how the organization’s actions make a difference.
  9. 9. Inputs, Activities, Outcomes, andImpactsInputs, activities, outputs, outcomes, and impacts over theshort, medium and long term are the building blocks ofsocial metrics.The building blocks are linked to the organizations missionthrough its theory of change.“Activities,” such as the number of staff trained or amountof goods purchased, and “outputs,” such as the number ofclients served, products distributed, and areas reached maybe the extent of measurement for short-term, one-offprojects. Sometimes simply identifying activities andmeasuring output may be all that is feasible.
  10. 10. Theory of Change• “Built around the pathway of change, a Theory of Change describes the types of interventions (a single program or a comprehensive community initiative) that bring about the outcomes depicted in the pathway of a change map. Each outcome in the pathway of change is tied to an intervention, revealing the often complex web of activity that is required to bring about change.” (TheoryofChange.org)
  11. 11. Logic ModelInput/Resources Activities Outputs Outcomes Impact IndicatorWhat are the resources What are the What is the evidence of What is the change that What will happen over How would youthat are needed to activities/products that service delivered to the will happen (short term the long term measure the outcome?accomplish the will generate the intended audience at or medium term) to the How will you know ifactivities? outputs? the intended dose? target group or change happened? (what others will be individuals? able to see, touch, count)Stakeholder 1
  12. 12. Conclusion• Blended value requires economic, social and environmental values be measured.• Social metrics help organizations understand their strengths.• Social metrics link mission, activities, outputs, outcomes and impacts.• No one size fits all.
  13. 13. Contact Dr. Tessa Hebb tessa_hebb@carleton.ca Director,Carleton Centre for Community Innovation www.carleton.ca/3ci
  14. 14. Impact Map
  15. 15. Theory of changeImpact Reporting and Investment StandardsGlobal Impact Investing Ratings SystemSocial Return on InvestmentDemonstrating ValueRandomized Control Trials
  16. 16. Inputs Activities Outputs Outcomes Impact IRIS GIIRS SROI Demonstrating Value Randomized Control Trials
  17. 17. The Impact Reporting and Investment Standards (IRIS) is a catalogue of metrics that can be used to describe an organization’s social, environmental, and financial performance. Pg 18
  18. 18. IRIS is… My Metrics Your Metrics Full Time Employees = Full Time Employees GHG Emissions Reduced = GHG Emissions Reduced
  19. 19. IRIS Framework 20 http://iris.thegiin.org
  20. 20. IRIS Framework 21
  21. 21. Inputs Activities Outputs Outcomes Impact IRIS GIIRS SROI Demonstrating Value Randomized Control Trials
  22. 22. Global Impact Investing Rating System Environment 377 B Corporations I $1.82B Revenues I 54 Industries I $1M Annual Savings Community Workers Governance Pg 23
  23. 23. Pg 24
  24. 24. B Impact Assessment 25
  25. 25. Inputs Activities Outputs Outcomes Impact IRIS GIIRS SROI Demonstrating Value Randomized Control Trials
  26. 26. What is SROI?Social Return on Investment is a framework formeasuring and accounting for the value createdor destroyed by our activities – where theconcept of value is much broader than that whichcan be captured by market prices.
  27. 27. val·ue/ˈvalyo͞o/nounthe regard that something is held todeserve; the importance, worth, orusefulness of somethingthe material or monetary worth ofsomething

  28. 28. Return on Investment (Net Profit)ROI = Cost of Investment 29
  29. 29. ) 9 3+% 9 31"D) * :, ") % * ("D2 2 +3">3$("$9 % ./ +") .3(">) * ./ "<$(.2 2 %<$.3" W/ 3+"./ 3", 29 9 3(3+% +"(3T 9 3+, 3(1"$+, "% () +2 ") 9 3+, 3(1"2 3"2 ) 9 / % 9 1"2 3"- 04"<() ($5 E "./ $."% () +2 ") 9 3+, 3(1"D) * :, "@ +"./ "$+, / % 9 3" 9 .) (3, "2 "./ 3"7XGY ) ("./ 3"- ) * ./ "0* 1.2 3"4) 5 5 2 $% +.) "9 % ..331E 3" "./2 3+.22 "@ 3"A 1.2 3"1>1.35 "@ 3"3+, ") 9 , 93, >"./ * % >"./ ">3$("SB 2""W/ :3" 1) %$:"=$:* 3" 3+3($.3, "./ () * / "./ 3"$<<() $% "$+, ") + ) 2 " 2 / +./ 3(3"5 $>"@ , 2 ) +$:E +T ) 2 "% 1.1"$11) %$.3, "D2 "./ 2 3"$, .2 ") + ) 2 ./ 1" 1* % 311"($.3") 9 * ./ "0* 1.2 3"% 5 5 2 % ">) % ) ..331" () D1"9 5 """"""""""" () SROI example: Calgary Youth Justice () * <") 9 ) 9 3+, 3(1E 3"7XGY ) ("./ 3"- ) * ./ "0* 1.2 3" "(3T 9 "./ "9 % ! "" B H"+"./ 3"9(1.">3$(".) "! "" KB S" +"$"./ (33">3$("@ 1B 6I I 2 2 6! ! ) $12 """ C :3" " $@ H6 7 ) %$ :"Q$ :* 3 "4 $ :% :$. 2 +6"C/ (33 "- 3 $ (E K! "- ) * + "G993 +, 3 (1E KK"(3T 993+, 3(1E 2 * ) "^ "! ) "SK"% () +2 ") 993 +, 3 (1 / % Y , 2 $. ) (" + % C) . $ :"Q$ :* 3 " 7 XGY +, 2 $ . ) ( 1"Y :* , 3 , 6 "Y % +% - X"H - X"S L ) . 31 4) , 3 - X"! ! c) :2 3"% % $::1"Z 2 3, [ $=) , 47c VK V` J E KK I V! OE O V` HI "<3("% O` $::"9) ("! KK"%$::1 c) :2 3"2 % +=31.2 $.2 +1" ) H 08 VK V! O^ E 8 S! V` ! E KOK V! O! H"<3("2 +=31.2 $.2 + ) Z 2 3, [ $=) , c) :2 3"$..3+, $+% % 3"$."% * (." ) S 0O VK VSHE OH! V! KE OH ! VS! O"<3("$..3+, $+%3 Z 2 3, [ $=) , - ) * ./ "4) * (."c() %311" J 0! V8OSE ^ I ^ V! S! E 8K I VJ KE S^ ^ V! E I "<3(">) * + ") 993+, 3(", 2 H^ =3(.3, ".) "- 041 Z 2 3, [" $=) , c() @ ) +"1* <3(=2 ("Z ." $.2 1) +) I 0H VK V! HSE K 8J VS8E ! SJ V! E HKK"<3(">3$(B "VHI B / ("9) ("c() @ ) +"G992 3( 8` $.2 % (3F* 2 [(3, 4) 1.") 9"2 $(% +% 3($.2 +" ) ` 0^ VK VO! E K HI VH8E ! HI VHI K , $>"e"S` I "l "VO! E K"Z ">) * ./ ") +:>[ HI ! Z 2 3, [" $=) ,; ++* $ :"7 ) %$ :"Q$ :* 3 "4 ( 3 $ . 3 , ". / ( ) * / "- 04 16 2 V8 OSE ^ I ^ V` J ! E KO J V! O8 E 8 K 4$:% :$.3, ".) "(3:93% +9:$.2 +"m"SB B I * ."2 ) HP "c ( 3 13 +. "=$ :* 3 ") 9"1) %$ :"=$ :* 3 "% 3 $ . 3 , 6"Z =3 ( "S" 2 ( ) V! E J ! E H! ` J Y +.3(31."($.3"SB P I>3 $ ( 1[6C) . $ :"2+=3 1. 5 3 + . "2 "- 3 $ ( "! 6 + V! ` HE KKK7 XGY 3 $ ( "! 6 "- V! 6"VIH" H IB B I IST>3 $ ( "7 XGY6 V! ! KB KB S 6"V!! S"!47c"l "42 :3"1) * (% +"<() % "% ((3+.:>"9 5 "(3:2 :3"1) * (% """ .$@ 3"2 311E * () $@ 3BY .$& $" 5 5 * +2." 31" 4) .>" 21"$..(2 * .3, ".) "./ 3"2 :=35 3+.") 9 $+>"% 5 5 * +2 @ +=) "5 ) .>"<$(.+3(1E " 2+=31.) (1"$+, " ) =3(+5 3+."$ 3+%31") 9 3(2 "<() ($5 1"$+, "" 2 9 +4) 5 5 * +2 ) :* +.33(1"$(3"./ 3"/ 3$(.") 9 2 .>"Q "./ 1"<() ($5 B 3+" "W/ ) <<) (.* +2 31"9 (">) * ./ E <<) (.2 "<() ($5 ") <3($.2 +1E " .2 ) "1* + ) "$+,
  30. 30. 7 Principles of SROI • Involve stakeholders • Understand what changes • Value the things that matter • Only include what is material • Do not overclaim • Be transparent • Verify the result
  31. 31. Inputs Activities Outputs Outcomes Impact IRIS GIIRS SROI Demonstrating Value Randomized Control Trials
  32. 32. Capacity building ‘Snapshot’ Capacity report to use informatio nSupport for CapacityMonitoring to collect Systems informati ClearDevelopme on picture of Information nt informatio Blueprint n needs
  33. 33. Performance Snapshot example
  34. 34. www.demonstratingvalue.org The workbook, snapshot gallery, blog, newsletter and more Twitter @demvalue
  35. 35. Rating/Benchmarking systemLibrary/Catalogue of indicators Framework Toolbox Framework Set of principles Study design/methodology Randomized Control Trials
  36. 36. The use of RCTs for Social ImpactAssessment: An ExamplePresentation to the 2012 Social Finance Forum,November 8 and 9, 2012
  37. 37. Challenges for Social Impact Assessment WHAT outcomes to measure – that will best align with your program goal WHICH methods to use – that will best isolate program effects and allow the attribution of these efects to your program HOW to quantify and monetize these effects – to demonstrate financial viability39
  38. 38. Demonstrating Impact The capacity to measure the difference a new approach or program makes. To measure the true impact of a new program, we need to know what would have happened if the program had not been introduced.........we need a counterfactual: • to account for natural maturation processes, e.g., children skills are constantly improving • to account for factors external to the program, e.g., state of the economy fluctuates and influences labour market outcomes • to account for “regression to the mean” phenomenon, e.g., lone mothers on welfare eventually go back to work
  39. 39. Displaced workers re-integrating labour market asa result of being offered an earnings supplement 60%Full-Time Employment Rate 50% 40% 30% Program Group Supplement Group 20% 10% 0% 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Month From Random Assignment
  40. 40. Displaced workers being offered supplementversus those not receiving the same offer 60%Full-Time Employment Rate 50% 40% 30% Program Group Supplement Group 20% Control Group Control Group 10% 0% 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Month From Random Assignment
  41. 41. Why Randomized Experiments? To measure the true impact of a new program, we need to know what would have happened if the program had not been introduced.........we need a counterfactual: Random assignment of participants to a treatment group and a control group provides the best method to create a counterfactual • Not the only method available • Not always possible to do RA
  42. 42. How Random Assignment Works Recruitment of participants A random sample of individuals from the population that isNote: Both program and targeted for the program intervention is recruited and interviewedcontrol group memberscontinue to have access to Informed Consentgovernment programs and Potential participants sign an informed consent agreeing to be partservices available to of the experiment and provide information for research purposesmembers of theircommunity. Random Assignment Volunteers are assigned at random to either the program group or the control group Program Group Control Group Eligible to receive program Ineligible intervention Informed of eligibility for the program intervention and the conditions Informed of ineligibility status attached to recipiency Do not meet conditions for Not eligible for program intervention, Meet conditions for recipiency recipiency, therefore, do not but continue to be part of the study receive program intervention for research purposes Receive program intervention
  43. 43. Properties of RCTs / Social Experiments With a large enough sample, random assignment insures that the two groups of individuals are identical, on average. • This is true for all observable and unobservable characteristics (such as motivation, self-confidence, determination, and all other personal attributes that can explain why an intervention will succeed with one individual but not another) Unlike nonexperimental methods, properly implement social experiments are guaranteed to provide internally valid impact estimates – no selection bias. Nonexperimental methods may be equally reliable in any given application; we simply cannot know a priori that they are reliable, as we can with experimental methods – you can only match participants on measured characteristics.
  44. 44. The Community Employment Innovation ProjectA research and demonstration project testing the value ofcommunity-based employment as an alternative to incometransfers in areas of chronic high unemploymentTwo parallel but related studies • Individuals: • Aims to preserve employability, through faster re-employment • Provides opportunities for skill development and strengthening of social capital • Communities: • Study of a model which utilizes strengths of local communities to create jobs • Aims to support their capacity growth and improve the social economy
  45. 45. The Program ModelThe Offer to Individuals • EI and IA recipients were offered 3 years of full-time employment, on locally developed projects in exchange for their entitlements to EI or IA • Employment was designed to replicate full-time market jobs • 35 hours per week, at $325 a week, EI/CPP insurable, 15 days annual leave, medical benefits • Support Services: some job-readiness and transferable skills trainingThe Offer to Communities • 6 communities in the Cape Breton Regional Municipality were offered a free workforce of 750 workers for up to five years • Each community was required to elect a representative board, develop a strategic plan, and approve projects • Local control given to communities – explicitly links projects to local needs
  46. 46. Participants Impact StudyKey outcomes of interest • Economic well-being • Employment, earnings, transfer receipt, income, poverty, and hardship • Longer-term employability: Skills and experience • Characteristics of post-CEIP employment, employability skills, attitudes to work • Social Capital, volunteering, life satisfaction
  47. 47. CEIP impacts on EI sample 100 Program GroupA 53 95 Control Grouppercentage 90 Impactspoint 85impact at 80 75peak 70 Percentage Employed Full Time 65 60 55 50 45 40 35 30 25 20 No significant 15 10 impacts a year 5 after end of 0 CEIP eligibility -5 -10 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 -15 -20 Months From Random Assignment
  48. 48. IA Sample: Large in-program impacts on employment, but not sustained 100 Program GroupNearly 80 95 Control Group 90 Impactspercentage 85point impact 80at peak 75 70 65 Percentage Employed Full Time 60 55 50 45 40 35 30 25 20 15 No significant 10 impacts a year 5 after end of 0 CEIP eligibility -5 -10 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 -15 -20 Months From Random Assignment
  49. 49. IA Sample: Permanent reductions in IA receipt three years after CEIP comes to an end 100 Program Group Control Group Impact 80 60 Percentage Receiving IA 40 20 0 A sustained 1 11 21 31 41 51 61 71 12 percentage point reductionA stable -20 in IA receipt42 percentagepoint reduction -40in IA receiptduring program -60 Months From Random Assignment
  50. 50. Impacts on Social Capital:Sustained increase in the size of networks Network Size Change in number of contacts from baseline to 54 months 2.5 2.0 54-month gain 1.5 1.0 0.5 0.0 Help with chores Emotional Specialized Help finding a job support advice Program Control
  51. 51. Community Engagement:Sustained increase in volunteering and social contact Volunteering is important to both individuals and communities Can be an avenue to skill development, improves social inclusion, and is a large resource for many community organizations 54-Month: Impacts on Formal volunteering with groups or organizations 50 45 40 35 Percent 30 Program 25 20 Control 15 10 5 0 Volunteered in the past 12 Volunteered in the past 12 months months EI Sample IA Sample
  52. 52. CEIP Theory of Change:Analytical framework of expected change Individual Community Response Organizational Early Engagement Organizing, Planning, Mobilization MobilizationMobilization Years 1-3 Process Process effects CEIP Projects effects Provide work experience and valued community Services ProjectDevelopment and Interim Effects Years 4-5 Skill Gains, Social Social Capital Inclusion, Cohesion Well Being and Community Improves Post-program Capacity Improves Long-run Effects Years 6-7 Employment Levels, Market Conditions Improves
  53. 53. Targeted Community Sectors The two largest project categories were similar across communities Youth projects were most prevalent in New Waterford and Whitney Pier Seniors projects were at greatest scale in Sydney Mines Figure ES.5: Percentage of Work Years Assigned, by Community and Sector Served 100% 90% Health, Environment, Percentage of total work years assigned to a sector Beautification 80% Recreation, Arts and Culture 70% Services for the Poor, 60% Unemployed 50% Supports for Seniors 40% Supports for Youth 30% Other: Services for 20% the Disabled, CED Agencies, CEIP Boards 10% 0% Glace Bay NewWaterford North Sydney Sydney Mines Whitney Pier Community
  54. 54. Results: Theory Supports ObservationPreponderance of positive change in more successful communities New Sydney Whitney North Dominion Glace Bay Waterford Mines Pier Sydney Social Capital + + + + + Social Cohesion + + + + Social Inclusion + + + - + Third Sector Relative Size + Organizational Capacity + + + + + Economic Outcomes Social Conditions + + + Youth Effects + + + + - Senior Effects + + + + + Effects on the Poor + + + +
  55. 55. 57
  56. 56. 58
  57. 57. Cost-Benefit Analysis Results (IA Sample)Net benefit-cost per IA program group member over the full 54-month follow-up Accounting PerspectiveComponent of Analysis Individuals Communities Government SocietyMonetized components Participant Impacts CEIP earnings 34 344 0 -34 344 0 Foregone non-CEIP earnings -10 974 0 0 -10 974 Transfer payments (EI & IA) -11 836 0 11 836 0 Tax payments (taxes and premiums) -3 559 0 2 921 -638 Other household member earnings 2 035 0 0 2 035 Third Sector Organizational Effects Value from CEIP jobs (to sponsors) 0 20 024 0 20 024 Volunteering (CEIP induced) 0 2 404 0 2 404 CEIP administrative costs 0 0 -4 274 -4 274 Admin costs of EI & IA transfers 0 0 471 471Net Benefit/Cost per Program Group Member 10 010 22 428 -23 390 9 048
  58. 58. Benefit-Cost Ratio CEIP is very cost effective when one considers the combined benefits to individuals and community -- $1.39 in net benefits per dollar for IA recipients Total net benefit for every dollar that government spent on CEIP 1.60 1.39 1.40 EI Sample IA Sample Benefit-Cost Ratio 1.20 Positive Net Present Value 1.02 0.96 1.00 0.81 0.80 0.60 0.43 0.40 0.21 0.20 0.00 Participants Communities Society Perspective
  59. 59. Cost-Benefit Analysis Accounting framework Accounting PerspectiveComponent of Analysis Individuals Communities Government SocietyNon-Monetized components Participant Impacts Reductions in Hardship, Stress + 0 0 + Improved Social Capital + 0 0 + Increased Trust in Networks + 0 0 + Foregone Leisure – 0 0 – Community Effects Increased Social Capital of Residents 0 + 0 + Improved Community Cohesion 0 + 0 + Increased Social and Civic Participation 0 + 0 + Foregone Leisure 0 – 0 –
  60. 60. 62
  61. 61. Valuing Intangibles – ExamplesSocial Capital – each additional contact is valued at 7% of income;therefore CEIP impact is worth $3,808 per participantTrust – each additional percentage point increase in trust is valued at2.5% of income; CEIP impact is worth $2,401 per participantHardship – the reductions in stress associated with lower hardshipduring CEIP is valued at $3,379
  62. 62. Benefit-Cost Ratio Including the intangible impacts improves the benefit cost ratio to $1.61 in net benefits per dollar spent About a 50 percent improvement in overall net benefit to society Total net benefit for every dollar that government spent on CEIP 1.80 1.61 1.60 EI Sample IA Sample 1.40 Benefit-Cost Ratio 1.20 1.20 Positive Net Present Value 0.96 1.00 0.81 0.80 0.65 0.60 0.40 0.40 0.20 0.00 Participants Communities Society Perspective
  63. 63. Take aways To find out what difference a program makes, one needs to find an appropriate counterfactual. RCTs provide the best approach to set up a counterfactual and derive an Impact. An Impact measure is required to do a proper Cost-Benefit Analysis; and a sound CBA is needed to report on the financial viability of a program Analytical frameworks and Cost-Benefit Analysis should and can incorporate social benefits Short of convincing private sector investors or government authorities to transform their accounting or fiscal framework to include environmental and social benefits, attempts should be made to monetize these benefits.
  64. 64. WWW.SRDC.ORG67
  65. 65. Randomized Control Trials

×