Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Evaluation and philanthropy: lessons from the third sector / Lisa Jordan

541 views

Published on

Published in: Education, Health & Medicine
  • Be the first to comment

  • Be the first to like this

Evaluation and philanthropy: lessons from the third sector / Lisa Jordan

  1. 1. Evaluation and Philanthropy Lessons from the Third Sector Lisa Jordan Executive Director Bernard van Leer Foundation
  2. 2. Mission StatementOur mission is to improve opportunities for children up toage 8 who are growing up in socially and economicallydifficult circumstances. We see this both as a valuable endin itself and as a long-term means to promoting morecohesive, considerate and creative societies with equalopportunities and rights for all.
  3. 3. Strategic Goals 2010-2015v Taking quality early learning to scalev Improving young childrens physical environmentsv Reducing violence in young childrens lives
  4. 4. AGEND Av The Bernard van Leer Foundationv Our Impact Assessmentv Common myths about evaluationv Lessons from the third sector
  5. 5. Philanthropic Tools for Greater1. Impact Development Knowledge o research, documentation, evaluation2. Service product development and delivery o investments, grants3. Capacity Enhancement and Skills Development o training, technical assistance4. Behaviour Change Programmes o campaigns, awareness5. Policy Development & Implementation Click to edit Master subtitle style o community organising, legal empowerment, lobby6. Enabling Systems & Infrastructure o networks, markets
  6. 6. ISRAELUniversal access to quality pre- Reduced incidence of Reduced rates of malnutritionschool among 3 to 6 year old depression, anxiety and among young Bedouin childrenchildren. aggression among young Jewish growing up in unhealthy physical and Arab children exposed to environments in the Negev. political violence.The National Ministry of Education, Parents and children experience Improved knowledge about dietarylocal municipalities and members of reduced levels of stress from and infrastructure causes of childthe business sector have increased exposure to political violence illness and malnutritiontheir financial investments inpreschools for childrenThe ratio of pre-school to supervisorThe Israeli government has an Increased access to clean water,has decreased appropriate public system for improved waste management and Click to edit Master subtitle style prevention and treatment of electricity depression, anxiety and aggression among young childrensupervisors are better equipped to Improved access to transportimprove pre-school quality among Bedouin communities, especially women and childrenThe quality of teaching and learningin pre-schools has improvedThe curriculum and reflects the
  7. 7. Pilot health Reduced rates of malnutrition among young Bedouin children education growing up in unhealthy physical environments in the Negev programm Lobby es for nurse Improved child health (esp. s reduction in digestive tract Improved access to illness) mother-child wellbeing clinics Better access to Improved waste Healthy dietary clean water management practices to combat Regional vitamin deficiencies and Improved housing Electricitymunicipal conditions (hygienic, ability to regulate Increased awareness amongadvocacy temperature) parents about dietary and (water, Government provision infrastructure causes of childsanitation, Bedouin of infrastructure in Increased malnutritionelectricity) families Bedouin towns and incomes invest in own villages among infrastructure Bedouin Increased mobility (e.g. families Government transport) among planners Legal recognition Bedouin communities Jointmunicipal Click to edit Master subtitle style and of villages More Bedouin (esp. women) Bedouins women join the and agree on a workforcecommunity planning child-friendlyResearch plan and Bedouin communities are well organized, have morecommunic positive public image and are supported by professional ations planners about PE and child Organizinhealth link g for transport Private sector transport options
  8. 8. What are the most cost- effective interventions to improve Bedouin children’s health and nutrition? Health and nutrition outcomes for children Individual evaluations combined with meta- analysis How are our strategies working to influence investment and service delivery in the Negev region for Review and Bedouin children?recommendations by Budgets advisory team withknowledge of broader Evaluation of cluster of Effective coalitions (Arab, political context grants from advocacy Jewish, government, perspective private sector, parents) Click to edit Master subtitle style Access to basic services How are our efforts to empower Health and nutrition Bedouin communities in the outcomes planning process working to influence investment and service delivery? Case study of selected municipalities
  9. 9. Main goals of the philanthropic activity The Foundation cannot solve directly any social  Test and validate  problems. Its mission aims therefore at testing  innovative  (especially with projects directly managed) innovative  policies solutions to social problems and at disseminating  successful solutions (“what works”).  Solutions to certain problems are well known and  Reward best  organisations implementing related initiatives are  practices numerous: in such cases the Foundation selects and  funds the best projects through specific calls for  proposals. Aimed at supporting (with institutional grants)  Support worthy  deserving nonprofit organisations (operating in the  institutions sectors of Arts & Culture, Environment, Scientific  Research, Social Services) prevailingly based in  Lombardy (Cariplo Foundation’s traditional intervention  territory)25/05/2011 Strategic Unit for  9
  10. 10. Overview: evaluation taskEvaluation works between Strategic Planning and Management, providing the whole structure of the foundation with guidance and lessons learned from philanthropic activities. Strategic  Evaluation Management Planning25/05/2011 Strategic Unit for  10
  11. 11. Main purposes of evaluation Test and validate  innovative  policies Reward best  practices Support worthy  institutions Critical  Knowledge  Accountability analysis/  sharing Learning25/05/2011 Strategic Unit for  11
  12. 12. What are the most cost- effective interventions to improve Bedouin children’s health and nutrition? Health and nutrition outcomes for children Individual evaluations combined with meta- analysis How are our strategies working to influence investment and service delivery in the Negev region for Review and Bedouin children?recommendations by Budgets advisory team withknowledge of broader Evaluation of cluster of Effective coalitions (Arab, political context grants from advocacy Jewish, government, perspective private sector, parents) Click to edit Master subtitle style Access to basic services How are our efforts to empower Health and nutrition Bedouin communities in the outcomes planning process working to influence investment and service delivery? Case study of selected municipalities
  13. 13. Tools: FIT FOR PURPOSEhttp://trasi.foundationcenter.org
  14. 14. AGEND Av The Bernard van Leer Foundationv Our Impact Assessmentv Common myths about evaluationv Lessons from the third sector
  15. 15. EFC MembershipSurvey 0 2 4 6 8 10 12
  16. 16. Of the three techniques listed, which one do you think was most common?1. Log frames2. Participatory Action Research3. Outcomes Mapping
  17. 17. Answer: Outcomes Mapping
  18. 18. Which problem do you think was most commonly reported?1. Defining & agreeing on the purpose of the evaluation2. Finding capable evaluators3. Making decisions based on evaluation results
  19. 19. Making decisions Percentage Reporting Problem 0 2 4 6 8 10 12
  20. 20. What % of foundations share their results with the general public?1. 20%2. 40%3. 60%
  21. 21. Answer (1) : 20% Prevalence 0 2 4 6 8 10 12
  22. 22. Myth 1: Evaluation is for donors WHAT ARE YOU EVALUATI NG THE DOING UP IMPAC OF OUR C T LEAN THERE? WATER PROGRAMME. DID YOUNOTICE THE SORRY, THAT’S NOTFLOODING IT I N MY CAUSED? QUES ONNAIRE. TI
  23. 23. Myth 1: Evaluation is for donors
  24. 24. Evaluation & AccountabilityEvaluatio at thehe o ac o n art f c untab : A fully ac o ility c untablephilanthro wo dob tte at g undingitswo in rig ro prac e py uld e r ro rk o us tickno dg ; e wle e xplainingno o fundings g sb thepre is sb hind t nly trate ie ut me ethe ; ac wle ingins m kno dg titutio value andb e ; and adm nal s ias s ittingwhatwedo ’t kno A fully ac o n w. c untab philanthro wo o r m rethan le py uld ffe osurfac data dis o c dfro there e c nne te m ality o iss sando anizatio that f ue rg nsfoundatio s ns uppo andfro thepurpo e andvalue that fram the rt m ss s e irde is ns Weo it too c ns nc stob thisc ar-m dab ut c io . we ur o titue ie e le inde owhe wearehe d andwhy. Witho pro ingc re ade ut b larity ando n inq , pe uiryfoundatio atte ptsat ac o n m c untab will b mre s b lic ility e e ly ym o .
  25. 25. Myth 2: Scientific methods are too rigid to understand my realityOUR PROGRAMSARE TOO YOU MEAN 9000 YEARSOF C OMPLI CATED TO S IENTIFI CPROGRES ISNOT C S EVALUATE! SUFFIC IENT TO MAKE SENS OF E YOUR PROGRAMS ?
  26. 26. Myth 2: Scientific methods are too rigid to understand my reality WE DEC IDE IT ISA S C S UC ES FUL PROGRAM IF THREE CHERRIES SHOW UP
  27. 27. Myth 2: Scientific methods are too rigid to understand my reality Community-Driven Reconstruction: led by the International Rescue Committee with support from Fearon, Macartan and Weinstein The challenge: Attribution of improvements in “community cohesion” and “democratic practice” The solution: Randomization at village level, standard surveys + tools from behavioral economics
  28. 28. Communitycohesion Socialinclusion Democratic practice andvalues Materialwellbeing 
  29. 29. But, let’s use RIGOR WITHIN REASON I ’M HERE TO EVALUATE HOW TRANS PORTATI ON I MPACTS C LDREN’SEDUCATI ON. HI I HOPE THE BUS ARRIVESTODAY. I I PROPOS TO RANDOMI ZE YOUR E DON’T WANT TO MI SSFAMI LY AND NAME YOUR KI DSWI TH S HOOL AGAIN! C ACRONYMS.
  30. 30. Myth 3: Evaluation is too expensive –we should just spend the money on the kids I AM NOT GOI NG TO WASTE ISUPPOS THEY KNOW E MONEY. THE CHI LDREN NEED WHAT TO DO, THEY ARE PROFES I ONALS S … ALL THE RES OURCESTHEY CAN GET! ISUPPOS THEY E KNOW WHAT TO DO, THEY ARE ADULTS …
  31. 31. Myth 3: Evaluation is too expensive –we should just spend the money on thekids Review of 140 community- based child protection DIMINISHED evaluations RETURNS? • rarely measured kids’ outcomes WASTED MONEY? • 84% had only ex-post measures HARM DONE? • only 3% had comparison groups and pre- and post- INTEGRITY? measures
  32. 32. AGEND Av The Bernard van Leer Foundationv Our Impact Assessmentv Common myths about evaluationv Lessons from the third sector
  33. 33. The Barry Knight Mantrawww.centris.org.uk1. Owned – People who use the evaluation feel that the system istheirs, rather than being imposed on them. The system is integrated withtheir day-today work.2. Useful – Results are relevant, and can be applied in day-to-daywork to promote learning.3. Robust – Results are valid and reliable. The system needs to besensitive to the complexity of what is likely to be involved in shifting thedeep-seated issues being worked on.4. Simple – The system works smoothly and easily without the needto have high technical knowledge. Note, however, that simple does notmean simplistic. Things should be as simple as possible but no simpler.
  34. 34. The Pareto Principle• At least 80 percent of the assessment should be driven by you and your learning needs• Mastering 20 percent of the jargon will get you 80 percent of the results you need• The first 20 percent of the cost/ time/ energy spent on impact assessment yields 80 percent of the learning
  35. 35. Theoretical calculation of efficiencysavings for UNICEF Child Protection UNICEF child Investment alone Efficiency Efficiency Cost of 42 protection potential lost RCTs (one per savings over savings over budget for 5 as a result of a country with 5 years 10 years years weak evidence low HDI) 1% - 4 million + 13 million 10% + 149 million + 319 million 1.7 billion 25% 21 million + 404 million + 829 million 50% + 829 million + 1.68 billion 75% + 1.25 billion + 2.53 billionNote: efficiency savings would only be felt after evaluation results began to feed programming.

×