Recommended
More Related Content
Viewers also liked
Viewers also liked (11)
Similar to Evaluating Philanthropy Programs and Scientific Methods
Similar to Evaluating Philanthropy Programs and Scientific Methods (20)
Recently uploaded
Recently uploaded (20)
Evaluating Philanthropy Programs and Scientific Methods
- 1. E va lua tions a nd P hila nthropy WE DECIDE IT IS A SUCCESSFUL PROGRAM IF THREE CHERRIES SHOW UP
- 2. S c ie ntific me thods a re too rig id to unde rs ta nd my re a lity OUR PROGRAMS ARE TOO COMPLICATED YOU MEAN 9000 YEARS OF TO EVALUATE! SCIENTIFIC PROGRESS IS NOT SUFFICIENT TO MAKE SENSE OF YOUR PROGRAMS?
- 3. Ag e nda • The Bernard van Leer Found ation • BvLF im pact m easurem ent • Lessons from the Third Sector
- 4. Mis s ion O ur m ission is to im prove opportunities for child ren up to age 8 who are growing up in sociall and econom ical y ly d ifficult circum stances. We see this both as a val uable end in itself and as a l ong-term m eans to prom oting m ore cohesive, consid erate and creative societies with equal opportunities and rights for al l
- 5. S tra te g ic G oa ls 2010 - 2015 • Red uce viole nc e in the lives of young child ren • Take quality e a rly le a rning to scal e • Im proving young c hildre n's he a lth by changing the cond itions in which they live
- 6. T he P hila nthropic T oolbox 1. Knowledge Development o research, documentation, evaluation 2. Service Product Development and Delivery o investments, grants 3. Capacity Enhancement and Skills Development o training, technical assistance 4. Behaviour Change Programmes o campaigns, awareness 5. Policy Development & Implementation o community organising, legal empowerment, lobby 6. Enabling Systems & Infrastructure o networks, markets
- 7. IS R AE L Reduced rates of malnutrition among youngamong young Reduced rates of malnutrition among Bedouin young Bedouin children growing up in Universal access to quality pre-school Reduced incidence of depression, among 3 to 6 year old children. anxiety and aggression children growingArab childrenunhealthy Negev.physical environments in Jewish and up in exposed to unhealthy political violence. the physical environments in the Negev. The National Ministry of Education, Improved knowledge about dietary dietary Parents and children experience Improved knowledge about local municipalities and members of reduced levels of stress from exposure and infrastructure causes of child the business sector have increased and infrastructure causes of child to political violence illness and malnutrition their financial investments in preschools for children illness and malnutrition The ratio of pre-school to supervisor has decreased Increased access to clean water, clean water, The Israeli government has an appropriate public system for Increased access to improved waste management and improved treatment of management and prevention and waste depression, anxiety and aggression electricity electricity among young children supervisors are better equipped to Improved access to transport among improve pre-school quality Bedouin communities, especially women and children The quality of teaching and learning in Improved access to transport among pre-schools has improved Bedouin communities, especially The curriculum and reflects the cultural backgrounds of the children women and children
- 8. Pilot health education programmes Reduced rates of malnutrition among young Bedouin children growing up in Lobby for unhealthy physical environments in the Negev nurses Improved child health (esp. reduction in digestive tract illness) Improved access to mother-child wellbeing clinics Better access to Improved waste Healthy dietary practices clean water management to combat vitamin Regional and deficiencies municipal advocacy Improved housing conditions (water, sanitation, (hygienic, ability to regulate Electricity electricity) temperature) Increased awareness among parents about dietary and infrastructure causes Bedouin of child malnutrition Government provision of Increased infrastructure in Bedouin families invest incomes towns and villages in own among infrastructure Bedouin families Increased mobility (e.g. Government transport) among Bedouin Joint municipal and planners and communities (esp. women) community planning Legal recognition of More Bedouin Bedouins villages women join the agree on a child-friendly workforce Research and plan communications about PE and child Bedouin communities are well organized, have more positive health link public image and are supported by professional planners Organizing for transport Private sector transport options
- 9. What are the most cost- effective interventions to improve Bedouin children’s health and nutrition? Health and nutrition Individual evaluations outcomes for children combined with meta-analysis How are our strategies working to influence investment and service delivery in the Negev region for Review and Bedouin children? recommendations by Budgets advisory team with Evaluation of cluster of grants Effective coalitions (Arab, knowledge of broader from advocacy perspective Jewish, government, private political context sector, parents) Access to basic services How are our efforts to empower Bedouin communities in the planning process working to influence Health and nutrition investment and service delivery? outcomes Case study of selected municipalities
- 10. B ut, le t’ s us e R IG O R WIT H IN R E AS O N I’M HERE TO EVALUATE HOW TRANSPORTATION IMPACTS CHILDREN’S EDUCATION. I HOPE THE BUS ARRIVES TODAY. I PROPOSE TO RANDOMIZE I DON’T WANT TO YOUR FAMILY AND NAME YOUR MISS SCHOOL KIDS WITH ACRONYMS. AGAIN!
- 11. Of th e th r e e te c h n iq u e s lis te d , w h ic h o n e d o yo u th in k w as m os t c om m on? 1. Log fram es 2. Participatory Action Research 3. O utcom es M apping
- 13. Main goals of the philanthropic activity The Foundation cannot solve directly any social Test and validate problems. Its mission aims therefore at testing innovative (especially with projects directly managed) policies innovative solutions to social problems and at disseminating successful solutions (“what works”). Solutions to certain problems are well known and Reward best organisations implementing related initiatives are practices numerous: in such cases the Foundation selects and funds the best projects through specific calls for proposals. Aimed at supporting (with institutional grants) Support worthy deserving nonprofit organisations (operating in the institutions sectors of Arts & Culture, Environment, Scientific Research, Social Services) prevailingly based in Lombardy (Cariplo Foundation’s traditional intervention territory) 25/05/2011 Strategic Unit for Philanthropy - 13 Evaluation Office
- 14. Main purposes of evaluation Test and validate innovative policies Reward best practices Support worthy institutions Critical Knowledge Accountability analysis/ sharing Learning 25/05/2011 Strategic Unit for Philanthropy - 14 Evaluation Office
- 15. T ools : F IT F O R P U R P O S E http://trasi.foundationcenter.org
- 16. Wh e n th e b o s s s a ys “ e va lu a te it,” d o e s it m e an… 1. The program is in j eopard y 2. The program is about to be com m unicated as a raving success and you need charts and graphs to prove it 3. She has never heard of the program
Editor's Notes
- I am going to click through some initial slides here on our mission and goals so that you have some understanding of the context within which we operate.
- EFC polling research and a large meeting held in Portugal last year with many foundations confirms that many foundations are actively hostile to evaluation.
- 8 countries + a Roma program on early learning across EU countries.
- This is our toolbox. These are social change instruments highlighting the types of strategies and change that each is associated with. Most philanthropists use one or two of these tools. We use all six, but we are an old foundation that is thoroughly professional.
- I am going to use Israel as an example. You can see here our goals and outcomes. Goals are at the top. Underneath each one we have some outcomes we would like to see. Let’s look at the third one.
- This is a Theory of Change on how to improve nutrition rates amongst Bedouin Children. The goal is in blue on the top. The tools we use, i.e. what sort of investments we want to make are in pink on the sides and the outcomes we seek and their relation to each other are noted in green. Theories of change are handy tools that help us communicate to others what we hope or think or expect to have happen when we make investments. Every goal we pursue in the BvLF has one of these little maps attached to it. A theory of change is a map of our assumptions of what needs to change in order to reach our goals.
- In order to evaluate whether our theories of change are working and our investments are having an impact we have developed an evaluation scheme. I want you to look at the grey boxes. These are three simple questions. Individual grants/investments are evaluated on whether they have directly changed children’s health outcomes and we usually want to look across all the approaches to determine which is most cost effective. Many of our grants are aimed at changing the conditions in which these children live as we believe that this is more sustainable than us just funding the services so we also evaluate the whole package of grants through on how they are influencing service delivery and investments. The outcomes are in pink; the methodologies in multiple colors in the center and the questions are in grey. We also know that we are not the smartest people on the planet so the smaller evaluations that look at specific outcomes will be reviewed not only by our team but by an independent team of advisors who can help us strengthen our approaches. This is the key slide of this presentation. If I take another example, we have been working in the Caribbean on a program framework for 10 years. We are in our third round of longitudinal studies to assess the impact assessement of home based visiting services on children’s capacity to learn and as a side event later – on the levels of violence in the household. Amsterdam Institute for International Development undertook this study for us. The latter we added to the design of the evaluation. It is a quasi-experimental design. However, this was not an adequate methodology to help us understand the full impact of the program so we also had John Gaventa’s dept at Sussex university undertake an evaluation that would help us understand how stakeholders other than children were empowered. This was a qualitative review. Excellent findings by the way.
- RCTs are not always a good option and should be used selectively.
- These are three different evaluation techniques. Log frames can be used when there is a high level of predictability. PAR when community engagement is part of your goal. Outcomes mapping when you have more than one objective.
- We are not the only foundations who do this kind of work. Cariplo in Italy has a strong approach as well that is at the center of what it does. Here they depict the main goals of the foundation.
- I particularly like Cariplo’s matrix of purposes on how to use information from evaluations. We wouldn’t want to consolidate just yet because there is no right or wrong way to make social change. There are many ways. Essentially you must be able to understand what you want to learn.
- Consolidating the evaluation tool kit in my view is not a good idea. Traci has documented 150 different methods or tools for assessing social impact and it is a free, valuable resource when searching for the right tool. The Foundation Center will tell you that you have to choose a good methodology for what you are trying to accomplish. RCTs for example are not good tools for evaluating advocacy, media interventions, the impact of cultural activities or the strength of networks or other interventions where it is hard to control the scope of impact or define a control group.