Valuing Voices Cekan Wageningen Conference 0315
•Download as PPTX, PDF•
0 likes•288 views
A conference presentation on post-project evaluation in international development. Given at Dutch Wageningen University's Monitoring and Evaluation Conference 3/15
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Valuing Voices Cekan Wageningen Conference 0315
Similar to Valuing Voices Cekan Wageningen Conference 0315 (20)
Recently uploaded
Recently uploaded (19)
Valuing Voices Cekan Wageningen Conference 0315
- 1. Jindra Cekan, PhD Valuing Voices, Sustainable Solutions for Excellent Impact Wageningen University M&E for Responsible Innovation Conference, March 2015 Who’s Listening? Community-led Post- Project Sustainability Evaluation
- 2. Great Evaluation and Learning work © 2015 Valuing Voices, Cekan Consulting LLC •Our industry is evaluates thousands of multi-sectoral projects each year •Donors have invested millions of dollars/Euros in learning ventures looking for real impact, such as: • Grand (Health) Challenges • Making All Voices Count • Aid Transparency (IATI) • Impact Evaluation by 3ie •100 national national evaluator associations in 93 countries support country-led evaluation
- 3. Overlap of M&E for Responsible Innovation + Valuing Voices • How can M&E responsibly support the management and governance of innovation … and contribute to deeper reflexivity and transparent decision making? Are Institutional changes needed? • How do we take responsibility for systemic change in: – M&E professional’s roles & responsibilities; – M&E process design, focus and approach Three questions: 1. What is in our way? 2. What can be done? 3. What methods are best? © 2015 Valuing Voices, Cekan Consulting LLC
- 4. Governance: Global Aid Effectiveness © 2015 Valuing Voices, Cekan Consulting LLC Paris Declaration (2005), Accra Agenda for Action (2008) and the Busan Partnership (2011) created shared principles to achieve common goals: • Ownership of development priorities by developing counties • A focus on results • Partnerships for development • Transparency and shared responsibility
- 5. Development becoming more transparent, focused on sustainability • Local Accountability has been proposed as a core feature of the new post-2015 development agenda, according to UNICEF. • Donors talk about Country-led Development, M&E Capacity (OECD), Local Systems (USAID), as do non-profits like Local First- wonderful. Let’s support community-owned and driven ones! © 2015 Valuing Voices, Cekan Consulting LLC • MDGs are becoming Sustainable Development Goals post-2015
- 6. Changes in Responsibilities, Approach, Focus © 2015 Valuing Voices, Cekan Consulting LLC For them to evaluate our efforts = how must we change?
- 7. 1. What’s in our way? © 2015 Valuing Voices, Cekan Consulting LLC
- 8. Definitions: ‘Sustainable’ Development? © 2015 Valuing Voices, Cekan Consulting LLC How do we know it’s sustainable if we do not go back and ask? o $1.6 trillion spent on international development by EU and US alone since 2000 o $100 billion spent in 2014 o Trillions more spent by private donors ALL UNEVALUATED o National evaluations nonexistent o Data + tracking systems discarded <1% of projects evaluated post- closeout for sustainability We don’t know, we haven’t asked about the other 99% projects
- 9. Scant Design for self-sustainability Communities’ voices are rarely elicited to design what activities they will make last… for RFPs are not designed in communities Yet to not evaluate sustainability for future design short-changes the communities which have entrusted us to improve their lives © 2015 Valuing Voices, Cekan Consulting LLC
- 10. What Exists Now in project sustainability 2-5 year duration, typically with great M&E …The continuity of people’s lives in communities… ? ? ? ? ?Earlier or concurrent Development projects ? ? ? ? ? ? ? ? ? ? STOP ? baselin e midterm final E V A L U A T I O N S Feedback loops?! © 2015 Valuing Voices, Cekan Consulting LLC Follow –on or new Project Project Project GAP
- 11. “Responsible Development” requires Innovation © 2015 Valuing Voices, Cekan Consulting LLC
- 12. Why We Value Voices © 2015 Valuing Voices, Cekan Consulting LLC Designing for sustained exit should be our first priority We ALL deserve self- sustaining projects: Communities/ countries, we development workers, taxpayers… it’s true respect. ‘They’ should evaluate our assistance for effectiveness, not the other way around A business model works – looking at Return on Investment through their eyes illuminates what should be repeated – or not. Once we know what communities could self- sustain, let’s design and fund future projects accordingly! ‘They’ are ‘Us’
- 13. Views from outside on our DIME* © 2015 Valuing Voices, Cekan Consulting LLC "You cannot set your own examination, take the examination and mark it. Then you cry out success or failure. The community will just 'look at you' and wonder what is the issue. ... It will be sustainable development if the people at community level are involved in designing and delivering their own dreams of development” Peter Kimeu, Catholic Relief Services’ Senior Technical Advisor for Partnership/ Solidarity, East Africa (30 years expertise) * Design, Implementation, Monitoring and Evaluations
- 14. Views from outside on our DIME* © 2015 Valuing Voices, Cekan Consulting LLC • “We push new ideas, rather than building on what communities have already created…. we must be able to look back and see what they've learned and can sustain themselves.… to show what they've done themselves, and where they think they're taking themselves. [Sustainability] is an ignored area. The challenge is, it's just been gathering numbers for 5 years, 10 years…. • We don't have the right feedback system from communities to the NGO world. It’s the tyranny of deadlines so they "just get their concerns" when staff go out [to the field]… ” R.K. Kenyan Independent Evaluation Professional (20 yrs) * Design, Implementation, M&E
- 15. VV Data Sustainability Approach: Open Data • Design data captured using open data international standards (such as IATI) – Aggregate with other data sets (project, demographic, maps) – Sharable beyond project staff/activities – Available after project period of performance – Allows for public accountability /transparency – Historical marker of “what happened back then” for learning from measurement © 2015 Valuing Voices, Cekan Consulting LLC Mapping malnutrition in Uganda
- 16. 2. What Can Be Done? © 2015 Valuing Voices, Cekan Consulting LLC
- 17. Discussion: what is in our way, what can be done? © 2015 Valuing Voices, Cekan Consulting LLC World Bank Civil Society Forum Meeting 2014 1. ARE WE RIGHT ABOUT WHAT’S IN OUR WAY? 2. WHAT CAN WE DO TO HELP DONORS LISTEN AND HELP IMPLEMENTERS COME ON BOARD? LISTENING TO YOUR VOICES!
- 18. 3. What methods are best? © 2015 Valuing Voices, Cekan Consulting LLC
- 19. Right Design of Post-Project Sustainability Evaluation and Objectives © 2015 Valuing Voices, Cekan Consulting LLCDEVELOPMENT INDUSTRY LEARNING ACCOUNTABILITY TO PARTICIPANTS LEARN FROM UNEXPECTED IMAPCTS EMPOWER LOCAL NGOS, COMMUNITIES, AND NATIONAL EVALUATORS LOCAL ROI SHAPES WISER FUNDING DECISIONS AND BETTER PROGRAMMING PROMOTE COMMUNITY-INFORMED AND JOINTLY LED LEARNING FROM SUSTAINED OUTCOMES
- 20. Moving from Forest to Weeds… How © 2015 Valuing Voices, Cekan Consulting LLC “Because… the devil is in the detail”
- 21. Site/ Partner Selection • Project sites closed within the last 2-5 years • Communities, partners available for consultations, final evaluations accessible • National evaluators are available and interested • Sampled sites comparable in livelihood, diversity • No similar work done by NGOs in gap years • Ideally web-savvy partners to handle data/ analysis/ curation/ storage • Analysis, presentation, sharing and curation of findings are clear (by whom and for whom, how, where, for how long) © 2015 Valuing Voices, Cekan Consulting LLC
- 22. Mixed-Methodologies © 2015 Valuing Voices, Cekan Consulting LLC • Qualitative: Participatory RRA methods/ tools (based on Appreciative Inquiry and Empowerment Evaluation, potentially also Outcome Harvesting) of past participants in projects • Quantitative (purposive, then random-sampled HH survey, possibly including mobile surveys for participants who’ve left • Small control group: non-participants in same communities •Use mix of internal and external evaluation teams covering all sectors • Findings shared widely and stakeholders invited to discuss
- 23. Qualitative process © 2015 Valuing Voices, Cekan Consulting LLC Research: 1. Update sectoral evaluator team and translators in RRA approach 2. Discuss best RRA tools (maps/calendars, rankings, focus groups) 3. Visits: Two to three days per site in 3-4 sites, with meetings in communities reminding them of project and asking: A. What activities/outputs were most self-sustainable and why? B. Which activities should be repeated? Which shouldn’t? C. What else would participants like to evaluate of the project?Distillation: 1. Daily team analysis 2. Each site summary analysis is presented to the community for input 3. Community invited to final presentation in region 4. Findings inform survey + reportLater, compare across projects/sectors
- 24. Quantitative process © 2015 Valuing Voices, Cekan Consulting LLC Research: 1. Survey questionnaires of a random sample subset of the population (past participants and some communities’ non- participants) 2. Stratification of intervention area by criteria; the strata would include a homogeneous stratum (same behaviors in project, e.g. all sustaining activity) and heterogeneous stratum (those not) but who are judged on the same key variables or indicators. 3. Randomly choose a fixed number of villages for interviews Distillation: 1. SPSS for analysis 2. Community invited to final presentation in region 3. Findings for integrated report Later, compare across projects/sectors
- 25. Looking for Unexpected Results © 2015 Valuing Voices, Cekan Consulting LLC
- 26. LWR Niger post-project unexpected impacts Evaluated Food security/ Livelihood/ Resilience project: sheep distribution, fodder, wells, peaceful transhumance for 500 women post-drought Expected findings: • Variable impacts on women’s economic benefits • Great impact on time gain from wells for income generation. Unexpected wonder: • Decreased domestic violence from presence of livelihood + water © 2015 Valuing Voices, Cekan Consulting LLC
- 27. 3. Methods: Discussion questions © 2015 Valuing Voices, Cekan Consulting LLC 1.COMMENTS ON METHODS FOR ANY ONE SITE? 1. THOUGHTS ON COMPARABILITY ACROSS SITES, ACTIVITIES AND PROJECTS?
- 28. Let’s change how international development works… together © 2015 Valuing Voices, Cekan Consulting LLC Thank you for your input! Jindra@ValuingVoices.com
Editor's Notes
- Results: Having a sustainable impact should be the driving force behind investments and efforts in development policy making. Partnerships: Development depends on the participation of all actors, and recognizes the diversity and complementarity of their functions. Transparency: Development co-operation must be transparent and accountable to all citizens
- Millions of dollars of development projects come and go from people’s lives after 2, 5, even 10+ years, usually without gaining the benefit of learning how sustainable their work has been. Excellent monitoring and evaluation (M&E) are done during the project, examining how well project design and implementation worked to meet objectives, goals, impact. Yet development implementers, donors and partners stop great M&E/ learning and hope for the best as they withdraw resources at project funding’s end.
- Sustainability is something left to community based organizations, often without staff or financial resources to go on anywhere nearly as robustly. Communities are routinely abandoned to their own (hopefully strengthened) devices. We must support communities to evaluate these impacts themselves and teach us what they need. This returns development to their hands and empowers them to challenge future programs to be different.
- There is a gaping hole in development and accountability: no longitudinal post-project evaluation of expected impacts and what has emerged after projects end, much less community-determined parameters of desired impact; Sustainability is now narrowly defined as continued orgazinat’l financing; Projects come and go, barely consult and withdraw resources in fixed times Hardly ever are unintended impacts documented and learned from… + or - Learning is often limited to within the projects, rarely across projects or later Data and systems end at close-out, none open-source (rarely searchable)
- Briefly Aminata (Mali) and ___ (Sio’s Mali story) We believe: We ALL (Africans, Asians, Latin Americans, Europeans/ Americans and others) deserve self-sustaining projects Communities living with projects should co-design them, use their own ICT systems and have strong capacity manage projects themselves and to evaluate how effective our assistance has been Much international development assistance is needed; we must jointly learn from the past for future resource efficiency, time-use, credibility and sustained impact Countries and communities must self-govern – how can we help them get there fastest with the best technology, know-how and respect
- Ethos of designing Requests for Proposals in HQ, far from where folks will live with it Barriers of fixed time (1, 3, 5 years) even if renewable, have set/ planned outcomes/ impacts Enormous pressure to report success to oversight (Congress, Parliament), so much control it Little (but growing) adaptability of projects to situations, or informative feedback loops Pressure of too few staff in the field, with too high deadlines, weak counterparts….
- · Help shape development industry learning about how to foster sustainability in programming · Demonstrate project implementer commitment to accountability · Add to development partners’ understanding of expected programming sustainability in sectors such as Agriculture, Health… · Add to project understanding of unexpected impacts · Empower communities’ appreciation of their evaluative capacities to participate in feedback loops after and during implementation · Inform current programming and design future ones better! · Build National evaluator and NGO staff capacity · Inform donor learning on exit strategies/sustainability
- Site Selection CRS programming (or comparable NGO projects) has been closed out for at least two years Evaluation documents need to be available, including village and participant lists Partners and CRS staff familiar with the closed project need to be consulted on-site Timing does not interfere with livelihoods in communities, holidays, or other CRS technical work, so staff (and other resources) are available A consultant is available to prepare communities for the visit and identify participants Possibility of learning exists from the post-project evaluation to support similar current implementation Community of intervention Representative of agro-ecological zones and other economic or livelihood criteria No other NGO doing similar work in the area Representative in terms of accessibility of the area Willingness of the community to participate in an evaluation
- Quantitative: Focused quantitative surveys in the form of questionnaires of a random sample subset of the population (past participants and some communities’ non-participants) will be used. The choice of the evaluation area and number of households will be done through a stratification of intervention area according to some relevant criteria; the strata would include a homogeneous stratum, in which individuals are from the same area and have the same behaviors, attitudes and beliefs pertaining to concepts applied by PROSAN, as well as a heterogeneous stratum in which individuals are from different areas and hold different beliefs, but who are judged on the same key variables or indicators. In each stratum we could randomly choose a fixed number of villages where households will be interviewed. Depending on the way the program was implemented, the choice of villages should be participant-based: a number (to be determined) of households will be determined in each village that participated versus those that did not and a methodology of choice of household to be interviewed will be chosen for each village. The number of households to be interviewed will take into account both participants and non-participants. Data from questionnaires will be analyzed using SPSS. ‘360-degree feedback’ will be elicited from group leaders, local NGOs and local government partners especially to track unexpected impacts and explore why some outcomes were continued and others were not, as well as to assess the project’s partnership and capacity building components. For this ex-post evaluation, the appropriate method to identify its impact will consist in using propensity score matching methods (PSM). To this end, the number of non-beneficiaries would be higher than beneficiaries in order to benefit from a wide range of PSM possibilities at the analysis side. After the interview, we will enter, clean and analyze data gathered.