Monitoring and Evaluation Plan for Africa RISING Naomi Sakana (IFPRI) Africa RISING East and Southern Africa Research Review and Planning Meeting, Arusha, Tanzania, 1-5 October 2012
OutlineAR Objectives, activities, and expected resultsMonitoringEvaluationM&E ObjectivesPrinciplesIndicatorsMethodsRoles and Responsibilities
Africa RISING purpose• Objective Transform agricultural systems through sustainable intensification Sustainable intensification: producing more (agricultural) output from same area of land, while: reducing negative environmental impacts increasing contributions to natural capital and flow of environmental services
Africa RISING activities • Implementation activities Situation Analysis and Program-wide Synthesis ( diagnosis & design of technology combinations) Integrated Systems Improvement (on-farm & on-station trials of new combinations of existing technologies) Scaling and Delivery (institutional and/or value chain improvements) • Systems and Sites Cereal-based farming systems – Guinea-Savannah zone of West Africa Crop-livestock systems - Ethiopian highlands Maize-legume-livestock & Rice-vegetable systems - Eastern & Southern Africa
Expected results• Expected key, direct results of implementation • Sustainably increased whole farm productivity • Reduced negative environmental impacts • Increased contributions to natural capital & flow of environmental services • Value chain conduct and performance improved? / Improved market efficiency? / Improved extension effectiveness?• Many other results possible: labor use, WUE, poverty, nutrition, hunger, etc.
Monitoring“process of systematic collection and analysis of data on specific”• Describes the “what” of implementation• Useful for: Management - to see if project is on track Reporting – to inform client & other stakeholders of progress• Indicators USAID FTF Others (custom) useful to AR for monitoring or evaluation or both
Evaluation“periodic assessment of worth or significance of an activity, policy or programme”• Helps understand “how” & “why” of implementation• Determines & attributes impact• Qualitative and/or quantitative• Dimensions Sites (different levels) Development domains Household types Technologies & combinations Implementation processes
M&E Objectives• Support effective project management (provides the project managers with timely information on the status of activities and the ongoing results they are achieving )• Help all stakeholders to learn about the project’s successes and failures (provide opportunities to learn what works and what does not)• Provide the data for timely reporting to USAID
M&E Commitments• FtF Compliance: M&E standards, best practices, and core indicators established for the entire FtF initiative.• Open-access platform: deliver and maintain an open-access, M&E data management and analysis platform to serve the needs of SI implementation partners and other stakeholders.• Monitoring & projection: generate ex ante evaluations (e.g. project targets) for a range of farming system and livelihood outcome indicators on an annual basis to provide enhanced research management and outcome mapping needs.• Multi-scale reporting: provide the capability to support multi-scale monitoring and evaluation SSA-wide: cross-system reporting to serve the needs of SI wide roll-up of indicators across the three investment geographies/system “project sites” (Guinea Savanna, Ethiopian Highlands, Eastern and Southern Africa)
M&E Commitments• Multi-scale reporting: provide the capability to support multi-scale monitoring and evaluation Site-wide report: for each of the three project sites Country report: Breakout of site-wide reports to serve the needs of national stakeholders (e.g., USAID country missions, national institutions) Custom/Sub-system reports: Some reporting needs will need to be met by customized aggregation of sub-system indicators (e.g. to generate reports by CRP or by farming system)• Scaling-up and out potential outcomes and impacts: To inform planning and longer-term projections of potential innovation impact at scales beyond the actual action research sites, forward looking analysis will explore the productivity and sustainability consequences of a range of adoption scenarios and geographic/system spillover pathways across broader landscapes and regions.
Africa Rising M&E Components, Activities, and OutputsProgram/Project Site Identification Data/Analysis Platform Outputs Guinea – Ethiopian Contextual Data M&E Outputs Savannah Highlands -FtF Indicators A Project Site (national/regional) Stratification -Outcome mapping (incl. B - Statistics A C (Development - HH survey & census nutrition & market effect) Domains) -Cost/Benefit analyses - Spatial data -Experimental /RCT East and Derived Indicators evaluation Southern Africa Ranking domains by key AR - HH Typologies -Adoption studies? Maize Mixed attributes - Intensification Index A ________ C ________ - Sustainability Index FtF Indicators / reports by B ________ - Nutrition index? - Research sites - Country / National level Action Research Site Project/Activity/ - Project sites Identify action Partner Inventory selection criteria - Program / SSA research sites in - Project DB (& maps) -Site access priority domains that -Existing activity/platforms satisfy selection SI Innovation Catalogue Innovation Inventory -Research design criteria - Inventory (cross-site) - Standard metadata -Intervention type - Characterization - User interfaces -M&E approach - Open access -……. Research Site Communities/ Farms/Plots Site Data M&E approach Activities - Climate, soils, ++ + Perfomance Variables Baseline survey market access, etc (modeling & validation) Project Planning & Set up trials + + - Community/HH -∆Whole farm productivity Management Monitoring + survey data - Experimental data -Technology performance Mid-line survey (?) ∆ Yield Improved insights into - Model input data End-line survey ∆ Labor prod.- by gender innovations , delivery platforms, and site selection ∆ NUE, WUE Whole-farm models - ∆Revenues, Costs, Profits Learning
Africa RISING Results Framework and Indicators (based on FtF) FTF Goal: Sustainably Reduce Global Poverty and Hunger FTF First Level Objective 1: Inclusive Agricultural Sector Growth IR 1: Improved Agricultural ProductivityOutcome Indicators:#10 Gross margin per hectare (whole farm and by crop) IR 3: Increased#11 Number of hectares under improved technologies or management Investments in Agriculture and Nutrition-Relatedpractices ActivitiesSub-IR 1.1: Enhanced Human and Sub-IR 1.2: Enhanced IR 2: Expanding Output Indicator:Institutional Capacity Development Technology Development, Markets and Trade • #9 Number of public-for Increased Sustainable Agriculture Dissemination, private partnerships formedSector Productivity Management, and Outcome Indicator: Innovation #14 Value of incremental Outcome Indicator:Outcome Indicators: sales • #16 Value of new private• #5 Farmers who applied new Output Indicators: sector investment in the technologies or management • #7 No. of new technologies Custom Outcome agriculture sector or food practices… or management practices: Indicator: chain• #6 Private enterprises/organizations 1) Under research, 2) Under #15 Farmer satisfaction that applied new management field testing, or 3) Made with quantity, quality and practices… available for transfer timeliness of extension Sub-IR 1.3: Improved IR 5: Increased• #12 Stakeholders implementing risk- • #13 No. of rural households and input supply services Agriculture Policy Resilience of reducing practices/actions to improve benefiting directly from (Sub-IR 2.3: Improved Environment Vulnerable resilience to climate change USG interventions Market Efficiency) Communities and Output Indicator: HouseholdsOutput Indicators: • #8 No. Policies/• #1 Individuals who received long- Regulations/Admin Output Indicator: term training Procedures in each • #17 No. of• #2 Individuals who received short- stage of vulnerable term training development as a households• #3 Private enterprises/ organizations USAID-provided IRs and indicators benefiting directly result of USG Additional suggested IRs and indicators from USG receiving assistance assistance interventions• #4 Producer/community based reorganizations receiving assistance
Improved connectivity to and Wider dissemination of integrated SI innovations leading Increased R4D community ability to design and Increased R4D community ability to design and utilization of markets and input to similar impacts beyond the AR Action Research Sites implement farm-scale SI action research, outreach and implement farm-scale SI action research, outreach and suppliers support approaches, and related D&KM systems support approaches, and related D&KM systems RO 1&2: Sustainable increase RO 1&2: Increased RO 1, 2&3: Improved RO 1, 2 &3: Increased ability of R4D RO 1, 2 &3: Increased ability of R4D RO 1, 2 &3: of whole-farm productivity by nutrition and understanding of landscape level community to design and implement farm- community to design Improved integrated innovations for reduced poverty, ecosystem stability from the scale action SI research, outreach and scale community- targeted households at especially for aggregate impact of farming support & related D&KM systems support based & on-farm research sites women and practices at the household level NRM children RO 1: Situation Analysis & Program Synthesis RO 2: Integrated Systems Improvement RO 3: Scaling and Delivery of Integrated RO 3: Scaling and Delivery of Integrated Innovations InnovationsDiagnosis - Site selection & Characterisation Participatory evaluation and adaptation of • Mega-site stratification by development domains appropriate combinations of technologies and Scaling up/out successful technologies and • Prioritizing mega-site strata geographic units interventions interventions • Project Action/Research Site Selection • Combining improved legumes with improved • Assessing the scalability of integrated innovations • Action/Research Site Characterisation livestock feeding practices • Identification and development of scaling • Problem diagnosis • Managing soil fertility and experimenting with approaches for targeted integrated innovations • Construction of farm household typologies novel approaches to increase productivity • indicators • Identifying constraints and opportunities (disabling • Developing incentives for better soil management • Testing approaches for scaling up and scaling out environment, options, entry points) • to similar site specific recommendations or SI, leading Developing impacts beyond the AR SI innovations in action sites with project area • Conducting value chain assessment • Combining action research sites identified technologies • Developing a costed program for scaling by • Literature review (e.g., Agroforestry / MPT, alternatives to draught development investors • Baseline survey power to save feed, CA with a livestock lens, • Evaluating aggregated impact of household level • Development of common key indicators fertilizer trees, fiderbia/ Acacia/ leguminous trees, interventions at landscape level (biophysical, socio-economic & institutional) improved management of seasonal feed resources, • Evaluation and validation of scaling approaches for • Technology inventory and characterisation kitchen gardens/ continued poultry, legume integrated systems • Ex-ante analysis of potential options rotations (effective rhizobia, biological N fix), micro • integrated systems priority setting & planning dosing, more effective contribution to livestock to • Cooperation and collaboration with partners nutrient management, supplemental irrigation, rainwater harvesting,etc.) Developing approaches for farm level interventions • Identifying key components of integrated systems Knowledge and Capacity building • Identification of intensification trajectories • Testing novel extension models • Sequencing interventions to suit stage of • Establishing a linked system of models intensification of household types / systems Figure 3: AR Results Framework based • Training on market oriented production • Developing participatory tech. selection methods on Research Design/Plan Document. • Addressing new research challenges and • Identifying models and decision support tools to opportunities emerging from the activities (compare with Figure 2) guide ex-ante technology identification • Ex-ante sustainability & resilience evaluation • Identify high impact sweet spots/ best bets/fits
Analytical Approaches & Tools• Delineation and characterization of target farming systems• Change estimation/projection models for selected indicators• Outcome mapping• Attribution assessment, including site selection and stratification, potential comparison group(s), and impact assessment design
Impact evaluation methodsMethod Pros Cons Likely to be usedOutcome Mapping Easier to implement Primarily qualitative. Subjective Yes and interpret. Forces assessment approach articulation of impact pathwaysDD (difference in Provide quantitative Not as rigorous if RCTs are not Yesdifference)/RCTs evidence carried outPSM/IPW Provide quantitative Not as rigorous if RCTs are not Yes(propensity score evidence, although carried outmatching methods) second-best optionRDD (regression Provide rigorous Big sample needed, sharp cut- Nodiscontinuity design) evidence off based on continuous eligibility criterion (which AR is currently not supporting)
Roles and Responsibilities• Monitoring implementing partners (IITA and ILRI and Collaborators), Sister CG centers, NARS, FOs, NGOs, Private sector• Evaluation (IFPRI and partners)
AR M&E Learning Task Force• 3 month brief (1st Qtr 2013)• M&E Task Force (AR, CSISA, FEEDBACK, CRSP, Local Institutions M&E specialists)• Provide CSISA-AR cross-learning and FtF FEEDBACK best-practice guidelines• Visit all sites, meet local teams and refine M&E strategy that; (1) appropriate for interventions being tested, (2) strikes appropriate balance between rigour and cost/feasibility, (3) lays out strategy for remainder of 5 years.