Your SlideShare is downloading. ×
0
Considerations for AAS CRP Impact Evaluation - Workshop on Strengthening Impact Evaluation in Natural Resource Management
Considerations for AAS CRP Impact Evaluation - Workshop on Strengthening Impact Evaluation in Natural Resource Management
Considerations for AAS CRP Impact Evaluation - Workshop on Strengthening Impact Evaluation in Natural Resource Management
Considerations for AAS CRP Impact Evaluation - Workshop on Strengthening Impact Evaluation in Natural Resource Management
Considerations for AAS CRP Impact Evaluation - Workshop on Strengthening Impact Evaluation in Natural Resource Management
Considerations for AAS CRP Impact Evaluation - Workshop on Strengthening Impact Evaluation in Natural Resource Management
Considerations for AAS CRP Impact Evaluation - Workshop on Strengthening Impact Evaluation in Natural Resource Management
Considerations for AAS CRP Impact Evaluation - Workshop on Strengthening Impact Evaluation in Natural Resource Management
Considerations for AAS CRP Impact Evaluation - Workshop on Strengthening Impact Evaluation in Natural Resource Management
Considerations for AAS CRP Impact Evaluation - Workshop on Strengthening Impact Evaluation in Natural Resource Management
Considerations for AAS CRP Impact Evaluation - Workshop on Strengthening Impact Evaluation in Natural Resource Management
Considerations for AAS CRP Impact Evaluation - Workshop on Strengthening Impact Evaluation in Natural Resource Management
Considerations for AAS CRP Impact Evaluation - Workshop on Strengthening Impact Evaluation in Natural Resource Management
Considerations for AAS CRP Impact Evaluation - Workshop on Strengthening Impact Evaluation in Natural Resource Management
Considerations for AAS CRP Impact Evaluation - Workshop on Strengthening Impact Evaluation in Natural Resource Management
Considerations for AAS CRP Impact Evaluation - Workshop on Strengthening Impact Evaluation in Natural Resource Management
Considerations for AAS CRP Impact Evaluation - Workshop on Strengthening Impact Evaluation in Natural Resource Management
Considerations for AAS CRP Impact Evaluation - Workshop on Strengthening Impact Evaluation in Natural Resource Management
Considerations for AAS CRP Impact Evaluation - Workshop on Strengthening Impact Evaluation in Natural Resource Management
Considerations for AAS CRP Impact Evaluation - Workshop on Strengthening Impact Evaluation in Natural Resource Management
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Considerations for AAS CRP Impact Evaluation - Workshop on Strengthening Impact Evaluation in Natural Resource Management

551

Published on

Presentation by Charlie Crissman

Presentation by Charlie Crissman

Published in: Investor Relations
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
551
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • *
  • *
  • *
  • Add figure from pp 2 of Stern WP. Discuss elements of figure Evaluation question – give examples of different demands for IA questions AAS wishes to take an active role in framing what should be the appropriate evaluation question *
  • *
  • *
  • *
  • *
  • *
  • *
  • *
  • *
  • *
  • *
  • *
  • *
  • *
  • *
  • *
  • *
  • Transcript

    • 1. Considerations forAAS CRP Impact Evaluation Workshop on Strengthening Impact Evaluation in Natural Resource Management WorldFish, Penang, Malaysia 4-5 September 2012 Making a difference in the lives of the poor
    • 2. Pre-implementation evaluation •Are the objectives well defined so that outcomes can be stated in measurable terms? •Is there a coherent and credible implementation plan that provides clearPerformance Logic Chain Evaluation evidence of how implementation is to proceed and how successful implementation can be distinguished from poor implementation?The performance logic chain assessment evaluation •Is the rationale for the deployment of resources clear and commensuratestrategy is used to determine the strength and logic with the requirements for achieving the stated outcomes?of the causal model behind the policy, program, orproject. Process implementation evaluation What did or did not get implemented as planned?Impact Evaluation Rapid Appraisals… the classic evaluation that provide timely, relevant information toattempts to find out the decision-makers on pressing issues they facechanges that occurred, and to in the project and program setting.what they can be attributed The aim of applied research is . . . to facilitate a more rational decision-making process in real-life circumstances Case Studies … use when a manager needs in-depth information to understand more clearly what happened with a policy, program, or project Making a difference in the lives of the poor
    • 3. AAS CRP Implementation strategyMaking a difference in the lives of the poor
    • 4. 1. A framework to guide selection of impact designsMaking a difference in the lives of the poor
    • 5. 2. Evaluation Challenge for AAS•Many CG stakeholders view experimental designs for IAas ‘best’•Mostly experimental or statistical methods based IA NRM-type programs demonstrate little evidence of impact•Tradition of CGIAR IA could result in mis-match ofquestions and methods that could bias results against AASprogram•Demonstrate the validity (legitimacy) of alternate methodsfor AAS CRP Making a difference in the lives of the poor
    • 6. 3. Evaluation Goals of AAS•Accountability •Respect Consortium and Fund needs for demonstration of performance and potential for impact•Learning •Respect AAS program philosophy of a learning, adaptive program Making a difference in the lives of the poor
    • 7. 4. Consortium Evaluation Perspectives: A Performance Management SystemIntermediate development outcomes: •represent a change, in the medium term, which is intended to affect positively the welfare of the targeted population or environment resulting, in part, from research carried out by the CGIAR (and its partners) •are attributable to CRP-level activitiesThe CRPs are expected to document these developmentoutcomes through periodic studies on outcome and impact.Source: ISPC. Strengthening Strategy and Results Framework through Prioritization. June 2012 Making a difference in the lives of the poor
    • 8. 4. Consortium Evaluation Perspectives: A Performance Management SystemBenefits for the centers are to be able to answer these questions•How are we performing and what impact are we having in achieving theSLOs?•Are we learning and using what we have learned to improve?•What changes can we make within the CG to be able to provide betteranswers to questions 1 and 2?•CRPs will be held accountable – but be left to implement flexibly Making a difference in the lives of the poor
    • 9. Impacts - change in problem - change in opportunitiesDevelopment - change in actions/behaviour of stakeholdersOutcomes - change in productivity - change in equity/empowerment - change in market conditions - change in investments - change in security of assets/habitatsResearch - recognition/appreciation of research knowledgeOutcomes - use of knowledge by partners - mobilisation of new capacity - extension of technology/materials - change in policy environmentOutputs - change in knowledge - change in capacity - change in technology - change in materials - change in policy options - change in awareness/understanding Making a difference in the lives of the poor
    • 10. 5. Attributes of NRM programsGeneric attributes of NRM programs•Multi-stakeholder participation and coordinated action•Multi-leveled: farm, landscape, region…•Uncertain and lengthy trajectories for impact•Knowledge generation•Contextualized knowledge is vital•Systems integration•Operates in areas of limited/little prior or reliable knowledge•Institutional concerns•Emerging outcomesSource: White paper Making a difference in the lives of the poor
    • 11. 5. Selected attributes of the AAS program•Strongly place based •Challenging agro-ecosystems •Marginalized populations•Multi-stakeholder participation •A major process outcome •Increases in network capital•Interventions at multiple levels •Strong commitment to community-level agendas •Regional and global agendas too•Multiple-interventions •‘Research in development’ implies that development is happening •Overlapping programs and many actors•Hub-level NRM issues offer opportunities and create constraints •Emergent effects •Contextualized causal mechanisms •Long term processes Making a difference in the lives of the poor
    • 12. 5. Hub Development ChallengesMalaita, Solomon Islands•Rural people in the hub face major challenges from rising population and diminishing marineresources. The development challenge is to improve their lives through more productive, diversifiedlivelihoods that empower communities to be better able to adapt to change and make more effectiveuse of their marine resources. The research challenge is to develop and test alternative approaches tolivelihood diversification and resource stewardship that will accelerate development and restore theproductivity of their resources.Barotse Floodplain, Zambia•To make effective use of the seasonal flooding and natural resources in the Barotse floodplain systemthrough more productive and diversified aquatic agricultural management practices and technologiesthat improve the lives and livelihoods of the poor.Khulna Floodplain, Bangladesh•We seek positive transformational change in the lives and livelihoods of poor AAS-dependent farmersand their communities, particularly women and youth. We strive for empowered communities thatlead in the innovation and adoption of more productive, diversified and resilient practices andtechnologies and demand a more equitable role in the management of natural resources. With thisenhanced capacity and leadership they will make more effective use of water, land and biodiversityresources, better access information and markets and continually adapt to a dynamic Khulna floodplainsystem. Making a difference in the lives of the poor
    • 13. 5. Hub Development ChallengesMalaita, Solomon Islands•NRM challenge – depleted reef resources, mangrove managementBarotse Floodplain, Zambia•NRM challenge – seasonal flood management, community canal management,uplands community forestry managementKhulna Floodplain, Bangladesh•NRM challenge – community polder management, salt water intrusion, landsubsidence Making a difference in the lives of the poor
    • 14. 5. AAS Intermediate Development OutcomesTime and level slicing – In X AAS Hubs:•Adoption outcomes –Y Households have adopted AAS technologies and practices•Sustainable Livelihoods Outcomes –Y communities with enhanced capital to respond tothe hub development challenge (HDC)•Local institutional change outcomes – an empowerment indicator (e.g. # of orgs utilizingresilience approaches to NRM challenges)•Hub-level institutional change – key stakeholders are supporting community action totackle HDC•CG system level institutional change – evidence of adoption of AAS thinking and practice Making a difference in the lives of the poor
    • 15. 6. Evaluation Questions••Did the intervention make a difference?•How has the intervention made a difference?•Will the intervention work elsewhere?Source: Stern et al. Broadening the Range of Designs and Methods for Impact Evaluations. 2012To what extent can a specific impact be attributed to the intervention? Making a difference in the lives of the poor
    • 16. Traditional IA question in CG Variations of question 1 To what extent can a specific impact be attributed to the intervention?•What is the net effect the intervention?•Did the intervention work?•How much impact can be attributed to the intervention?•What would have happened without the intervention? Making a difference in the lives of the poor
    • 17. What is the net effect of the intervention?Making a difference in the lives of the poor
    • 18. NRM Evaluation questionsQuestions that focus consideration of:•The continued rationale for the program•Implementation of the program•To what extent the program worked (had intended results)•Why the program works the way it does•Whether the program will continue to work Making a difference in the lives of the poor
    • 19. 7. Available DesignsDesign approaches•Experimental•Statistical•Theory-based•Case-based•Participatory•Synthesis studiesSource: SternMaking a difference in the lives of the poor et al. Broadening the Range of Designs and Methods for Impact Evaluations. 2012
    • 20. Table 2: Design, Variants and Causal InferenceDesigns Specific Variants Basis for Causal InferenceExperimental RCTs Counterfactuals; the co-presence of cause and Quasi Experiments, effects Natural ExperimentsStatistical Statistical Modelling Correlation between cause and effect or between Longitudinal Studies variables, influence of (usually) isolatable Econometrics multiple causes on a single effect Control for ‘confounders’Theory-based Causal process designs: Theory of Change, Process tracing, Identification/confirmation of causal processes Contribution Analysis, Impact Pathways, or ‘chains’, Causal mechanism designs: Realist evaluation, Congruence Supporting factors and mechanisms at work in analysis context‘Case-based’ Interpretative: Naturalistic, Comparison across and within cases of Grounded theory, Ethnography combinations of causal factors Structured: Configurations, QCA, Within-Case- Analysis, Analytic generalisation based on theory Simulations and network analysisParticipatory Normative designs: Participatory or democratic evaluation, Validation by participants that their actions and Empowerment evaluation, experienced effects are ‘caused’ by programme Agency designs: Learning by doing, Policy dialogue, Collaborative Adoption, customisation and commitment to a Action Research goalSynthesis studies Meta analysis, Narrative synthesis, Realist based synthesis Accumulation and aggregation within a number of perspectives (statistical, theory based, ethnographic etc.) Making a difference in the lives of the poor

    ×