VALUE CHAIN MONITORING  AND EVALUATION GUIDEMODULE11              Evaluations and Value Chain                        Proje...
GETTING               STARTED1May 22, 2012
GETTING               STARTED2May 22, 2012
GETTING                                STARTED         IMPACT                   PERFORMANCE                         PROCES...
GETTING                                STARTED         IMPACT                   PERFORMANCE                         PROCES...
GETTING               STARTED5May 22, 2012
STEP-BY-STEP GUIDE    1               • Determine the Purpose for the Evaluation               • Determine the Financial R...
STEP-BY-STEP GUIDE:               STEP 1                 FIRST!Ask yourself what is thepurpose of this evaluation andwho i...
STEP-BY-STEP GUIDE:               STEP 1            Other Angles to Consider    •Typically, there are 3 M&E clients that  ...
STEP-BY-STEP GUIDE    1               • Determine the Purpose for the Evaluation               • Determine the Financial R...
STEP-BY-STEP GUIDE:                    STEP 2                             Evaluation Costs•More rigorous evaluations = Mor...
STEP-BY-STEP GUIDE:               STEP 2                        WA R N I N G !               Best practice evaluation stan...
STEP-BY-STEP GUIDE     1               • Determine the Purpose for the Evaluation               • Determine the Financial ...
STEP-BY-STEP GUIDE:                STEP 3      Activities and Responsibilities for External Research                      ...
STEP-BY-STEP GUIDE:                 STEP 3               Proposals from Potential Research                            Part...
STEP-BY-STEP GUIDE     1               • Determine the Purpose for the Evaluation               • Determine the Financial ...
STEP-BY-STEP GUIDE:               STEP 4               How to ID Questions•Benefit from research partner’sknowledge and ex...
STEP-BY-STEP GUIDE     1               • Determine the Purpose for the Evaluation               • Determine the Financial ...
STEP-BY-STEP GUIDE:               STEP 5 Creating a Comparison Group•Must be a group of farmers,entrepreneurs, business ow...
STEP-BY-STEP GUIDE:                    STEP 5                   2 Sources of Selection                   OBSERVABLE       ...
STEP-BY-STEP GUIDE:                 STEP 5                Experimental Methods of                      Evaluation•Follows ...
STEP-BY-STEP GUIDE:                STEP 5        Downsides of Experimental                Method•Randomization protocols c...
STEP-BY-STEP GUIDE:                 STEP 5                Quasi-Experimental Methods•Does not randomly assign subjects int...
STEP-BY-STEP GUIDE:                STEP 5In choosing your method, ask the following:•Will our M&E system clients be less w...
STEP-BY-STEP GUIDE     1               • Determine the Purpose for the Evaluation               • Determine the Financial ...
STEP-BY-STEP GUIDE:                STEP 6                Other Considerations•Sample size and composition•Trend study vs. ...
STEP-BY-STEP GUIDE     1               • Determine the Purpose for the Evaluation               • Determine the Financial ...
STEP-BY-STEP GUIDE:               STEP 7                  The Final Act        •Implement the evaluation        •Work clos...
Resources                   Impact Evaluation Resources                                                        Donor Organ...
COMMON                            PITFALLS•Teams do not conduct appropriate      •Teams do not seek advice ondue diligence...
QUESTIONS                   ?               COMMENTS                  ?30May 22, 2012
Want to Learn More?Multiple ways to continue the discussion and continue learning:• Initiate a monthly session on the M&E ...
Upcoming SlideShare
Loading in...5
×

VC M&E Module 11 - Evaluations and Value Chain Projects

1,015

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,015
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
50
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Explain from warning box on page 91
  • Explained in depth on page 95
  • Explained in depth on page 95
  • VC M&E Module 11 - Evaluations and Value Chain Projects

    1. 1. VALUE CHAIN MONITORING AND EVALUATION GUIDEMODULE11 Evaluations and Value Chain Projects
    2. 2. GETTING STARTED1May 22, 2012
    3. 3. GETTING STARTED2May 22, 2012
    4. 4. GETTING STARTED IMPACT PERFORMANCE PROCESSAim is to determine if Also aims to determine if Aim to assess whether andchanges have taken place changes have taken place to what degree projectsin the VC or among VC in the VC or among VC have been implemented inactors actors. line with initial plan.To what degree those To what degree CARE’s Does not consider resultschanges can be attributed interventions contributed to directly, but how theto CARE’s work. those changes. initiative is managed.Designed to answer what Use non experimental Typically internal.would have happened if designs.CARE had not intervened. Assesses timeliness and Typically only gather and quality of performance.Answered via a statistically analyze data from thosevalid counterfactual using directly engaged or Looking to ID areas forcontrol groups and impacted. improvement to enhanceexperimental or quasi-3 implementation process.experimental designs.May 22, 2012 Lower cost, but less rigorous.
    5. 5. GETTING STARTED IMPACT PERFORMANCE PROCESSAim is to determine if Also aims to determine if Aim to assess whether andchanges have taken place changes have taken place to what degree projectsin the VC or among VC in the VC or among VC have been implemented inactors actors. line with initial plan.To what degree those To what degree CARE’s Does not consider resultschanges can be attributed interventions contributed to directly, but how theto CARE’s work. those changes. initiative is managed.Designed to answer what Use non experimental Typically internal.would have happened if designs.CARE had not intervened. Assesses timeliness and Typically only gather and quality of performance.Answered via a statistically analyze data from thosevalid counterfactual using directly engaged or Looking to ID areas forcontrol groups and impacted. improvement to enhanceexperimental or quasi-4 implementation process.experimental designs.May 22, 2012 Lower cost, but less rigorous.
    6. 6. GETTING STARTED5May 22, 2012
    7. 7. STEP-BY-STEP GUIDE 1 • Determine the Purpose for the Evaluation • Determine the Financial Resources Available for the 2 Evaluation 3 • Identify Research Team and Partners 4 • Identify Research Questions 5 • Choose a Research Methodology 6 • Determine the Other Details of the Research Design6 7 • Implement the Impact EvaluationMay 22, 2012
    8. 8. STEP-BY-STEP GUIDE: STEP 1 FIRST!Ask yourself what is thepurpose of this evaluation andwho is it for?Not asking this can result in amethodology poorly matchingdonor requirements and cancost precious time, money andenergy.7May 22, 2012
    9. 9. STEP-BY-STEP GUIDE: STEP 1 Other Angles to Consider •Typically, there are 3 M&E clients that might want an impact evaluation •Motivations for conducting an impact evaluation •Impact evaluations are not always the right choice.8May 22, 2012
    10. 10. STEP-BY-STEP GUIDE 1 • Determine the Purpose for the Evaluation • Determine the Financial Resources Available for 2 the Evaluation 3 • Identify Research Team and Partners 4 • Identify Research Questions 5 • Choose a Research Methodology 6 • Determine the Other Details of the Research Design9 7 • Implement the Impact EvaluationMay 22, 2012
    11. 11. STEP-BY-STEP GUIDE: STEP 2 Evaluation Costs•More rigorous evaluations = More $$•Attributable evidence is expensive• Cost depends on: Sample size # of research rounds Survey length Sampling methodology Geographic dispersion of respondents International evaluation experts10 Price of local researchMay 22, 2012 talent
    12. 12. STEP-BY-STEP GUIDE: STEP 2 WA R N I N G ! Best practice evaluation standards strongly recommend outsourcing impact evaluations.11May 22, 2012
    13. 13. STEP-BY-STEP GUIDE 1 • Determine the Purpose for the Evaluation • Determine the Financial Resources Available for the 2 Evaluation 3 • Identify Research Team and Partners 4 • Identify Research Questions 5 • Choose a Research Methodology 6 • Determine the Other Details of the Research Design12 7 • Implement the Impact EvaluationMay 22, 2012
    14. 14. STEP-BY-STEP GUIDE: STEP 3 Activities and Responsibilities for External Research Partners Refining Sharpening Translating Pilot testingEvaluation Research research research Design Questions instruments instruments Developing Training Managing Enteringthe research survey the field data results into instruments enumerators collection data shellCleaning the Transcripts Data Final reports data set of interviews analysis 13 May 22, 2012
    15. 15. STEP-BY-STEP GUIDE: STEP 3 Proposals from Potential Research Partners •Receiving proposals •Evaluating proposals •Evaluation and selection criteria •World Bank guide14May 22, 2012
    16. 16. STEP-BY-STEP GUIDE 1 • Determine the Purpose for the Evaluation • Determine the Financial Resources Available for the 2 Evaluation 3 • Identify Research Team and Partners 4 • Identify Research Questions 5 • Choose a Research Methodology 6 • Determine the Other Details of the Research Design15 7 • Implement the Impact EvaluationMay 22, 2012
    17. 17. STEP-BY-STEP GUIDE: STEP 4 How to ID Questions•Benefit from research partner’sknowledge and experience•Questions should measure critical linksand associated key performanceindicators•Goal is to verify results•Involve the team and M&E clients early•Check USAID publications16May 22, 2012
    18. 18. STEP-BY-STEP GUIDE 1 • Determine the Purpose for the Evaluation • Determine the Financial Resources Available for the 2 Evaluation 3 • Identify Research Team and Partners 4 • Identify Research Questions 5 • Choose a Research Methodology 6 • Determine the Other Details of the Research Design17 7 • Implement the Impact EvaluationMay 22, 2012
    19. 19. STEP-BY-STEP GUIDE: STEP 5 Creating a Comparison Group•Must be a group of farmers,entrepreneurs, business owners,etc. as similar as possible to theactual project beneficiaries•AKA Control group vs. Treatmentgroup•Isolates different impacts18May 22, 2012
    20. 20. STEP-BY-STEP GUIDE: STEP 5 2 Sources of Selection OBSERVABLE CHARACTERISTICS Bias of an individual’s personality UNOBSERVABLE CHARACTERISTICS •Aspects•Include things that can be seen or that play a large role in determiningtangibly measured success •Personal initiative, entrepreneurial spirit,•Sex, education, age, location, etc. risk orientation, persistence, self-•If treatment group is 90% male / confidence, optimism, etc.10% female and control group is •Those who volunteer for VC projects40% male / 60% female, you will have more of these qualities than otherscome up with invalid conclusions. •Comparing a group of new-seed•Educated vs. uneducated adopters to a group of non-adopters•Urban vs. rural would not allow us to know to what extent any observed differences in farming outcomes are the result of the project of the result of pre-existing personality 19 differences among the groups May 22, 2012
    21. 21. STEP-BY-STEP GUIDE: STEP 5 Experimental Methods of Evaluation•Follows the same basic approach as a placeboexperiment•Of a selected group of maize farmers, somereceive project assistance while others do not•Theoretically eliminates all sources of selectionbias•Also referred to as randomized controlled trials 20(RCTs) May 22, 2012
    22. 22. STEP-BY-STEP GUIDE: STEP 5 Downsides of Experimental Method•Randomization protocols can be complicated, time consumingand operationally burdensome•May be perceived as unethical•Not ideal for projects with small #’s of beneficiaries, impromptuprojects, specified locations or groups of people, or projectswith no available control group (broad-based policy reform)•VC projects are flexible, easily changed, while thismethodology requires consistent variables•Difficult for evaluation designers to reasonably ‘control for’changes in the environment that was not influenced by the 21 May 22, 2012project.
    23. 23. STEP-BY-STEP GUIDE: STEP 5 Quasi-Experimental Methods•Does not randomly assign subjects into treatment and controlgroups•Instead, compares pre-existing groups via a matching process•Treatment groups are selected via random sampling•Control groups are selected by ID’ing areas and communitieswith matching observablecharacteristics and then randomlysampling the relevant populationliving in those areas andcommunities.•But, quasi-experimental methodsare less rigorous thanexperimental 22 May 22, 2012
    24. 24. STEP-BY-STEP GUIDE: STEP 5In choosing your method, ask the following:•Will our M&E system clients be less well served if we opt for aquasi-experimental design over an experimental design?•Is our project amenable to random assignment?•Is random assignment operationally feasible?•Can we manage/overcome the anticipated opposition from ourproject staff and external stakeholders?•Is the tradeoff of an increased operational burden worth theimprovement we get in statistical credibility? If ‘Yes’ to each, then experimental If ‘No’ to any, then quasi- 23 May 22, 2012 experimental
    25. 25. STEP-BY-STEP GUIDE 1 • Determine the Purpose for the Evaluation • Determine the Financial Resources Available for the 2 Evaluation 3 • Identify Research Team and Partners 4 • Identify Research Questions 5 • Choose a Research Methodology • Determine the Other Details of the Research 6 Design24 7 • Implement the Impact EvaluationMay 22, 2012
    26. 26. STEP-BY-STEP GUIDE: STEP 6 Other Considerations•Sample size and composition•Trend study vs. Panel study•Single method vs. Mixed methods•Early vs. Delayed Baseline DataCollection 25 May 22, 2012
    27. 27. STEP-BY-STEP GUIDE 1 • Determine the Purpose for the Evaluation • Determine the Financial Resources Available for the 2 Evaluation 3 • Identify Research Team and Partners 4 • Identify Research Questions 5 • Choose a Research Methodology 6 • Determine the Other Details of the Research Design26 7 • Implement the Impact EvaluationMay 22, 2012
    28. 28. STEP-BY-STEP GUIDE: STEP 7 The Final Act •Implement the evaluation •Work closely with the local research firm, project staff and (as relevant) implementing partners and local authorities/community leaders •Assign one person specific task of monitoring the research firm’s performance at every stage27May 22, 2012
    29. 29. Resources Impact Evaluation Resources Donor Organizations Evaluation Firms •International Program for Development Evaluation•Abdul Latif Jameel Poverty Action TrainingLab •United Nations Evaluation Group•Innovations for Poverty Action •USAID Private Sector Development Impact•International Food Policy Research Assessment InitiativeInstitute (IFPRI) Impact Assessment •World Bank Development Impact Evaluation InitiativeProgram •World Bank Independent Evaluation Group Associations and Networks•American Evaluation Association Web Resources•Donor Committee for Enterprise Development •Evaluation Portal•InterAction Monitoring & Evaluation •Evaluation Virtual Library•International Initiative for Impact Evaluation •Free Resource for Program•Network of Networks on Impact Evaluation Evaluation and Social Research Methods 28 May 22, 2012
    30. 30. COMMON PITFALLS•Teams do not conduct appropriate •Teams do not seek advice ondue diligence about their evaluation sampling from qualified technicaloptions experts•Teams implement the baseline data •Teams inappropriately attributecollection too soon evaluation findings•Teams implement a trend study •Teams attempt to implement thewhen a panel study would have impact evaluation using project staffbeen both preferable and possible •Donors demand rigorous•Teams load up the impact survey evaluations but do not allocatewith excess questions sufficient funding•Teams do not monitor the local •Projects make compromises to theresearch firm’s adherence to the evaluation methodologyTOR •Evaluation reports do not fully•Teams do not budget or plan for disclose the tradeoffs mademixed-methods evaluations •Projects do not closely monitor the performance of external research firms29May 22, 2012
    31. 31. QUESTIONS ? COMMENTS ?30May 22, 2012
    32. 32. Want to Learn More?Multiple ways to continue the discussion and continue learning:• Initiate a monthly session on the M&E guide and case studies from across CARE. Contact cpennotti@care.org• Join the Market Engagement Community of Practice on LinkedIn.• Join a task force to review and refine the universal 31 indicators. Contact nardi@careinternational.org May 22, 2012
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×