Session 4 logic models and indicators

609 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
609
On SlideShare
0
From Embeds
0
Number of Embeds
71
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Session 4 logic models and indicators

  1. 1. Chris NicolettiActivity #267: Analysing the socio-economicimpact of the Water Hibah on beneficiaryhouseholds and communities (Stage 1)Impact EvaluationTraining CurriculumSession 4April 19, 2013
  2. 2. MEASURING RESULTSFrom Promises into EvidenceIMPACT EVALUATIONANDThis material constitutes supporting material for the "Impact Evaluation in Practice" book. This additional material is made freely but please acknowledge itsuse as follows: Gertler, P. J.; Martinez, S., Premand, P., Rawlings, L. B. and Christel M. J. Vermeersch, 2010, Impact Evaluation in Practice: Ancillary Material,The World Bank, Washington DC (www.worldbank.org/ieinpractice). The content of this presentation reflects the views of the authors and not necessarilythose of the World Bank.Some of the data collection management material was developed by Adam Ross
  3. 3. 3Tuesday - Session 1INTRODUCTION AND OVERVIEW1) Introduction2) Why is evaluation valuable?3) What makes a good evaluation?4) How to implement an evaluation?Wednesday - Session 2EVALUATION DESIGN5) Causal Inference6) Choosing your IE method/design7) Impact Evaluation ToolboxThursday - Session 3SAMPLE DESIGN AND DATA COLLECTION9) Sample Designs10) Types of Error and Biases11) Data Collection Plans12) Data Collection ManagementFriday - Session 4INDICATORS & QUESTIONNAIRE DESIGN1) Results chain/logic models2) SMART indicators3) Questionnaire DesignOutline: topics being covered
  4. 4. 4Using a Results ChainWhat are the intended results of theprogram?How will we achieve the intendedresults?How will we know we have achievedthe intended results?A Results Chain answers 3 questions:Impact Evaluation Training Curriculum - Activity 267
  5. 5. 5The Results Chain in a Typical Programo Budgeto Staffingo Trainingo Studieso Constructiono Training plancompletedo Cash transferdeliveredo Roadconstructedo School builto New practicesadoptedo Use of the roado School attendanceupo Health service useupo Poverty reducedo Income inequalityreducedo Labor productivityincreasedResults-based managementImplementation ResultsFinancial,human, andother resourcesmobilized tosupportactivities.Actions taken orwork performedto convert inputsinto specificoutputs.Projectdeliverableswithin the controlof implementingagencySUPPLY SIDE.Use of outputs bybeneficiaries andstakeholders outsidethe control ofimplementing agencyDEMAND SIDE.Changes inoutcomes that havemultiple drivers.INPUTS ACTIVITIES OUTPUTS OUTCOMESLONGER-TERMOUTCOMESHIGHER ORDERGOALS
  6. 6. 6Example 1: Results ChainActivities Outputs OutcomesLonger-termOutcomesEducationo Teachertrainingo Textbooksdevelopedo Teacherstrained in newmethodso Textbooksdeliveredo New methodsusedo Increasedcompletionrateso Increased testscoreso Increased laborproductivityHealtho Doctorshiredo Birthattendantstrainedo New doctorspracticingo Attendantsapplyingmethodso Increased useof healthclinics fordeliverieso ImprovedmaternalmortalitySocialProtectionand Laboro CCTsdeliveredo Targetingsystemo MISo CCTs deliveredto targethouseholds inaccordancewith conditionso Increased foodconsumptiono Increasedchild healthvisitso Decreasedpovertyo Lower childmortality
  7. 7. 7Identify the sequence of inputs, activities, outputsand outcomes:Example 2: Results Chain123456Information is available for parents about theimportance of breast feeding.Children in community healthier.Fewer children are having diarrheal diseases.Mothers breast feeding rather than using formula.New funds available to implement a health projectto reduce child malnutrition rates.Design information campaigns on the importanceof breast feeding
  8. 8. 8Identify the sequence of inputs, activities, outputsand outcomes:Example 2: Results Chain123456Information is available for parents about theimportance of breast feeding.Children in community healthier.Fewer children are having diarrheal diseases.Mothers breast feeding rather than using formula.New funds available to implement a health projectto reduce child malnutrition rates.Design information campaigns on the importanceof breast feedingInputActivityOutputOutcomeOutcomeOutcome
  9. 9. Additional LandAccess toIrrigationAnchor FarmerCost of Inputs# of Crops GrownValue of Crop MixAccess to MarketsAccess to AdditionalInputsIrrigation and Farm-managementtrainingContract Farmingand marketopportunitiesEfficiency ofIrrigation andfarming practicesAnchor FarmerEmploymentHousehold LaborEmploymentFarm Productionand OutputOn-Farm IncomeOff Farm IncomeFarm Gate PricesPrimary Inputs Secondary Inputs Intermediate Impacts OutcomesKeySolid Line indicates positiverelationshipDashed Line indicatesnegative relationshipOrange Line indicates thatthe relationship assumessignificant anchor farmerInvolvementExample #3: Irrigation Activity – Logic Model
  10. 10. 10
  11. 11. 11• Take moment to look at the Logic models that arein the M&E plan to identify the parts that aretestable. Would you please identify the ones thatare testable, using impact evaluation?• Remember, these are not processes and or monitoringefforts.• You are interested in the causal impact of an IndIIprogram on ________.Activity #2 : Review the logic modelsfrom the M&E PlanImpact Evaluation Training Curriculum - Activity 267
  12. 12. 12SMART: Identifyinggood indicatorsSpecificeasurablettributableargetedealisticMART
  13. 13. 13SpecificeasurableMMeasure as closely as possible what you want to know.Outcome: Children treated for malariaIndicators:1. Increased utilization of clinics2. Increased use of malaria drugsBe clear about how it will be measured – specific.Indicators:1. % of health centers without stocks of drugs x, y & z formore than a week at a time2. % of health centers with availability of drugsSource: Kathouri and Kusek, 2006
  14. 14. 14ealisticRData obtainable at reasonable cost, frequency and accuracy.Indicators:1. HIV prevalence among 15-24 year-oldpregnant women2. HIV prevalence among the total populationSource: Kathouri and Kusek, 2006ttributableALogically and closely linked to a program’s efforts.Indicators:1. Life expectancy2. % of children fully immunized at 1 year
  15. 15. 15argetedTSpecific to the program’s target group.Indicators:1. % increase in employment2. % increase in employment of graduates oftechnical training center X, in the first yearafter completion of trainingSource: Kathouri and Kusek, 2006
  16. 16. 16
  17. 17. 17• Are the components of the logic model that aretestable using SMART indicators? How might they berefined for the purposes of impact evaluation?• Focus on the testable part of the logic model that youidentified earlier.• Think about the actual complexities of how you wouldaccurately and reliably measure your outcome of interest.• Be creative and think about your own life… !Activity #2 : Come up with a set ofSMART indicators for your projectImpact Evaluation Training Curriculum - Activity 267
  18. 18. 18• What do you want to know?• Tailor your survey to capture outcome indicators ofinterest  Those that are suitable for impactevaluation.• Take the time, before you start developing yourquestionnaire, to list EVERY indicator you areinterested in measuring, and every theoreticalcharacteristic that could affect those indicators.– This should come from your logic model!• Know how you will do your analysis!• Don’t figure out how to do your analysis after thefact… start early and have a plan in place.Before you begin working onthe questionnaire…
  19. 19. 19• Use reliable and valid instruments.• If you are going to have surgery on your knee do youwant the doctor to use this… or this…?Use valid instruments…
  20. 20. 20• Often times, we will use existing surveyinstruments that we know worked well.• We will modify them for the cultural context that we areworking in.• Adapt the instrument to the local language.• A good question is one where the respondentshave the necessary knowledge to answer• A good question is one where the respondent iswilling to provide the “true” answer.• This is can be difficult with health- or income-relatedquestions.Building of reliableinstruments.
  21. 21. 21• Ask about first hand experience• Ask one question at a time.• Bad: Are you physically able to do things like walk orcarry a full water bucket without difficulty?• Better: Are you physically able to carry a full waterbucket without difficulty?• Obtain specific information, but do it simply(example: total expenditures)• Bad: How much do you spend per month on water?• Better: How much do you spend on a single containerof water? How many containers of water do you buyper week?Developing a survey…
  22. 22. 22• Use open questions for frequencies ofundesirable behavior.• Use familiar words  important for healthquestions.• For socially-undesirable questions ask about thepast, before asking about the present.• 1) Ask “have you ever done ______?”• 2) “Are you currently doing ______?”• Put sensitive questions in between less sensitivequestions, so the respondent does not getoverwhelmed.Sensitive questions…
  23. 23. 23• There are 4 types of questions• Numeric• Pre-coded– Read the response options to the respondent• Field-coded– Do not read the response options to the respondent• Verbatim– Completely open-endedTYPES OF QUESTIONS YOUCAN USE…
  24. 24. 24TYPES OF QUESTIONSExample: On average How many days per week does [NAME] collect water fordrinking, cooking or bathing currently?|__|Example 2: What water source do you use most often for drinking?1. WATER VENDOR (CART, WATER TRUCK, ETC.)2. REFILL KIOSK3. PUBLIC/COMMUNITY TAPExample: What is your family name?_________________________________________What is each type of question?NumericPre-CodedField-CodedVerbatimE1.2. Does the plot belong to your HH or has it been rented / borrowed?1 = belongs to HH2 = rented3 = borrowed
  25. 25. 25• Lowercase vs. UppercaseQUESTION CONVENTIONSWhat treatments did you receive?SEPARATE MULTIPLE RESPONSES WITH COMMASSEE CODESExample of Field-Coded question:How many days per week would [NAME] collect water for drinking,cooking or bathing on average in January 2010?-88 = DO NOT KNOW-99 = NO RESPONSEExample of Pre-Coded question:What symptoms did [NAME] experience?1 Abdominal Pain2 Frequent Evacuation3 Watery Feces4 Bloody Feces5 Vomiting-99 NO RESPONSEREAD THELOWERCASELETTERSDO NOTREAD THEUPPERCASELETTERS
  26. 26. 26• Parentheses• Square brackets• Underlined wordsQUESTION CONVENTIONSExample:How old was [NAME] on (his/her) last birthday?Example:Is [NAME] responsible for collecting water for drinking, cooking orbathing from a source outside of the home currently?Example:How many days per week would [NAME] collect water for drinking,cooking or bathing on average in January 2010?
  27. 27. 27• We use skip instructions to filter out irrelevant questions• Skip patterns must be followed correctly• Asking questions in a tableSKIP INSTRUCTIONSMember IDBASIC DEMOGRAPHICS OF HOUSEHOLD MEMBERSB3-A B3-B B3-C B3-D B3-E B3-F B3-G B3-HPlease tell methe name ofeach of themembers of thishousehold,starting withyourself.After yourself,please list oldestto youngest.What is[HOUSEHOLDMEMBERS]’s sex?1 Male2 FemaleWhat is[NAME]’srelationship to thehead ofhousehold?SEECODESHow old is[NAME]?Years-99 = NoResponse(IF B3-D>13years)Was [NAME]employed inJanuary 2010?1=Yes2=No  B3-G-88 = Do notknow-99 = NoResponse(IF B3-D >13years)What type ofemploymentdid [NAME]Have in January2010?SEE CODES(IF B3-D>11years)Is [NAME]employedcurrently?1=Yes2=No  B3-I-88 = Do notknow-99 = NoResponse(IF B3-D >11years)What type ofemploymentdoes [NAME]Havecurrently?SEE CODES01 |_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_|02 |_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_|D7-A Does your household have a piped water connection from the PDAM?1 YES2 NO  SKIP TO D7-B|_|
  28. 28. 28• It also helps to be aware of other impactevaluations going on in your sector. You canlearn from their results when writing your survey.• At NORC we have people whose entire career iswriting reliable questionnaires.• There are a lot of biases that come from poorquestionnaires.• Instrument bias – when the instrument you are usingdoes not capture the information you intend to capture.This takes a lot of practice…
  29. 29. TEST YOUR QUESTIONNAIRE!It saves you from having dodgy data!
  30. 30. 30• Evaluate your questions – including the codingthat you use.• Focus groups• Cognitive Testing• Think-aloud interviews• Evaluate your questionnaire and procedures –especially the timing of the survey and data flows.• Conduct full field pretest.• Revise questionnaire and proceduresQuestionnaire testing
  31. 31. 31
  32. 32. 32• Using one of the SMART indicators that you identifiedpreviously, I would like for us to come up with a seriesof questions to capture the information necessary forthis indicator.• Think about question format – categorical, continuous,verbatim, etc.• Write out the questions and make sure they capture theinformation you Intend. Are they confusing? Are theyambiguous?• Is it a complex indicator? If so, how will you break it intopieces so the respondent can answer more easily?– Think about your own ability to answer these questions.Activity #3 : QuestionnairedevelopmentImpact Evaluation Training Curriculum - Activity 267
  33. 33. 33Tuesday - Session 1INTRODUCTION AND OVERVIEW1) Introduction2) Why is evaluation valuable?3) What makes a good evaluation?4) How to implement an evaluation?Wednesday - Session 2EVALUATION DESIGN5) Causal Inference6) Choosing your IE method/design7) Impact Evaluation ToolboxThursday - Session 3SAMPLE DESIGN AND DATA COLLECTION9) Sample Designs10) Types of Error and Biases11) Data Collection Plans12) Data Collection ManagementFriday - Session 4INDICATORS & QUESTIONNAIRE DESIGN1) Results chain/logic models2) SMART indicators3) Questionnaire DesignOutline: topics being covered
  34. 34. Thank You!
  35. 35. MEASURING RESULTSFrom Promises into EvidenceIMPACT EVALUATIONANDThis material constitutes supporting material for the "Impact Evaluation in Practice" book. This additional material is made freely but please acknowledge itsuse as follows: Gertler, P. J.; Martinez, S., Premand, P., Rawlings, L. B. and Christel M. J. Vermeersch, 2010, Impact Evaluation in Practice: Ancillary Material,The World Bank, Washington DC (www.worldbank.org/ieinpractice). The content of this presentation reflects the views of the authors and not necessarilythose of the World Bank.Some of the data collection management material was developed by Adam Ross

×