Approach To and Findings From Farming Practices Survey


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Approach To and Findings From Farming Practices Survey

  1. 1. Approach To and Findings From Farming Practices Survey October 17, 2008
  2. 2. Overview of Presentation <ul><li>How an Impact Evaluation Works </li></ul><ul><li>Evaluation Design for Water-to-Market Training </li></ul><ul><li>About the Farming Practices Survey </li></ul><ul><li>Findings from the Farming Practices Survey </li></ul>
  3. 3. Impact Evaluation <ul><li>Was the program effective? </li></ul><ul><li>Estimating impacts involves comparing: </li></ul><ul><ul><li>Outcomes with the program with outcomes if there were no program </li></ul></ul><ul><li>Counterfactual: What participants would have experienced if there were no program </li></ul><ul><ul><li>True counterfactual is not directly observed </li></ul></ul><ul><li>Goal of impact study is to identify a comparison group to approximate the counterfactual </li></ul>
  4. 4. Importance of the Counterfactual: An Illustrative Example <ul><li>Rural poverty has decreased in recent years, prior to MCA-Armenia programs </li></ul><ul><li>In this case, we want to see how much more did poverty decline because of MCA-Armenia? </li></ul>
  5. 5. How Much More Did Poverty Decline Because of MCA-Armenia? Impact
  6. 6. Random Assignment: Often Considered the “Gold Standard” <ul><li>Randomly assign into two groups, similar to a lottery </li></ul><ul><li>Program and comparison groups are the same on average, except one group has access to program </li></ul><ul><ul><li>Subsequent differences in outcomes can be attributed to the program </li></ul></ul><ul><li>Random assignment most feasible when there are limited resources/excess demand </li></ul>
  7. 7. II. Evaluation for Water-to-Market Training
  8. 8. Water-to-Market Activities <ul><li>Focus on evaluation of training programs </li></ul><ul><ul><li>Water management techniques </li></ul></ul><ul><ul><li>Transition to higher value agriculture </li></ul></ul><ul><li>ACDI/VOCA providing training </li></ul><ul><ul><li>Primarily in Compact Years 2-5 </li></ul></ul>
  9. 9. Evaluation Design for Water-to-Market Training <ul><li>Ideally, would want to randomly assign farmers who apply for training </li></ul><ul><li>Not practically feasible or politically viable </li></ul><ul><ul><li>New program starting, so no excess demand </li></ul></ul><ul><ul><li>Intervention at the village level (potential for spillover/contamination) </li></ul></ul><ul><li> </li></ul><ul><li>We randomly assigned villages to program or “control” group </li></ul><ul><ul><li>Use a phased-in approach </li></ul></ul>
  10. 10. Shortcomings of the Approach <ul><li>Less statistical power to detect impacts than random assignment at the individual level </li></ul><ul><ul><li>Not all farmers in a village may want to participate in training </li></ul></ul><ul><ul><li>Clustering effects (village specific effects) </li></ul></ul><ul><li>Challenge can be overcome to some extent if: </li></ul><ul><ul><li>Take-up rates of training high </li></ul></ul><ul><ul><li>More villages included </li></ul></ul><ul><li> </li></ul>
  11. 11. Random Assignment Design <ul><li>Randomly assign when training will start in each village, with three groups: </li></ul><ul><ul><li>Compact Year 2 </li></ul></ul><ul><ul><li>Compact Years 3 and 4 </li></ul></ul><ul><ul><li>Compact Year 5 </li></ul></ul><ul><li>Compare Compact Year 2 villages to Compact Year 5 villages </li></ul>
  12. 12. Random Selection of Villages (Implemented in August 2007) <ul><li>Started with a list of villages with “good water” and which could benefit from training </li></ul><ul><ul><li>Excluded villages served in pilot phase </li></ul></ul><ul><li>Random selection conducted publicly to: </li></ul><ul><ul><li>Ensure transparency </li></ul></ul><ul><ul><li>Allow for greater accountability and implementation fidelity </li></ul></ul>
  13. 13. Selection of Villages <ul><li>Total of 277 village clusters were assigned </li></ul><ul><ul><li>Year 2: 120 clusters </li></ul></ul><ul><ul><li>Years 3 and 4: 77 clusters </li></ul></ul><ul><ul><li>Year 5: 80 clusters </li></ul></ul><ul><li>Selection of clusters stratified by WUAs </li></ul><ul><ul><li>In proportion to the number of villages in the WUA </li></ul></ul><ul><li>To ensure balance across treatment and control villages </li></ul><ul><ul><li>Also for political reasons </li></ul></ul>
  14. 14. Selection of Villages (cont’d) <ul><li>No. of Villages By Year of Training and Ag. Zone </li></ul>
  15. 15. Overall Approach to the Impact Evaluation <ul><li>Compare outcomes for farmers in treatment and control villages </li></ul><ul><li>Examine impacts on key outcomes </li></ul><ul><ul><li>Participation in training and adoption of new practices </li></ul></ul><ul><ul><li>Changing crop patterns, improved yields, and increases in income </li></ul></ul><ul><li>Data and Sample </li></ul><ul><ul><li>Survey of farmers in the treatment and control villages </li></ul></ul><ul><ul><li>Sample of 5,000 farmers to detect impact on poverty of 5 percentage points </li></ul></ul>
  16. 16. About the Farming Practices Survey
  17. 17. Farming Practices Survey <ul><li>Baseline survey conducted by AREG </li></ul><ul><li>Implemented during Nov 2007- Feb 2008 </li></ul><ul><ul><li>In 223 communities </li></ul></ul><ul><ul><li>Target sample of 5,000 farmers </li></ul></ul><ul><li>Topics covered in the survey </li></ul><ul><ul><li>Demographic characteristics </li></ul></ul><ul><ul><li>Agricultural practices and productivity </li></ul></ul><ul><ul><li>Income and consumption </li></ul></ul>
  18. 18. Identifying the Sample Frame <ul><li>Sample frame is critical as it defines what population the study represents </li></ul><ul><ul><li>Need comparable frame for treatment and control villages </li></ul></ul><ul><ul><li>Want persons who farm as their main occupation </li></ul></ul><ul><li>No viable sample frame exists for the surveys </li></ul><ul><ul><li>Originally attempted to use WUA member lists </li></ul></ul><ul><ul><li>Worked with WUA heads to get names of farmers </li></ul></ul><ul><ul><li>During pretest, AREG found some lists were bad </li></ul></ul><ul><ul><li>All lists had to be verified by AREG in the field and developed as necessary </li></ul></ul>
  19. 19. Implementing the Survey and Checking Data Quality <ul><li>Interactive interviewer training (Q by Q, etc) </li></ul><ul><li>Piloting of both survey instrument and sample list </li></ul><ul><ul><li>Revisions to questionnaire </li></ul></ul><ul><ul><li>Sample verification process </li></ul></ul><ul><li>Regular generation of field reports </li></ul><ul><li>MCA did field visits to review processes </li></ul><ul><li>Independent review of initial batches of data </li></ul>
  20. 20. Planned Improvements to the Upcoming Survey <ul><li>More use of photos in training interviewers on farming practices </li></ul><ul><li>Pretesting both survey instrument and sample list </li></ul><ul><li>Double key data entry </li></ul>
  21. 21. Baseline FPS Survey <ul><li>Completed interviews with 4,855 farmers </li></ul><ul><li>Sample focuses on households farming for five years or more in these communities </li></ul><ul><li>Sample not representative of all farmers in Armenia </li></ul><ul><li>However, sample internally valid for study </li></ul><ul><ul><li>Will show comparison of farmers in treatment and control villages later </li></ul></ul>
  22. 22. IV. Baseline Findings <ul><li>Farmer and Household Characteristics </li></ul><ul><li>Crop Cultivation and Sales </li></ul><ul><li>Income and Poverty </li></ul>
  23. 23. General Approach to Analysis and Data Cleaning <ul><li>Descriptive analysis </li></ul><ul><ul><li>Averages and distributions </li></ul></ul><ul><ul><li>Examined conditional and unconditional measures </li></ul></ul><ul><ul><li>Used weights to adjust for treatment-control balance </li></ul></ul><ul><li>Data checks </li></ul><ul><ul><li>Missing values and outliers </li></ul></ul><ul><ul><li>Logical skips </li></ul></ul><ul><ul><li>Internal consistency </li></ul></ul><ul><li>Measuring income and consumption is challenging </li></ul><ul><ul><li>Sensitivity tests </li></ul></ul><ul><ul><li>Looking at distributions critical </li></ul></ul>
  24. 24. Farmer and Household Characteristics
  25. 25. Farmer and Household Characteristics <ul><li>Characteristics (Percentages) </li></ul>
  26. 26. Farmer and Household Characteristics (cont’d) <ul><li>Land Cultivated by Respondents (Sq. Meters) </li></ul>
  27. 27. Few Farmers Use Water Management Practices <ul><li>Respondents’ Irrigation Practices </li></ul><ul><li>(Percentages) </li></ul>
  28. 28. Crop Cultivation and Sales
  29. 29. Crop Cultivation and Sales <ul><li>Respondents Growing and Selling </li></ul><ul><li>Crops (Overall Percentages) </li></ul>
  30. 30. Crop Cultivation and Sales Vary by Agricultural Zone <ul><li>Farmers in Ararat Valley grow more fruit, and much more likely to sell their fruits and vegetables </li></ul><ul><li>Farmers in the Mountainous Zone grow much more grain and potatoes </li></ul><ul><li>Mountainous Zone farmers sell very little </li></ul>
  31. 31. Income and Poverty
  32. 32. How to Measure Agricultural Income? <ul><li>Crop sales tell us monetary income </li></ul><ul><li>Much of production is consumed by household; not reflected in sales </li></ul><ul><li>Assign monetary value to crops consumed by household </li></ul><ul><li>Subtract agricultural costs to get profit </li></ul>
  33. 33. Crop Sales and Value of Production <ul><li>Respondents’ Average Crop Sales and Values (AMD) </li></ul>
  34. 34. Measuring Income <ul><li>Survey is relatively short (30 minutes) </li></ul><ul><li>Focus on agricultural production, but learn about non-agricultural income/consumption as well </li></ul><ul><li>Income measures include employment income, pensions, remittances, other benefits </li></ul><ul><li>Consumption measures include groceries, household products, utilities, transportation </li></ul>
  35. 35. Household Income, Only Considering Monetary Income <ul><li>Annual Household Monetary Income (AMD) </li></ul>
  36. 36. Household Income, Including Value of Crops Consumed <ul><li>Annual Household Economic Income (AMD) </li></ul>
  37. 37. Measuring Poverty <ul><li>Consumption-based measure </li></ul><ul><li>Adopt similar approach as NSS/World Bank </li></ul><ul><li>Food poverty line: Meets minimal caloric needs </li></ul><ul><li>Complete poverty line: Allowance for other basic necessities </li></ul>
  38. 38. Estimates of Poverty Rates <ul><li>Respondent Households in Poverty (Percentages) </li></ul>
  39. 39. Poverty Varies Appreciably by Zone <ul><li>Respondent Households in Poverty by Zone (Percentages) </li></ul>
  40. 40. Many Households are Near Poverty Line <ul><li>Consumption Relative to Complete Poverty Line (CPL) </li></ul>
  41. 41. Values of Variables by Treatment Status
  42. 42. Demographic Characteristics and Land Size of Sample Members <ul><li> Treatment Control </li></ul><ul><li>Respondent’s Age (years) 49.2 49.2 </li></ul><ul><li>Female Respondent 13.0 10.8 </li></ul><ul><li>Respondents Education </li></ul><ul><li>Less than Secondary 13.8 13.4 </li></ul><ul><li>Full Secondary 40.6 41.1 </li></ul><ul><li>Secondary Vocational 28.9 27.1 More than Secondary 16.7 18.4 </li></ul><ul><li>Area of land cultivated (ha) 1.99 1.98 </li></ul><ul><li>Kitchen plot size (ha) .171 .173 </li></ul><ul><li>Use varying furrow spacing 7.5 7.4 </li></ul>
  43. 43. Crop Cultivation and Sales and Income for Sample Members <ul><li> Treatment Control </li></ul><ul><li>Percent Cultivating </li></ul><ul><li>Grain 45.7 43.5 </li></ul><ul><li>Fruit 76.3 77.6 Vegetable 33.1 31.7 </li></ul><ul><li>Revenue from Crops Sold (AMD) </li></ul><ul><li>Grain 21,711 32,531 </li></ul><ul><li>Fruit 317,619 206,355 Vegetable 88,858 57,653 </li></ul><ul><li>Market Value of Harvest </li></ul><ul><li>Grain 112,784 108,144 </li></ul><ul><li>Fruit 332,365 282,692 Vegetable 110,924 142,928 </li></ul>
  44. 44. Household Income and Poverty <ul><li> Treatment Control </li></ul><ul><li>Household Income (AMD) </li></ul><ul><li>Nonagricultural Income 674,780 630,414 </li></ul><ul><li>Total Monetary Income 929,592 708,041** </li></ul><ul><li>Total Economic Income 1,115,105 1,036,766 </li></ul><ul><li>Poverty </li></ul><ul><li>Complete Poverty Rate 17.9% 18.7% </li></ul><ul><li>Consumption Relative to CPL 238% 237% </li></ul>
  45. 45. Lessons Learned and Plans for the Future <ul><li>Modifications to the questionnaire </li></ul><ul><ul><li>More detailed measure of employment income </li></ul></ul><ul><ul><li>Consistent units for crop production </li></ul></ul><ul><li>Enhancements to data collection procedures </li></ul><ul><ul><li>Use pictures of irrigation practices </li></ul></ul><ul><ul><li>Focus on high response rates </li></ul></ul><ul><li>More focus on longitudinal analysis </li></ul><ul><ul><li>Useful to factor out preexisting differences </li></ul></ul><ul><li>Final impact report (2011) </li></ul><ul><ul><li>Possibility of interim report </li></ul></ul>