IFPRI -AOS Survey in Bhutan

844 views

Published on

International Food Policy Research Institute (IFPRI) organized a three days Training Workshop on ‘Monitoring and Evaluation Methods’ on 10-12 March 2014 in New Delhi, India. The workshop is part of an IFAD grant to IFPRI to partner in the Monitoring and Evaluation component of the ongoing projects in the region. The three day workshop is intended to be a collaborative affair between project directors, M & E leaders and M & E experts. As part of the workshop, detailed interaction will take place on the evaluation routines involving sampling, questionnaire development, data collection and management techniques and production of an evaluation report. The workshop is designed to better understand the M & E needs of various projects that are at different stages of implementation. Both the generic issues involved in M & E programs as well as project specific needs will be addressed in the workshop. The objective of the workshop is to come up with a work plan for M & E domains in the IFAD projects and determine the possibilities of collaboration between IFPRI and project leaders.

Published in: Data & Analytics
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
844
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

IFPRI -AOS Survey in Bhutan

  1. 1. AOS Survey in Bhutan Devesh Roy on behalf of Bhutan team
  2. 2. What is the annual outcome survey and why to do it? • IFAD has developed a standard methodology for impact measurement, these impact surveys are not providing the type of results’ information that can allow Project Management Teams to take timely, corrective action during the course of project implementation. • Such impact surveys are indeed primarily intended to document project impact at completion. • To shift focus from impact documentation at completion to outcome measurement during project implementation, IFAD now encourages all projects Asia and the Pacific to survey annually a small sample of beneficiaries to: • measure more regularly the positive or negative changes/outcomes taking place at the household level • provide early evidence of project success or failure • provide timely performance information so that corrective actions may be taken if required • assess targeting efficiency • AOS is a simple Hh survey undertaken annually and cover a small sample of 200 households selected randomly. • The survey conducted exclusively in villages targeted by the project or receiving project interventions, and will include both project beneficiaries and non-beneficiaries (the latter being used as a comparison group). • Exercise expected to take no more than 3 months and can be implemented by project staff and extension officers, with or without external support.
  3. 3. Points in AOS
  4. 4. AOS survey design • AOS should be conducted every year starting from the 2nd year of implementation. • The survey to be implemented in conjunction with qualitative assessments that will complement the household-level information, providing information on “why” and “how” some outcomes were or were not achieved. • FGD • KII • These two to be conducted after the surveys and data analysed • Suppose large groups of intended beneficiaries dissatisfied with the project, FGD and KII explore why?
  5. 5. Steps in AOS (IFAD prescription) Activity/step Estimated Duration Comments Responsibility Step 1 Fine-tuning of the standard survey questionnaire and preparation of interview guides for qualitative assessments (see TG 3 and 4) 1-3 days Can take longer if questions are added or modified from the standard template. M&E Officer, with inputs from component coordinators and project coordinator. Step 2 Sample selection 1-3 days Depending on availability of reliable lists. M&E Officer. Lists to be provided by project staff (villages) and local authorities (households). Step 3 Training of enumerators and field testing of questionnaire and qualitative assessment guides 1-2 days The training should include a session on sampling. M&E officer and external consultant if any. Step 4 Logistical planning, preparation for data collection 1-2 days M&E Officer. Step 5 Data collection 1-3 weeks Depends on availability of cars/motorbikes. Enumerators and their supervisors Step 6 Data entry 1-2 weeks If Excel file is used, data entry can only be conducted by one person at a time. M&E Officer for supervision, Assistant for data entry. Step 7 Data analysis 1 week M&E Officer with support from external consultant (if any). Step 8 Report writing 1 week M&E officer, with inputs from Component Coordinators and Project Coordinator/Director. Step 9 Communication and sharing ad hoc events Project coordinator, M&E and KM Officer.
  6. 6. Steps in AOS Activity/step Estimated Duration Comments Responsibility Step 1 Fine-tuning of the standard survey questionnaire and preparation of interview guides for qualitative assessments (see TG 3 and 4) 1-3 days Can take longer if questions are added or modified from the standard template. M&E Officer, with inputs from component coordinators and project coordinator. Step 2 Sample selection 1-3 days Depending on availability of reliable lists. M&E Officer. Lists to be provided by project staff (villages) and local authorities (households). Step 3 Training of enumerators and field testing of questionnaire and qualitative assessment guides 1-2 days The training should include a session on sampling. M&E officer and external consultant if any. Step 4 Logistical planning, preparation for data collection 1-2 days M&E Officer.
  7. 7. Steps in AOS: Continued Activity/step Estimated Duration Comments Responsibility Step 5 Data collection 1-3 weeks Depends on availability of cars/motorbikes. Enumerators and their supervisors Step 6 Data entry 1-2 weeks If Excel file is used, data entry can only be conducted by one person at a time. M&E Officer for supervision, Assistant for data entry. Step 7 Data analysis 1 week M&E Officer with support from external consultant (if any). Step 8 Report writing 1 week M&E officer, with inputs from Component Coordinators and Project Coordinator/Director. Step 9 Communication and sharing ad hoc events Project coordinator, M&E and KM Officer.
  8. 8. Questionnaire development A – HOUSEHOLD IDENTIFICATION A.1 Village ____________________________ A.1.a District or Province ___________________________ A.2 Name of respondent : ______________________________________________________________ A.2.a What is the gender (sex) of the respondent? A. Male B. Female A.3 Name of the head of household : ______________________________________________________________ A.4 What is the gender (sex) of the household head? A. Male ( Go to A.5) B. Female A.4.1. Is there a relative who takes care (financially) of your household’s needs? A. Yes, a relative is taking care of us B. No, no-one is helping us A.5 What is the size of your household/family (e.g. nb of persons living under the same roof)? /_____/_____/ persons A.6 How many children (under 16 years), adults and elderly persons are there in your household/family? /_____/ children /_____/ adults /_____/ elderly A.7 In terms of income group, in which of the following categories would you consider that your households belongs? A. Well off B. Average C. Poor D. Very poor
  9. 9. QUESTIONNAIRE A - FOR BENEFICIARIES B – PARTICIPATION IN PROJECT ACTIVITIES Since when is your household involved in project activities (year)? /_______________/ [year] Over the past 12 months, have you (or a member of your household) participated in the following project activities? [Adapt the response options below to the context of your project] a - [Insert name of activity a] A. Yes B. No b - [Insert name of activity b] A. Yes B. No c - [Insert name of activity c] A. Yes B. No Of the activities in which you have participated, which ones do you find were the most useful ? [ Up to three responses only] /_____/ ; /_____/ ; /_____/ [Use same activity Code as above] Of the activities in which you have participated, which ones do you find were the least useful ? [ Up to three responses only] /_____/ ; /_____/ ; /_____/ [Use same activity Code as above] How often do you have contact with project staff (extension workers; facilitators, etc.)? A. Frequently (e.g. more than 2 times/year) B. Occasionally (e.g. at least one time/year) C. Rarely (e.g. less than one time/year) A. Highly satisfied
  10. 10. Stylized AOS questionnaire • AOS stylized questionnaire.docx (see the elements of the questionnaire) • A standard questionnaire. Questionnaire to be adapted to the specificity of project interventions. • All sections of the questionnaire to be used as they are, or they can be modified according to the specific information needs of the project.
  11. 11. Questionnaire: Sampling • The sampling frame (i.e. population from where the sample is selected) is the list of villages targeted by the project. • Sample will be composed of 200 households (20 villages, 10 households per village). • The selection of the sample done in two stages: (1) Selection of 20 villages; and (2) Selection of 10 households in each village. • The same sampling frame will be used every year, but the sample selection will be re-done every year. Hence different villages and different households surveyed each year.
  12. 12. AOS: Sampling Guidelines (Continued) • Selection of villages • If villages similar- random sampling • If villages dissimilar- stratified random sampling • After selection of villages random selection of households • Sampling interval = N/n (to the integer value) • Random sample of villagers if list available (using sample interval) • If the villagers list not available (transect line) • Project beneficiaries and non-beneficiaries (how can they be ensured based on the sampling proposed?)
  13. 13. AOS survey in Bhutan • AOS survey seems like a small activity but is quite involved • I wrote a long letter because I did not have time to write a short one • The frequency can be demanding • Well laid down rules- helpful always?
  14. 14. Market Access and Growth Intensification Project (MAGIP)
  15. 15. Some background on Bhutan and MAGIP • Six eastern Dzongkhags of Lhuentse, Tashiyangtse, Trashigang, Mongar, Pemagatshel and Samdrup Jongkhar • Implementing agencies – Ministry of Agriculture and Forestry in collaboration with district authorities and RAMCO (Combined ministry quite indicative) Separate Ministry of Finance and Economic Affairs also indicative • Goal –Reduce poverty and improve food security • Objective –enhance productivity of subsistence based farming where no road access, with road access promote cash crops and HVA, focus also on Alpine herders (incidentally not the poorest like pastoralists elsewhere since big herds) • One Geog 3 product plan (OGTP)- what does it imply for MAGIP? • Hierarchical geography like Nepal and young population like India • Almost 90% rural population raises livestock in Bhutan
  16. 16. Characteristics of the targeted areas • Six agro-ecological zones • Alpine (3600-4600 meters) – mainly summer pastures for yak herding • Cool temperate (2600-3600 meters) –transhumant yak and sheep farming • Warm temperate (1800-2600 meters)-migratory cattle herders with the cultivation of rice, wheat, potato and other vegetables • Dry subtropical (1200-1800 meters)- comparatively warm with moderate rainfall- major crops rice and maize followed by wheat and mustard • Humid subtropical (600-1200 meters)- high rainfall and temperature, cattle reared in sedentary system – rice, maize, mustard and citrus fruits • Wet subtropical (150-600 meters)-very high rainfall, rice grown as the main crop in summer, maize and wheat in winter depending on irrigation • Barring Alpine and Cool temperate others suitable for sustainable agricultural and livestock production. • Yet lack of infrastructure and market access have stymied progress in eastern region
  17. 17. Farmer differentiation by type in MAGIP
  18. 18. Background for AOS survey • The improvements in reducing human deprivation in Bhutan have come largely from reducing malnutrition and improving access to safe drinking water. • Both the proportions of population without access to safe drinking water or in malnourished category are below 10 percent (a remarkable achievement) • Gender equality is built into Bhutanese society (matrilineal system of inheritance) and governance (yet disparities remain in education and employment-possibly a little late start) • The RGOB does mainstream gender in its development plans
  19. 19. Facets about MAGIP • MAGIP is in essence a targeted intervention (set of targeted interventions) • MAGIP focuses on a small set of commodities and sub-sectors to maximize impact and support the emergence of rural market economy (in line with OGTP initiative) • Impact is enshrined in the project • MAGIP has new system of extension (Farmer Field Schools) and customized to clients like subsistence farmers, farmers with market access and Alpine hereers (losing ground with issues like fodder scarcity, changing consumers taste) • MAGIP adopts a value chain approach
  20. 20. Elements in MAGIP • Agricultural development (Extension (FFS), fruit trees related extension for better nutrition, mitigate post harvest losses (e.g. drying storage facilities), backyard poultry development, community foresty management, Pilot Agro Tourism Initiative • Agricultural Intensification and support to market access (targeting farmers in 37 gewogs who have marketable surplus) (OGTP – vegetables, potato, dairy and citrus –giving support to producer groups in these)
  21. 21. Outputs of MAGIP • In 11 remote gewogs where communities have no or difficult road access 880 households supported with low cost labor saving technologies in subsistence farming • In 37 gewogs with good road access and agro-ecological potential project to support 3600 producers organized in groups with training inputs and services for cash crops and dairy production • Marketing and storage infrastructure, farm roads and irrigation schemes constructed or rehabilitated
  22. 22. Broad characteristics of households in MAGIP area
  23. 23. AOS survey –closest collaboration between Bhutan project team/IFAD and IFPRI • Work collaboratively to develop questionnaire • Try to make it more quantitative • In fact try to get a mix of quantitative and qualitative • Work collaboratively for sampling • Could have longitudinal data • Data collection methods • Digitalization of survey • Possible link with questionnaire development
  24. 24. Results based Monitoring and Evaluation under MAGIP • All RGOB funded programs routine M&E activities are responsibility of district staff • Gewog level RNR extension staff should regularly collect data at activity, output, outcome and impact levels • Results data to be entered into the web based PLAMS • RIMS first and second level indicators submitted annually to IFAD • Given the difficult geographical terrain standard RIMS impact surveys not to be conducted, alternative methodology to be devised
  25. 25. AOS questionnaire in Bhutan • FinalforShanti_docx • Get household identification- some more details –why • Try tracking households- why? • Mix of quantitative and qualitative questions why?

×