International Food Policy Research Institute (IFPRI) organized a three days Training Workshop on ‘Monitoring and Evaluation Methods’ on 10-12 March 2014 in New Delhi, India. The workshop is part of an IFAD grant to IFPRI to partner in the Monitoring and Evaluation component of the ongoing projects in the region. The three day workshop is intended to be a collaborative affair between project directors, M & E leaders and M & E experts. As part of the workshop, detailed interaction will take place on the evaluation routines involving sampling, questionnaire development, data collection and management techniques and production of an evaluation report. The workshop is designed to better understand the M & E needs of various projects that are at different stages of implementation. Both the generic issues involved in M & E programs as well as project specific needs will be addressed in the workshop. The objective of the workshop is to come up with a work plan for M & E domains in the IFAD projects and determine the possibilities of collaboration between IFPRI and project leaders.

1. AOS Survey in Bhutan
Devesh Roy on behalf of Bhutan team

2. What is the annual outcome survey and why to do it?
• IFAD has developed a standard methodology for impact measurement, these impact surveys are not
providing the type of results’ information that can allow Project Management Teams to take timely,
corrective action during the course of project implementation.
• Such impact surveys are indeed primarily intended to document project impact at completion.
• To shift focus from impact documentation at completion to outcome measurement during project
implementation, IFAD now encourages all projects Asia and the Pacific to survey annually a small
sample of beneficiaries to:
• measure more regularly the positive or negative changes/outcomes taking place at the household
level
• provide early evidence of project success or failure
• provide timely performance information so that corrective actions may be taken if required
• assess targeting efficiency
• AOS is a simple Hh survey undertaken annually and cover a small sample of 200 households selected
randomly.
• The survey conducted exclusively in villages targeted by the project or receiving project interventions,
and will include both project beneficiaries and non-beneficiaries (the latter being used as a
comparison group).
• Exercise expected to take no more than 3 months and can be implemented by project staff and
extension officers, with or without external support.

3. Points in AOS

4. AOS survey design
• AOS should be conducted every year starting from the 2nd year of
implementation.
• The survey to be implemented in conjunction with qualitative
assessments that will complement the household-level information,
providing information on “why” and “how” some outcomes were or
were not achieved.
• FGD
• KII
• These two to be conducted after the surveys and data analysed
• Suppose large groups of intended beneficiaries dissatisfied with the project, FGD and KII
explore why?

5. Steps in AOS (IFAD prescription)
Activity/step Estimated Duration Comments Responsibility
Step 1 Fine-tuning of the standard survey
questionnaire and preparation of interview
guides for qualitative assessments (see TG 3
and 4)
1-3 days Can take longer if questions are added or
modified from the standard template.
M&E Officer, with inputs from component coordinators
and project coordinator.
Step 2 Sample selection 1-3 days Depending on availability of reliable lists. M&E Officer. Lists to be provided by project staff (villages)
and local authorities (households).
Step 3 Training of enumerators and field testing of
questionnaire and qualitative assessment
guides
1-2 days The training should include a session on
sampling.
M&E officer and external consultant if any.
Step 4 Logistical planning, preparation for data
collection
1-2 days M&E Officer.
Step 5 Data collection 1-3 weeks Depends on availability of
cars/motorbikes.
Enumerators and their supervisors
Step 6 Data entry 1-2 weeks If Excel file is used, data entry can only be
conducted by one person at a time.
M&E Officer for supervision, Assistant for data entry.
Step 7 Data analysis 1 week M&E Officer with support from external consultant (if
any).
Step 8 Report writing 1 week M&E officer, with inputs from Component Coordinators
and Project Coordinator/Director.
Step 9 Communication and sharing ad hoc events Project coordinator, M&E and KM Officer.

6. Steps in AOS
Activity/step Estimated Duration Comments Responsibility
Step 1 Fine-tuning of the standard
survey questionnaire and
preparation of interview
guides for qualitative
assessments (see TG 3 and
4)
1-3 days Can take longer if
questions are added or
modified from the
standard template.
M&E Officer, with inputs from
component coordinators and project
coordinator.
Step 2 Sample selection 1-3 days Depending on availability
of reliable lists.
M&E Officer. Lists to be provided by
project staff (villages) and local
authorities (households).
Step 3 Training of enumerators and
field testing of questionnaire
and qualitative assessment
guides
1-2 days The training should include
a session on sampling.
M&E officer and external consultant
if any.
Step 4 Logistical planning,
preparation for data
collection
1-2 days M&E Officer.

7. Steps in AOS: Continued
Activity/step
Estimated
Duration
Comments Responsibility
Step 5 Data collection 1-3 weeks Depends on availability of
cars/motorbikes.
Enumerators and their supervisors
Step 6 Data entry 1-2 weeks If Excel file is used, data
entry can only be
conducted by one person at
a time.
M&E Officer for supervision, Assistant
for data entry.
Step 7 Data analysis 1 week M&E Officer with support from
external consultant (if any).
Step 8 Report writing 1 week M&E officer, with inputs from
Component Coordinators and Project
Coordinator/Director.
Step 9 Communication and sharing ad hoc events Project coordinator, M&E and KM
Officer.

8. Questionnaire development A – HOUSEHOLD IDENTIFICATION
A.1 Village ____________________________ A.1.a District or Province ___________________________
A.2 Name of respondent : ______________________________________________________________
A.2.a What is the gender (sex) of the respondent?
A. Male
B. Female
A.3 Name of the head of household : ______________________________________________________________
A.4
What is the gender (sex) of the household head?
A. Male ( Go to A.5)
B. Female
A.4.1. Is there a relative who takes care (financially) of your household’s
needs?
A. Yes, a relative is taking care of us
B. No, no-one is helping us
A.5
What is the size of your household/family (e.g. nb of persons living under the
same roof)?
/_____/_____/ persons
A.6
How many children (under 16 years), adults and elderly persons are there in your
household/family?
/_____/ children
/_____/ adults
/_____/ elderly
A.7
In terms of income group, in which of the following categories would you
consider that your households belongs?
A. Well off
B. Average
C. Poor
D. Very poor

9. QUESTIONNAIRE A - FOR BENEFICIARIES
B – PARTICIPATION IN PROJECT ACTIVITIES
Since when is your household involved in project activities (year)? /_______________/ [year]
Over the past 12 months, have you (or a member of your household) participated in the following project activities?
[Adapt the response options below to the context of your project]
a - [Insert name of activity a] A. Yes B. No
b - [Insert name of activity b] A. Yes B. No
c - [Insert name of activity c] A. Yes B. No
Of the activities in which you have participated, which ones do you find were the most
useful ? [ Up to three responses only]
/_____/ ; /_____/ ; /_____/
[Use same activity Code as above]
Of the activities in which you have participated, which ones do you find were the least
useful ? [ Up to three responses only]
/_____/ ; /_____/ ; /_____/
[Use same activity Code as above]
How often do you have contact with project staff (extension workers; facilitators, etc.)?
A. Frequently (e.g. more than 2 times/year)
B. Occasionally (e.g. at least one time/year)
C. Rarely (e.g. less than one time/year)
A. Highly satisfied

10. Stylized AOS questionnaire
• AOS stylized questionnaire.docx (see the elements of the
questionnaire)
• A standard questionnaire. Questionnaire to be adapted to the
specificity of project interventions.
• All sections of the questionnaire to be used as they are, or they can
be modified according to the specific information needs of the
project.

11. Questionnaire: Sampling
• The sampling frame (i.e. population from where the sample is
selected) is the list of villages targeted by the project.
• Sample will be composed of 200 households (20 villages, 10
households per village).
• The selection of the sample done in two stages: (1) Selection of 20
villages; and (2) Selection of 10 households in each village.
• The same sampling frame will be used every year, but the sample
selection will be re-done every year. Hence different villages and
different households surveyed each year.

12. AOS: Sampling Guidelines (Continued)
• Selection of villages
• If villages similar- random sampling
• If villages dissimilar- stratified random sampling
• After selection of villages random selection of households
• Sampling interval = N/n (to the integer value)
• Random sample of villagers if list available (using sample interval)
• If the villagers list not available (transect line)
• Project beneficiaries and non-beneficiaries (how can they be ensured based
on the sampling proposed?)

13. AOS survey in Bhutan
• AOS survey seems like a small activity but is quite involved
• I wrote a long letter because I did not have time to write a short one
• The frequency can be demanding
• Well laid down rules- helpful always?

14. Market Access and Growth Intensification
Project (MAGIP)

15. Some background on Bhutan and MAGIP
• Six eastern Dzongkhags of Lhuentse, Tashiyangtse, Trashigang, Mongar,
Pemagatshel and Samdrup Jongkhar
• Implementing agencies – Ministry of Agriculture and Forestry in collaboration
with district authorities and RAMCO (Combined ministry quite indicative)
Separate Ministry of Finance and Economic Affairs also indicative
• Goal –Reduce poverty and improve food security
• Objective –enhance productivity of subsistence based farming where no road
access, with road access promote cash crops and HVA, focus also on Alpine
herders (incidentally not the poorest like pastoralists elsewhere since big herds)
• One Geog 3 product plan (OGTP)- what does it imply for MAGIP?
• Hierarchical geography like Nepal and young population like India
• Almost 90% rural population raises livestock in Bhutan

17. Characteristics of the targeted areas
• Six agro-ecological zones
• Alpine (3600-4600 meters) – mainly summer pastures for yak herding
• Cool temperate (2600-3600 meters) –transhumant yak and sheep farming
• Warm temperate (1800-2600 meters)-migratory cattle herders with the
cultivation of rice, wheat, potato and other vegetables
• Dry subtropical (1200-1800 meters)- comparatively warm with moderate
rainfall- major crops rice and maize followed by wheat and mustard
• Humid subtropical (600-1200 meters)- high rainfall and temperature, cattle
reared in sedentary system – rice, maize, mustard and citrus fruits
• Wet subtropical (150-600 meters)-very high rainfall, rice grown as the main
crop in summer, maize and wheat in winter depending on irrigation
• Barring Alpine and Cool temperate others suitable for sustainable agricultural
and livestock production.
• Yet lack of infrastructure and market access have stymied progress in eastern region

18. Farmer differentiation by type in MAGIP

19. Background for AOS survey
• The improvements in reducing human deprivation in Bhutan have
come largely from reducing malnutrition and improving access to safe
drinking water.
• Both the proportions of population without access to safe drinking water or in
malnourished category are below 10 percent (a remarkable achievement)
• Gender equality is built into Bhutanese society (matrilineal system of
inheritance) and governance (yet disparities remain in education and
employment-possibly a little late start)
• The RGOB does mainstream gender in its development plans

20. Facets about MAGIP
• MAGIP is in essence a targeted intervention (set of targeted interventions)
• MAGIP focuses on a small set of commodities and sub-sectors to maximize
impact and support the emergence of rural market economy (in line with
OGTP initiative)
• Impact is enshrined in the project
• MAGIP has new system of extension (Farmer Field Schools) and customized
to clients like subsistence farmers, farmers with market access and Alpine
hereers (losing ground with issues like fodder scarcity, changing consumers
taste)
• MAGIP adopts a value chain approach

21. Elements in MAGIP
• Agricultural development (Extension (FFS), fruit trees related
extension for better nutrition, mitigate post harvest losses (e.g. drying
storage facilities), backyard poultry development, community foresty
management, Pilot Agro Tourism Initiative
• Agricultural Intensification and support to market access (targeting
farmers in 37 gewogs who have marketable surplus) (OGTP –
vegetables, potato, dairy and citrus –giving support to producer
groups in these)

22. Outputs of MAGIP
• In 11 remote gewogs where communities have no or difficult road
access 880 households supported with low cost labor saving
technologies in subsistence farming
• In 37 gewogs with good road access and agro-ecological potential
project to support 3600 producers organized in groups with training
inputs and services for cash crops and dairy production
• Marketing and storage infrastructure, farm roads and irrigation
schemes constructed or rehabilitated

23. Broad characteristics of households in MAGIP area

24. AOS survey –closest collaboration between
Bhutan project team/IFAD and IFPRI
• Work collaboratively to develop questionnaire
• Try to make it more quantitative
• In fact try to get a mix of quantitative and qualitative
• Work collaboratively for sampling
• Could have longitudinal data
• Data collection methods
• Digitalization of survey
• Possible link with questionnaire development

25. Results based Monitoring and Evaluation under MAGIP
• All RGOB funded programs routine M&E activities are responsibility of district
staff
• Gewog level RNR extension staff should regularly collect data at activity,
output, outcome and impact levels
• Results data to be entered into the web based PLAMS
• RIMS first and second level indicators submitted annually to IFAD
• Given the difficult geographical terrain standard RIMS impact surveys not to be
conducted, alternative methodology to be devised

26. AOS questionnaire in Bhutan
• FinalforShanti_docx
• Get household identification- some more details –why
• Try tracking households- why?
• Mix of quantitative and qualitative questions why?