ICAR-IFPRI: Methodologies in Agricultural Extension Research capacity development program from September 21-24, 2015 in New Delhi

1. Questionnaire development
Devesh Roy (September 22, 2015)
IFPRI-ICAR training

2. Some definitions
• Semi-structured Interview:
• A semi-structured interview is an interview with an individual or
individuals that follows a pre-defined set of questions. It is flexible,
allowing new questions to be brought up during the interview as a
result of what the interviewee says.

3. questionnaire
• The survey instrument for every evaluation should be carefully designed to gather all the
information needed to test the hypotheses underscoring the project logic as well as to construct
other control variables for impact estimation.
• A questionnaire to collect agricultural data should usually be based on the agricultural household
model. This model suggests that agricultural production units behave as both a firm and a
household and, in the presence of market failure, production decisions cannot be separated from
household consumption decisions. Hence, collecting household as well as farm information
should be a priority in every questionnaire that aims to analyze agricultural household behavior.
• In general, the modules to be included should obtain a full picture of the household livelihood
strategies, including crop and livestock activities but also non-agricultural activities, migration and
remittances among others.
• Many of these variables may not be measured directly so a series of questions may need to be
asked to obtain a single key variable. For example, questions on crop profits or gross margins
require having information on revenues and costs of crop production. Questions that directly ask
this information are unlikely to be reliable so it requires asking a series of questions related to
revenue and costs that help to obtain the information.

4. Questionnaire: essential elements (World
Bank 2008)
• What do you want to know?
• Tailor survey to capture outcomes of interest
• Use reliable and valid instruments
• Gender disaggregation!!
• Be careful: what‟s reliable and valid in one context may not be so in
another
• test your questionnaire!
• In house
• In the field

5. Mistakes in questionnaire development (medal
level from the point of correction potential)
• Gold medal mistakes
• Start work on the questionnaire before finalizing the evaluation design
• Start work on the questionnaire before setting up the methods
• Start work on the questionnaire before fixing the sampling design
• Finalizing questionnaire before pre-testing
• No validation checks built in the instrument
• Silver medal mistakes
• Not incorporating the environment of the household
• High density of open ended questions
• Data prone to noise (like income)sought in questionnaire
• Objective of the survey (the starting point not fixed in a team but by an individual)
• Household roster not specified – who is to be included in household
• Bronze medal mistakes
• Enumeration load, sensitive questions, loaded words

7. Questionnaire guidelines (world bank 2008)
• Define topics and concepts to avoid confusion
• Define your identity properly – are you government, are you NGO etc. affects the
responses
• Question order matters
• Keep it short and make it user-friendly
• Every time you write a question, ask yourself “Why do I want to know this?”
Answer it in terms of the way it will help you to answer your research question.
“It would be interesting to know” is not an acceptable answer.
• Phrase questions clearly
• Use established techniques to minimize respondent mistakes (e.g. calendars for
event histories). Asking monthly or weekly rather than asking regularly
• If continuous markers cannot be obtained get categorical information like income
in ranges.
• Operate in modules – treat modules as you treat paragraphs in writing

8. Questionnaire guidelines
• Time period and saliency –purchase of house more salient should be
timed in the questionnaire in terms of say one year period while time
period for purchase of clothing could be say one month
• If asking questions about others focus more on strong ties (son
daughter) and less on weak ties (such as co-worker) unless it is by
design such as in social network analysis.
• Role of multiple options possible
• Role of other as an option in responses – use but use sparingly
studies show this is underreported

9. Example of a good question (trace test- get 3-4 people and
check the interpretation –this is part of in
house test)
• Aside from any work you do here at home or at a job, do you do
anything regularly—that is, on a daily basis—that helps you keep
physically fit?
• Yes
• No

10. Best practice
• Search for questions on the same topic by other researchers
• Precautions:
• Mark the change in the wording even if it is small
• Take account for the differences in timing of the earlier and current surveys

11. Ask exactly what you want to know (Bradburn
et al)
• The precise wording of questions plays a vital role in determining the
answers given by respondents. This fact is not appreciated as fully as
it should be, even in ordinary conversation. For example, someone
mentioned that he needed to buy something. Only day he could come
was Saturday before labor day weekend. Although he called on Friday
to make certain the store was open, he arrived at the store on
Saturday only to find a sign on the door that said “Closed Labor Day
Weekend.” When asked if he remembered what question he had
asked the clerk at the store, he said, “I asked him what hours he was
open on Saturday, and he replied ‘Nine to five.”

12. Some points to note (can get in the revision
after pretesting)
• Use of aided recall –Instead of asking what did you eat in the last
week you would produce a list of items to choose from
• Long lists anchor on last or first ones
• Use of proxy means (dwellings, assets, capital etc)

13. Bias and variability
• Bias refers to an estimate that is either more or less than the true value.
Variability is measured by the susceptibility of measurements to
differences in question wording. This variability is sometimes called the
reliability of a measure, since random errors may arise from the form of
the measurement itself (rather than from systematic error due to a sample
bias or some other aspect of the measurement instrument).
• In order to clarify the sources of response effects, let us look at a particular
behavioral question. A common question in surveys is “What was your
total family income from all sources last year?” There is a true answer to
this question, even though we may never know what it is. However, even
though there is a true answer to this question, we may get an erroneous
answer because the respondent simply forgot about certain amounts of
income, particularly those from less obvious sources.

14. Bias and variability: continued
• Pure recall errors could be classical in the sense that on average they
might cancel out some overestimate and some underestimate
• But there could be misreporting because of motivation to hide or
blow up income (to look better)

15. Bradburn et al (Loaded words in questions
produce loaded results) (Bradburn et al)
• Karl Marx in a survey of workers asked Does your employer or his
representative resort to trickery in order to defraud you of part of
your earnings
• Could have been asked as are you fairly compensated?

16. Best practices
• A good question is understood consistently by all respondents
• A good question is administered consistently to all respondents
• A good question elicits the kind of answers the researcher wants:
• • BadQ: When did you move to Seoul, Korea?
• • A: In 1964
• • A: When I was 20 years old
• • A: After I finished college
• • BetterQ: In what YEAR did you move to Seoul, Korea?

17. Best practices
• A good question is one where the respondents have the necessary
knowledge to answer
• A good question is one where the respondent is willing to provide the
„true‟ answer
• difficult for sensitive questions

18. Compatibility of questionnaire with methods
• If cross-sectional variation relied on what questions should be there
• If double difference methods employed what should be included
• What in case of most significant change

19. Other elements – accounting for environment
• Account for environment – what other projects
• Make a distinction between time varying and time invariant
• Compatibility with methods- if there is time invariant participation in some
project given the method may not need to include
• Intensity should always be asked
• For the project and others timing of engagement should always be asked
• A community questionnaire should be employed for accounting for
environment (how to define a community?)

20. Some leads from LSMS surveys
• Have metadata such as dates of interviews, household roster
information on any children identities of respondents, and time
required for each interview.
• Household roster well defined
• Consumption measures should be used measure- The measurement
of consumption is highly sensitive to differences in methods, and that
consumption measurement must be standardized over time.

21. Lessons: continued
• First, detailed information on household members' work for
household to collect in agriculture and enterprise modules not in
employment modules
• Generically pre test with short and long modules in specific cases
• Household enterprises are very important in developing countries-
have a separate module on them