This document provides an overview of questionnaire development. It discusses what questionnaires measure, including factual information, behaviors, and attitudes. The main parts of questionnaires are also outlined, including the title, instructions, questions, and a thank you. Questionnaires can use open or closed-ended questions and Likert scales. Proper administration involves an appropriate sample size and cover letter. Key considerations in developing questionnaires include writing clear, unbiased questions and arranging questions in a logical order.

1. Questionnaire Development
Dr Christine Coombe
Dubai Men’s College

2. Presentation Agenda
• What are questionnaires and what do
they measure?
• Objectives in questionnaire research
• Advantages/Disadvantages
• Main parts and formats
• Do’s/Don’ts of questionnaire writing
• Questionnaire administration
• Issues in questionnaire research
• Ethical principles of questionnaire
research

3. What are Questionnaires?
• Questionnaires are any written
instruments that present respondents
with a series of questions or statements
to which they are to react either by
writing out their answers or selecting
them from among existing answers.
(Brown, 2001:6)
• Referred to by different names
– Inventories, forms, opinnionaires, tests,
batteries, checklists, scales, surveys,
profiles, indexes (Aiken, 1997)

4. What do Questionnaires
Measure?
• Questionnaires (Qs) can yield different
types of information (Dornyei, 2003)
– Factual questions
• Find out information about who respondents are;
demographics
– Behavioral questions
• Find out what respondents are doing or have
done in the past
– Attitudinal questions
• Find out what people think
• Attitudes, opinions, beliefs, interests & values

5. Objectives in Questionnaire
Research
• To obtain accurate and relevant
information/data
• To maximise the response rate (the
proportion of subjects answering
the Q)

6. Deciding What to Ask
• Information we are primarily
interested in (dependent variables)
• Information which might explain the
dependent variables (independent
variables)
• Other factors related to DV and IV
factors which may distort the results
and have to be adjusted for
(confounding variables)

7. Advantages
• Unprecedented efficiency in terms of
researcher time, effort and financial resources
(Dornyei, 2003)
• Versatility– can be used successfully with a
variety of people in a variety of contexts
targeting a variety of topics
• Collect a large amount of data in a short time
(Brown, 1988)
• Easier and less expensive than other forms of
data collection (Seliger & Shohamy, 1989)
• Can be used to research any aspect of teaching
and learning (Nunan, 1989)
• Can be easily used in field settings such as
classrooms (Nunan, 1992)

8. Disadvantages
• Simplicity and superficiality of answers
– Qs must be understood by all; written in
simplistic way; can’t probe deeply into an
issue; results in superficial data (Moser &
Kalton, 1971)
• Unreliable and unmotivated respondents
– Results vary greatly from person to person
• Respondent literacy problems
– Assumption that all can read and write well

9. Disadvantages
• Social desirability or prestige bias
– People do not always provide true answers
about themselves
– Over reporting “good” behavior or under
reporting “bad” or undesirable behavior
– Happens because Qs are often transparent
(i.e. Rs have a fairly good idea of what the
desirable/acceptable answer is)
– Presenting ourselves in a good light is a
natural human tendency

10. Disadvantages
• Acquiescence bias
– Tendency for people to agree with sentences when
they are unsure or ambivalent
– Go along with anything that sounds good
• Halo effect
– One's judgments of a person’s character can be
influenced by one's overall impression of him or her
– Concerns the human tendency to overgeneralize
– If our overall opinion about someone/thing is positive
we are disinclined to say anything negative
• Fatigue effects
– Kicks in if Q is overly long

11. Main Parts of Questionnaires
• Title
– To provide Rs with initial orientation & activate
content schemata (Dornyei, 2003)
– Qs with titles are generally perceived as more
serious/credible than those without
• Instructions: Two Types
– General: opening greeting at beginning of Q
– Should contain
• What the study is about and why it’s important
• Organization responsible for conducting study
• Emphasis that there are no ‘right’ or ‘wrong’ answers
• Promise of confidentiality
• A ‘thank you’

12. Main Parts of Questionnaires
• Instructions
– Specific
– Explains and demonstrates how Rs should
answer questions
– Each new task type requires instructions
• Questionnaire Items
• Additional Information
– Contact details of researcher; how Qs should
be returned; note promising a copy of
results; invitation for follow-up interview
• Final ‘thank you’

13. Open or Closed Formats?
• Responses can be in open or
closed formats
– Open-ended formats
• Rs formulate their own answers
– Closed formats
• Rs are forced to choose between several
given options
• It is possible to use a mixture of
these formats

14. Open or Closed Formats:
Advantages
• Open-ended formats
• Allows exploration of
the range of possible
themes arising from
an issue
• Can be used if a
comprehensive
range of alternative
choices cannot be
compiled
• Closed (or forced)
choice format
• Easy and quick to fill
in
• Minimise
discrimination
against the less
literate or the less
articulate
• Easy to code, record,
analyze results
quantitatively
• Easy to report results

15. Closed or Forced Choice
Formats
• Choice of Categories
– What is your marital status?
– Circle/tick: single, married, divorced,
separated
• Likert Scale
– LSS 2533 Research Methods is an
interesting course.
– Circle/tick SD D NA or D A SA
• Differential Scale
– How would you rate this presentation?
– Very Boring 1 2 3 4 5 6 7 8 9 10 Very
interesting

16. Closed or Forced Choice
Formats
• Checklists
– Circle the leisure time activities you
are particularly interested in
• Video games
• Watching TV
• Playing football
• Reading
• Going to movies
• Going to the beach
• Other: __________

17. Closed or Forced Choice
Formats
• Ranking
– Please rank your abilities in English.
(1=best; 5=worst)
• Listening
• Reading
• Writing
• Speaking
• Grammar
• Vocabulary
– Probably the least used format because of
difficulty with analysis

18. Open-ended Questions
• Specific open questions
– Ask about concrete pieces of information
(i.e. facts, past activities, preferences)
– How many languages have you studied?
• Clarification questions
– Questions that are so important you need
follow-up information
– If you rated the textbook you are using as
‘poor’ or ‘very poor’, please briefly explain
why

19. Open-ended Questions
• Sentence completion questions
– One thing I liked about this class
was……
– I found this activity……
• Short-answer questions
– Exploratory inquiry about an issue
– What was it that you found most
useful about this class?

20. Length?
• General temptation is always to cover
too much ground by asking everything
that might turn out to be interesting
• Experts agree that anything over 4-6
pages and requires more than 30 min to
complete is too long
• As a general rule, long questionnaires
get fewer responses than short Qs

21. Question Arrangement
• Go from general to particular
• Go from easy to difficult
• Go from factual to abstract
• Start with closed format questions
• Start with questions relevant to the
main subject
• Do not start with demographic and
personal questions

22. How to Write Good Items
• Most problems can be traced back to the
design phase
– Ask only Qs that address the study
goals
• Use short and simple sentences.
– less confusing & ambiguous
– most sentences should contain
one/two clauses & not exceed 20
words.
• Ask for only one piece of information at a
time.
– “Please rate the lecture in terms of its
content and presentation” asks two
questions.

23. How to Write Good Items
• Ask respondents to rate both positive and
negative statements
– tendency to mark every item at the same
end of the scale
– by offering positive/negative responses Rs
must evaluate every statement
• Begin with non-threatening and interesting
items
• Leave space for comments
• Place most important Qs in the first half of Q
– Ss often send back partially completed Qs
• Hold the Ss interest
– provide a variety of item types; this will prevent
‘response sets’

24. What to Avoid
• Avoid scale point proliferation
– Never/rarely/occasionally/fairly
often/often/very often/almost always/always
• Research has shown that most Ss cannot
reliably distinguish between more than six or
seven levels of response
– four to five scale points are sufficient
• Distinguish between a neutral and no response
– when you don’t want to have a neutral option
you can sometimes have a “prefer not to
answer”, “not applicable”, or “no basis for
judgement”

25. Qualities of Good
Questionnaires
• Brings out the truth
– Qs must be non-threatening
– if your Q has sensitive items, be sure to
clearly state your policy on confidentiality
• Ask for an answer on only one
dimension
– ex. Where you satisfied with the time and
location of your class?
• Can accommodate all possible answers
• Has mutually exclusive options

26. Qualities of Good
Questionnaires
• Produces a variety of responses
• Follows comfortably from the previous
question
– transitions between Qs should be smooth
– grouping Qs that are similar will make the Q
easier to complete
• Do not presuppose a certain knowledge
• Do not imply a desired answer
– “Don’t you think that the textbook should be
changed?”
• Do not use emotionally loaded or
vaguely defined words; unfamiliar words
or abbreviations

27. Questionnaire Administration
• Sample
– Difference between sample and population
• Different types of samples
– Convenience/opportunity: most common
– Snowball: where researcher asks a group to
recommend others who have similar
qualities
– Quota: defines certain subgroups within a
population and samples according to
population (i.e. male vs females in UAE
tertiary education)
– Random: selection by random basis; said to
be the most powerful sampling technique

28. Questionnaire Administration
• How large should the sample size be?
– No hard/fast rules
– 10% of population is generally ideal
recommendation
– Basic requirement is normal distribution
which equates to at least 30 Ss (Hatch &
Lazarton, 1991)
– For statistical significance at least 50
– For multivariate statistical procedures like
factor analysis at least 100

29. Issues in Questionnaire
Administration
• Most common way to administer is by
mail or online
• The cover letter/cover email
– Necessary to ‘sell’ the Q & create rapport
with Rs
– What they should include?

30. Ethical Principles of
Questionnaire Research
• Five Ethical Principles (Oppenheim,
1992; Sudman & Bradburn, 1983)
– No harm should come to Rs as a result of
their participation
– Rs right to privacy should always be
respected
– Rs should be provided with sufficient info to
complete the Q
– In the case of children, permission should be
sought from caretakers
– Researcher should not promise a higher
degree of privacy and confidentiality than
he/she can deliver

31. Issues in Questionnaire
Research
• Rs that tend to “sit the fence”
– relevant for Likert scales with odd
numbered response options
• Sensitive issues
– Difficult to obtain truthful answers to
sensitive questions.
• Controversial topics
• Various biases in research

32. Conclusion
• Qs must be carefully designed to
yield valid information.
• Attention must be paid to ensure
that individual questions are
relevant, appropriate, intelligible,
precise and unbiased.
• The order of Qs must be carefully
arranged and the layout of the Q
must be clear.

33. Final Thoughts
• In writing questionnaire items
”no amount of textbook admonition
can take the place of common
sense.” (Moser & Kalton, 1971:310)

34. Presenter contact
• Dr Christine Coombe
• TESOL President (2010-2013)
• ccoombe@hct.ac.ae