Qdesign

1. Questionnaire Design

2. Questionnaires
• Standardized, structured instrument for
collecting/gathering data.
• … a prepared set of questions (or measures)
to which respondents or interviewers record
answers.
• Purpose
– To obtain information that cannot be easily
observed or is not already available in written or
electronic form

3. Step 1: Initial Considerations
Step 2: Clarification of Concepts
Step 3: Typology of a Questionnaire
Step 4: Pre-testing of a Questionnaire
Step 5: Administering a Questionnaire
Steps in Questionnaire design

4. Questionnaire Design:
Initial Considerations
• Clarify the nature of the research problem and
objectives.
• Develop research questions to meet research
objectives.
• Define target population and sampling frame.
• Determine sampling approach, sample size, and
expected response rate.
• Make a preliminary decision about the method of
data collection.

5. Research Questions (Recall)
Examples:
• Is staff motivation a problem in this organization?
• Do employees in this organization support
diversity in the workplace?
• What factors influence productivity in this
organization?
• What are the most important factors influencing
the purchase of a laptop computer?
• What are the good and bad issues about the
government’s policies on fighting corruption?

6. When to use questionnaires
• Questionnaires are useful in gathering
information from key respondents
about:
– Attitudes
– Beliefs
– Behaviors
– Characteristics

7. When to Use Questionnaires
• Questionnaires are valuable if
– Respondents are widely dispersed
– Many members are involved with the project
– Exploratory work is needed
– Problem solving prior to interviews is
necessary
– Others…

8. When to use questionnaires
• Best used when:
– There is a large sample
– You want fairly straightforward information
– You want standardized data from
identical questions
– You are more interested in what occurs
rather than why or how

9. Limitations of questionnaires
• Can be superficial
– difficult to capture the richness of meaning
• Cannot deal with context
– information is collected in isolation of environment
• Information is not causal
– cannot attribute cause-effect relationships
• Information is self-report
– which does not necessarily reflect actual behavior

10. Questionnaire Language
• Questionnaire language should be:
– Simple
– Specific
– Free of bias
– Not patronizing
– Technically accurate
– Addressed to those who are knowledgeable
– Appropriate for the reading level of the
respondent

11. Types of surveys
• Mail
– cheapest, wide coverage, standardized, low
response rate (?)
• Telephone
– medium cost, wide coverage, medium response
rate, standardization depends on interviewer
• Face to face
– most expensive, coverage depends on personal
contact, highest response rate

12. Basic Levels of Scales (recall)
• Nominal
– Gender (1=male; 2=female)
• Ordinal
– Rank order preference
• Interval
– Rating scales (e.g. Likert, semantic
differential)
• Ratio
– Absolute zero point

13. Designing a questionnaire
• Is a questionnaire (the most) appropriate?
• Identify the resources that are available.
• Decide what information you need.
• Select items for inclusion.
• Design the individual questions.

14. Developing your questions
• Search the literature
– bibliographic databases
– citation searches of key articles
• Preliminary research
– focus groups
– key informants interviews

15. Broad considerations
• Sequencing of questions.
• Identification of concepts.
• How many questions are required to capture each
concept.
• Question wording.
• Overall length of questionnaire.
• Placing of sensitive questions.
• Level of measurement.
• Open-ended vs. close-ended questions.

16. Type of information
• Knowledge - what people know
• Opinions, attitudes, beliefs, values - what
people think about an issue
• Behavior - what people do
• Attributes - what are people’s characteristics
• Remember - based on self-report

17. Knowledge
What is the recommended interval between
maintenance/service of a grader?
 200 hours
 300 hours
 500 hours
 Not sure
At what age should kids join school?
 2 years
 3 years
 4 years
 Not sure

18. Opinions etc.
What do you think are the major issues
affecting Uganda’s economy at the
moment?
_________________________________________
_________________________________
Whom do you think will be elected
speaker in the 9th
parliament?
_________________________________________
______________________

19. Behavior
• Do you have an insurance plan for your
family?
Yes
No
• Do you often use the gym for your
personal fitness?
• Do you use fortified products in your
family?

20. Attributes
• When did you graduate from university?
___________________________________________
• Where do your kids go to school?
________________________________
• Do you earn income from extra jobs?
_____________________________

21. Question Types
• Questions are designed as either
– Open-ended
• Try to anticipate the response you will get
• Well suited for getting opinions
• Useful in explanatory situations
– Closed
• Used when all the options may be listed
• When the options are mutually exclusive

22. Open-Ended and Closed Questions
Open-ended Ease of: Closed
Slow Speed of completion Fast
High Exploratory nature Low
High Breadth and depth Low
Easy Ease of Preparation Difficult
Difficult Ease of Analysis Easy

23. Types of questions
• Open-ended question(s)
– What…? Why…? How…?
– No predetermined responses given
– Able to answer in own words
– Useful for exploratory research and to generate ideas
– Flexible
– Requires skill in asking questions and interpreting
results
– Answers can lack uniformity and be difficult to
analyze

24. Open-ended questions
• What do you think about the quality of
products made in Uganda?
__________________________________________
__________________________________________
_______________________________________
• How do you typically decide which
restaurant you will eat at?
• How are your investment funds performing?

25. Types of questions
• Close-ended question(s)
– Designed to obtain predetermined responses
(Yes/No; True/False; strongly agree-strongly
disagree, etc..)
– Easy to count and analyze
– Easy to interpret
– May not have catered for all possible answers
– Questions may not be relevant or important

26. Closed-ended questions
• The curriculum of Mak encourage graduates to become
job creators:
• Which country in Europe have you traveled to in the last
one year?
a) Belgium, b) Germany, c) the UK, d) Holland, e) Spain.
• Please rate the quality of groceries from Uchumi Supermarket:
Poor Fair Good

27. Opening questions
Screening or filter questions
• … are used to ensure respondents included in the
study are those that meet the predetermined criteria
of the target population.
• E.g. Tonight we are talking with individuals who are 18
years of age or older and have 50% or more of the
responsibility for banking decisions in your household.
Are you that person?
___ Yes ___ No.

28. Opening Questions
Rapport Questions
• … are used to establish rapport with the
respondent by gaining their attention and
stimulating their interest in the topic.
• Have you watched any good movies in the
last month? __ Yes __ No.
• What is your favorite restaurant in Kampala?

29. Filter questions
• Filter questions useful to ensure respondents
only answer relevant parts of questions
• Unfiltered:
– If you use a drawing software program, which one
do you use?
• Filtered:
– Do you use a drawing software program?
 No - skip to next question
 Yes - which one?

30. Branching Questions
• … are used to direct respondents to answer
the right questions as well as questions in the
proper sequence.
– Have you seen or heard any ads for mobile
phones in the last 30 days?
– If “No”, go to question #10.
– If “Yes”, were the ads on radio or TV or both?
– If the ads were on TV or both radio and TV, then go
to question #6.
– If the ads were on radio, then go to question #8…

31. Getting the question right
• Question wording
– Questions need to be clear, simple and precise
– Poorly written questions lead to ambiguity and
misunderstandings and can be wasteful
• Responses
– open, closed, what type of response set will you
use?

32. Common problems with wording
• Leading questions: a question that suggests or
implies a certain answer.
– Do you prefer being examined by a doctor of
your own sex?
• Better: Would you rather be examined by a:
 Male doctor
 Female doctor
 Either/doesn’t matter

33. Leading questions
• The bandwagon effect
• e.g. Most Ugandans have stopped eating junk food.
Do you eat junk food?
• Partially mentioning some alternatives
• e.g. which fast food restaurant do you prefer,
Nandos or others?
• Questions with the phrase:
• Like “… Don’t you think that ..”

34. Common problems with wording
• Vague questions:
– Taken altogether, how happy are you with your
stay in hospital?
– Have you seen a doctor during the past year?

35. Common problems with wording
• Vague questions
– Taken altogether, how happy are you with your
stay in hospital?
– Overall, how would you describe the care you
received in hospital?

36. Common problems with wording
• Have you seen a doctor during the past
year?
• In the last 12 months, have you visited a
general practitioner?
• How long has it been since you last visited a
general practitioner? (within the last month,
between 1 and 12 months ago, more than
12 months ago)

37. Common problems with wording
Biased or value-laden/loaded questions:
• A question that is designed to suggest a
socially desirable answer.
– Usually it is emotionally charged.
– E.g. Do you think fizzy drinks are bad for teeth?
– E.g. In your opinion is it fair that Multiplex should be
harassing car owners with parking tickets?
• Framing question such that honest answer is painful
or embarrassing  use counter biasing statement.

38. Common problems with wording
• Biased or value-laden questions:
– Do you think evidence-based medicine is a
waste of time?
– What do you think of evidence-based medicine?

39. Common problems with wording
• Threatening questions
– How often do you smack your child?
– Do you know enough about treating patients at
risk of stroke?

40. Common problems with wording
Threatening questions
– How often do you smack your child?
– How often do you use each of the following to
discipline your child?
– Do you know enough about treating patients at
risk of stroke?
– How would you rate your knowledge of X for
treating patients at risk of stroke (I know very little, I
need to learn a little more, I need to learn a lot
more etc..)

41. Common problems with wording
Double-barreled questions
– two concepts in one question
• Example: Have you had a neck ache or a
back ache since your last visit?

42. Common problems with wording
• Double-barreled questions
– Have you had a neck ache or a back ache since
your last visit?
– Since your last visit, have you had any of the
following symptoms (tick as many that apply):
 Neck ache
 Back ache
 Headache

43. Common problems with wording
• Negative questions:
– avoid using negative wording ‘not’, ‘rarely’,
‘never’, or words with negative prefixes ‘in-’,
‘im-’, ‘un-’.
– Doctors should not be required to see patients
outside surgery hours: agree/disagree
– Doctors should be required to see patients
outside regular hours:
Agree/Disagree

44. Common problems with wording
• Complex questions:
– On a scale of 1 to 10, please rate for each of the
12 categories listed below, your level of
knowledge, confidence and experience.

45. Common problems with wording
• Complex questions:
– On a scale of 1 to 10, please rate for each of the
12 categories listed below, your level of
knowledge, confidence and experience.
– Please complete the table below about your level
of knowledge, confidence and experience in
each of the following areas.
– Please complete the table below about your level
of knowledge, in each of the following areas.

46. Responses
• Closed-ended questions are usually followed by
a set of responses
• Choose type of scale:
– nominal
– ordinal
– continuous (summed items with ordinal response
scale)

47. Responses
Nominal
• Are you:
 Male
 Female
• What is your marital status:
 Single
 Married
 Widowed
 Divorced
 Separated
 Other

48. Responses
• Limited choices of responses, lack of consistency in what
a yes/no, agree/disagree response means
Do you have trouble climbing stairs?
• Attitudes and behaviors lie on a continuum
To what extent do you experience difficulty when
climbing stairs in your house?
 None
 A little
 Quite a bit
 A lot
 I do not have stairs in my house

49. Responses
• Ordinal
• What is the highest level of education
you have reached:
 Did not complete primary school
 Completed primary school
 Did not complete O-Level
 Completed O-Level
 HSC or equivalent
 University

50. Problems with responses
• Effort required to answer questions
– E.g.: During your last consultation with your
doctor, did the doctor discuss medications to
help lower your blood pressure?
– What is meant by discuss?; relies on recall of
discussion
– Many respondents will tick a response that is
‘satisfactory’; that is, to just ‘tick a box’.

51. Problems with responses
• Fatigue/boredom/disinterest
– agree with everything
– just say ‘don’t know’
– always choose first response
– ‘randomly’ respond without considering the
question
• Social appeal
• Aversion to extreme ends of the scale

52. Problems with responses
Minimizing fatigue/boredom
• Keep task simple
– E.g. easier to recall more recent events
• Keep words short and easy to understand
• Maintain motivation of participants
– ensure task is relevant
• Ask people to justify their responses

53. Problems with responses
• Minimizing social desirability
– is difficult
– instruct that it is ok not to know something

54. Problems with responses
Aversion to extreme ends of scale
• Avoid absolutes ‘never’, ‘always’
• Expand number of categories by including
throw away categories at the end:
– never, almost never, infrequently, sometimes,
usually, almost always, always

55. Problems with responses
Minimizing ceiling effects
• ‘Average’ response doesn’t have to be
middle response

56. Problems with responses
Halo effects
– The tendency for an impression created in one
area to influence opinion in another area.
– often occurs when evaluating individuals
– judgments made on aspects of a person’s
performance influenced by overall impression of
the person
– a global summary just as informative

57. Problems with responses
Order effects
– Maybe more likely to endorse first or last
response
– Preceding questions may influence
responses to questions that follow

58. Problems with responses
• Randomize order of response sets between
individuals
• Randomize order of items within questions
• Maybe possible to randomize order of questions
• Don’t always present ‘positive’ or ‘negative’
sounding response first
• Easier to randomize in computer-assisted interviews
than paper and pen questionnaires

59. Ordering questions
• Sequence should be logical to the
respondents and flow smoothly from one
question to the next.
• Questions tend to flow from:
– general to specific
– impersonal to personal
– easy to difficult

60. Questionnaire Design
• Question Sequence
– Order Bias
– Funnel Technique
– Filter Bias
• Question Layout

61. Do’s of Questionnaire Design
• Include instructions to participant.
• Awaken interest and motivate participation.
– Place interesting questions and/or attention-getting questions early
– Place classification questions at end
• Obtain trust and willingness to disclose non-sensitive
information first.
– Place sensitive or ego-involving questions in middle
• Start with simple questions and then move toward more
complex.
– Use funnel approach (general to specific)
• Use transition statements between different topics.

62. Validity and reliability
• Validity
– question measure what you claim it measures
– problem with self-report
• Reliability
– results are reproducible or consistent with similar
groups of respondents, over time and when
other people administer the questionnaire

63. Layout
• Just as important as wording
• Aim for a professional look
• Tips:
– cover letter/introductory page giving study title,
organization, aims of the survey
– enough space for open-ended questions
– font large enough to read without strain
– consistent and clear instructions
– don’t split questions or answers across pages
– enough white space

64. Designing a questionnaire
• Pilot
– Discuss it with your colleagues
– Pilot it with the target group and as you intend to
administer it
• Evaluate and modify on basis of pilot
• Conduct survey, including protocol for
maximizing response rates

65. Pretesting Questionnaires
• Objective is to identify possible shortcomings of the
questionnaire.
• Approaches could be formal or informal.
• Can assess:
– Clarity of instructions
– Cover letter
– Clarity of questions
– Adequacy of codes and categories for pre-coded questions.
– Quality of responses
– Likely response rate

66. Pretesting Questionnaires
• No hard and fast rules:
– Ability to preform meaningful analyses
– Time to complete questionnaire
– Cost of data collection
– Which questions are relevant
– Whether questions have been overlooked
– Sources of bias.