Questionnaire data collection tool. Practical guide to questionnaire development, design and validation

1. Developing Data Collection Tool
“Practical Guide to Questionnaire Design”
Dr. Majdi N. Al-Jasim
Consultant Family Medicine

2. OBJECTIVES
By the end of this lecture, you should be able to know:
 What is questionnaire and its advantages and
disadvantages.
 Common ways of administering questionnaire.
 Types of questions in the questionnaire.
 Common pitfalls in writing questionnaire questions.
 How to develop and design your questionnaire.
 How to validate questionnaire.

3. It is the tool or instrument used to collect data and
information of the topic of interest.
Data could be:
▪ Primary
▪ Secondary
Data Collection Tool

4. ▪ Freshly collected data by you that is used for the
original purpose it is collected for (your specific
objectives in your research).
▪ Time, energy and or money may be needed to
achieve the collection.
Examples:
Observational and experimental studies, personal
interview, and questionnaire.
Primary Data

5. ▪ Refer to re-usage of pre-existing data that have been
collected by someone else for another purpose.
▪ Relatively no time, energy or money needed to
achieve the collection but it could be outdated and
doesn’t fully serve your original purpose.
Examples:
Taking data from governmental census, published
articles, book, newspaper, websites… etc.
Secondary Data

6. Questionnaire

7. The questionnaire is a set of questions, focused on
specific objectives of topic of interest, used to gather
information from target responders.
What is Questionnaire

8. ▪ Cost effective way to gather
information.
▪ Less time and energy needed to
administer.
▪ Relatively easy to analyse.

9. ▪ Low respondent rate.
▪ Chance of misinterpretation and
misunderstanding.
▪ Hard to chase missing data.

10. ▪ As short as possible, only long enough to get the
essential data.
▪ Attractive in appearance, and neatly arranged.
▪ Directions are clear and complete.
Characteristics of Good Questionnaire

11. How is Questionnaire Administered
No method consistently outperforms others

12. Question
Types

13. This type of question required respondents to choose
one or more pre-set answers.
Multiple Choice Question
How many cigarettes do you smoke per day:
□ I don’t smoke
□ Less than 10 cigarettes
□ 10 – 20 cigarettes
□ More than 20 cigarettes

14. This type of question present answers in ordinal scale in
which order does matter.
Ordinal Question
What is your current educational level:
□ Illiterate
□ Primary school
□ Intermediate school
□ Secondary school
□ University
□ Higher education

15. This type of question required respondents to choose
only one of two possible answers.
Dichotomous Question
What is your gender:
□ Male
□ Female
What is your current marital status:
□ Married
□ Unmarried

16. This type of question required respondents to choose
only one answer on a scale.
Likert Scale Question
Folic acid intake during pregnancy can prevent fetal Down Syndrome:
□ Strongly agree □ Agree □ Neutral □ Disagree □ Strongly disagree
How frequent you visited emergency department in the last month:
□ Very frequent □ Frequent □ Rarely □ Very Rarely □ Never

17. Bipolar questions are the ones having two extreme
answers written at the opposite ends of the scale. The
respondents are asked to mark their responses between
those two.
Bipolar Scale Question
How would you describe the services provided by our training program:
Efficient □ □ □ □ □ □ Inefficient
Reliable □ □ □ □ □ □ Unreliable

18. A question that is answered only if the respondent gives
a particular respond to a previous question.
Dependent Question (Jump Question)
Q1. Did you have abortion previously:
□ Yes
□ No
If answer is (Yes), please answer Q2;
otherwise go to Q3
Q2. About how many times you got
abortion:
□ Once
□ Twice
□ More than twice

19. In this type of question the researcher doesn’t provide
any options to answer. Rather, he gives the respondents
an opportunity to express their opinion.
Open Ended Question
Please mention two natural resources of folic acid:
A. ………………………………………………………………………………….……….
B. ………………………………………………………………………………….…..…….

20. When you may have a partial list of answer choices, but
you may still uncertain about other possible responses.
Mixed Question
Do you have any of the following chronic disease
(you may choose more than one):
□ Hypertension
□ Diabetes Mellites
□ Bronchial asthma
□ Epilepsy
□ Anemia
□ Other (specify) ………………………………….………………

21. Why Does Question
Type Matter?!!
It is very important for analysis and
coding later on

22. Common
Mistakes

23. What could be wrong with this question?
Common Mistakes
How many children
do you have?
□ One
□ Two
□ More than two
How many children
do you have?
□ I have none
□ One
□ Two
□ More than two
You should cover all possible options

24. What could be wrong with this question?
Common Mistakes
What is your age?
□ 0 – 18 years
□ 18 – 30 years
□ 30 – 45 years
□ 45+ years
What is your age?
□ 0 – 18 years
□ 19 – 30 years
□ 31 – 45 years
□ 46+ years
Options should be mutually exclusive with no overlapping

25. What could be wrong with this question?
Common Mistakes
Options should be balanced and unbiased
How do you rate our services:
□ Very excellent □ Excellent □ Very good □ Good □ Poor

26. What could be wrong with this question?
Common Mistakes
A leading question that ‘leads’ respondents to a particular
response.
Our training center is well known for its strong training and preparing candidates for
passing clinical exams. How satisfy are you in our center?
□ Very satisfied □ Satisfied □ Neutral □ Unsatisfied □ Very unsatisfied

27. What could be wrong with this question?
Common Mistakes
What do you think of our new website color, layout, font size, font color, image quality,
topics variation and its new logo?
□ Excellent □ Good □ Average □ Bellow average □ Poor
Asking more than a question at once. This happens when researcher may
not see them as mutually exclusive ideas. But respondents may have a
different view and will not know how to answer the question.

28. What could be wrong with this question?
Common Mistakes
Use opt-out (NA) responses appropriately
How do you rate appointment services in King Fahad Hospital - Hofuf:
Good Average Poor
Website
Phone
Reception
NA

29. How to get
Questionnaire

30. This can be obtained from previously validated
questionnaire on the same topic of interest. However, it still
needs further validation by Pilot Study.
Pre-made Existing Questionnaire
You SHOULD take PERMISSION from
the author of the questionnaire
before using it (ethical issue).

31. Constructing new questionnaire seems to be a very
annoying task. But if you follow the following steps, you will
be able to develop good, validate and yet reliable
questionnaire yourself.
Developing New Questionnaire

32. Questionnaire
Development Steps

33. Review Your Objectives
Brainstorming
Define Parameters
Questions Development
Validation
Expert Review Pilot Study Revise Cronbach Finalize

34. In this step, you should specify your specific objectives.
Your final questionnaire SHOULD FULFIL ALL SPECIFIC
OBJECTIVES of your research.
Step 1: Review your objectives
Keep in mind your research will
be evaluated based on your
specific objectives.

35. Here you should ask yourself, what things can lead to your
problem of interest. Extensive literature review about the
topic can help you during storming phase!
Example:
Let’s say your objective is “to determine factors that affect
physicians knowledge about osteoporosis”
What could be the factors that affect physicians
knowledge?
Step 2: Brainstorming – Problem Analysis

36. Step 2: Brainstorming – Problem Analysis
Factors affecting
physician knowledge
of osteoporosis
Years after
graduation
Having post-
graduation training
Having education
resources: books,
internet
Exposure to cases of
osteoporosis
Physician
gender
Physician
age
Physician
medical
school
Physician
specialty

37. Here put the main head of parameter and then the way
of measurement.
Example:
Let’s go back to “Factors affecting physician knowledge
of osteoporosis”
Step 3: Define Parameters

38. Factors affecting
physician knowledge
of osteoporosis
Years after
graduation
Having post-
graduation training
Having education
resources: books,
internet
Exposure to cases of
osteoporosis
Physician
gender
Physician
age
Physician
medical
school
Physician
specialty

39. Step 3: Define Parameters
Parameters Measurement
Age Number of years (continuous data)
Years after graduation Number of years (continuous data)
Highest degree obtained MBBS, Diploma, Board, Master, PhD
Medical school Local, Foreign
Gender Male, Female
Knowledge level Set of questions with likert scale
Resources Set of dichotomous data (Yes / No)
“Factors affecting physician knowledge of osteoporosis”

40. After you define parameters, now have some time to put
appropriate wordings into questions.
BE ORGANIZED:
Make section for each group of parameters that are
related to each other.
Step 4: Develop Relevant Questions

42. Keep in mind:
Some objectives can’t be assessed by single item in the
questionnaire. These will need several items (questions) in
order to have good assessment.
Examples:
▪ Knowledge level
▪ Attitude level
▪ Awareness level
Step 4: Develop Relevant Questions

43. 10 items are used to assess
ATTITUDE LEVEL of objective of
interest

44. 10 items are
used to assess
people
perception
about Family
Medicine

45. Now you have finished construction and
developing. In next steps we will validate
our developed questionnaire.

46. Questionnaire
Validation

47. Questionnaire Validation
Validity of an instrument refers to the
degree to which an instrument
measures what it is supposed to be
measuring.

48. Steps of Questionnaire Validation:
1. Expert Panel Review
2. Pilot Study
3. Revise items as needed
4. Cronbach’s Alpha
5. Finalization
6. Sit back, relax, and let out a guttural laugh celebrating
your accomplishment because you can start data
collection using your questionnaire!.
Steps of Questionnaire Validating

49. Validating #1 – Expert Panel Review
Send your developed questionnaire with your specific
objectives to group of experts in the topic of interest (at
least 2 experts) for their input and rating of questionnaire
items.
This step will ensure that you didn’t miss any important item
and or mixed unrelated item to the questionnaire.

50. Validating #1 – Expert Panel Review
Example:
Let’s go back to the questionnaire you design to assess
“Factors affecting physician knowledge of osteoporosis”
Who would be the experts?
Any expert who deals directly with such case and has
experience with researches. Here, this could be
endocrinologist or geriatrician.

51. Validating #2 – Pilot Study (Pre-test)
In this step you will test the questionnaire you developed
after experts review on a small sample before conducting
your main data collection.
Take a small sample (ideally 10% of actual calculated
sample size. However, not less than 20 participants) that
has the same characteristics of your actual sample, then
distribute the questionnaire to them.

52. Validating #2 – Pilot Study (Pre-test)
Instruct participants to think loud:
This will helps reveal unanticipated problems as well as It
can help see if the participants are understanding your
questions appropriately.
The average time to complete questionnaire then
calculated.

53. Validating #3 – Revise Items
After Pilot Study, some modification and rephrasing of
some items may be needed. Some items might have
problem in understanding and so need major
modification.
If modification is major to most of questionnaire items,
another run of Pilot Study is needed after modifications.

54. Validating #4 – Cronbach’s Alpha
This score is used to assess statistical reliability (validity) of
the questionnaire that uses Likert scale, or Bipolar scale.
Statistical reliability means you can trust this item to
measure the desired outcome.
Cronbach’s Alpha is calculated by analyzing your Pilot
Study data in SPSS program.

55. Validating #4 – Cronbach’s Alpha
Any Cronbach’s Alpha score more than or equal 0.7 (70%)
is considered acceptable.
In this step, you should ask for help from experts especially
if the score is less than 0.7

56. Validating #5 – Finalization
Final Editing to your questionnaire to look neat!
Suggested way of finalization:
➀ Topic title:
It should be very clear and precise.
➁ General introduction:
This has a description of the purpose of study. The respondent is
assured of confidentiality of information. Honest answers are also
requested.

57. Validating #5 – Finalization
➂ Additional information:
Includes the full contact information of the researcher and or
administrator. May include a promise that a copy of the summary of
the final report would be sent to the respondent on request.
➃ Specific instruction:
This offers concise demonstration on how to carry on with the
responding to the questionnaire.

58. Validating #5 – Finalization
➄ Questionnaire items:
It is the main part of the questionnaire, to be clearly separated from
the previously parts.
➅ Thank you:
May end the questionnaire.

59. Example of Questionnaire Finalization

60. Example of Questionnaire Finalization
Clear Big Title

61. Example of Questionnaire Finalization
General Introduction

62. Example of Questionnaire Finalization
Confidentiality Assured

63. Example of Questionnaire Finalization
Additional Information

64. Example of Questionnaire Finalization
Specific Instruction

65. Example of Questionnaire Finalization
Questionnaire items in sections

66. Questionnaire
Response Rate

67. Questionnaire Response Rate
Questionnaire response rate is referred to how much
respondents had completed the questionnaire and
returned it back to the researcher.

68. Questionnaire Response Rate
The minimal accepted response rate must be ≥ 80%
The smaller the sample, the more response rate is required.
𝑹𝒆𝒔𝒑𝒐𝒏𝒔𝒆 𝑹𝒂𝒕𝒆 =
𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑅𝑒𝑠𝑝𝑜𝑛𝑑𝑒𝑛𝑡𝑠
𝑆𝑎𝑚𝑝𝑙𝑒 𝑆𝑖𝑧𝑒
× 100

69. Tips to Increase Response Rate
 Shorter questionnaire.
 Good communication:
▪ Contact the respondents beforehand
▪ Follow-up contact with non-responders
 Offer rewards:
▪ Monetary incentives.
▪ Physical rewards.
 Assure confidentiality.

70. Review Your Objectives
Brainstorming
Define Parameters
Questions Development
Validation
Expert Review Pilot Study Revise Cronbach Finalize
Summary
Questionnaire
development

71. THANK
YOU
Any Question?!