Development of questionnaire for survey research. Understanding how research aims/goals, variable categories, variable types and measures inform the development of the questionnaire instrument. Content covers types of quantitative instruments and the step-by-step development process based on the research focus using a sample study.

1. Dr. Bosede I. Edwards
Adj. Lecturer, Raffles University, Iskandar Puteri, Malaysia
Snr. Consultant (Learning & Research), Arrows Education, Malaysia

2.  Understanding your research
 Quantitative instruments
 Significance & Types
 Choosing instruments
 Questionnaire vs Survey
Bosede I.Edwards

3. Standardization/uniformity
Easy comparison of feedback
Higher reliability
Generalization of results
Bosede I.Edwards

5. Continuous
Numerical, directly measured
Numbers are meaningful
Distance between consecutive
numbers are equal
Categorical
Non-numerical, discrete categories
Numbers are only representative
Types of
Variables
Categorical
Nominal
(Gender, Nationality)
Ordinal
(Stage 1 Cancer;
SD/D/A/SA)
Continuous
Interval
(Age, Height,Temp.,,
Income,Test score)
Ratio
(Heart rate, BMI, GDP,
Happiness Index)
Bosede I.Edwards

6. Independent Variable (IV)
 Fixed; not measured
 Fixed conditions on which the DV is measured
 The ‘cause’; results in changes in the DV
 Value does not depend values of other variables
 E.g. Age, Gender, Education, Country of Study, etc.
Bosede I.Edwards
Dependent Variable (DV)
 Not fixed; measured
 Value depends on the IV
 The ‘effect’; value changes with IV
 Income, Anxiety level, BMI, Engagement, etc.

7. Other Variable Types
 Mediator variable
 explains the process through which two
variables are related
 Moderator variable
 changes causal (strength & direction)
relationship between DV and IV
 Intervening variables
 hypothetical; for explaining relationships
between DVs and IVs
Gender
Work Experience Salary
#Publications
IV DV
Level of Educ.
Int’l Educ.
Experience
Perception of
Interracial
Marriage
Intervening Variable
Bosede I.Edwards

8. IV causes a change in DV
e.g. Age causes a change in height/IQ
Gender causes a change in perception of colour
DV cannot cause a change in the IV
e.g. Height/IQ cannot cause a change in Age
Perception of colour cannot cause a change in Gender
Bosede I.Edwards

9. Some variables can take on the
position of DVs under certain
conditions and IVs under a different
condition.
E.g. Consider these 2 hypotheses:
H1: A healthier diet leads to more
activity.
H2: More activity leads to increased
happiness.
For H1:
DV: Activity
IV: Diet
For H2:
DV: Happiness
IV: Activity
Bosede I.Edwards

10. DV as… Categorical Measures Continuous Measures
Activity Sedentary vs Active WFH vs WFO Exercise Hrs/week
Diet Bland vs Sweet Low vs High-Sugar Calories/day
Happiness Happy vs Unhappy Self-report/Rating Happiness Index
SES Low . Med . High Income Range Income
Anxiety High/Low Self-report Anxiety Index
Performance Rating Scales Self-evaluation Test Scores
Age Above/Below Age Group Years
Teaching Method Active vs Inactive Lecture/non-Lecture ABC vs XYZ
Body Composition Normal vs Obese Body Fat/BMI Cholesterol Level
Bosede I.Edwards

12. • Questionnaire: Survey Instrument
• Survey: Research technique for estimating
population parameters from sample statistics.
• Sample: Selected reps of population.
• Questionnaire Administration: Interviews, Emails,
Online questionnaires
• Googleform, SurveyMonkey,TypeForm, etc).
Bosede I.Edwards

13. • Simple, logical, unobtrusive, reliable, practical
• Introduction | Demographics | Body | Closing
• Balance between too short and too long
Bosede I.Edwards

14. Sections of the questionnaire
Type & Levels of DV & IV
Demographics: Personal info, MVs
Practicality versus reality
Aim/goal of study
Open-/closed-ended design
Sensitivity in design (cultural, gender,
personal, administration)
Pre-study/Piloting
Bosede I.Edwards

15. Research findings on questionnaire length:
 Length directly affect response rates, survey costs, and data quality.
 “Increasing the length of a questionnaire from 5 to 7 pages reduces response rates
from women aged 70 yrs and over. However, it does not seem to affect the quality of
responses to questions near the front of the questionnaire”.
 “Shorter surveys have higher completion rates, which means they have overall
better data quality.”
 Long surveys…negatively affect response rate, abandonment rate,
thus impacting sample representativeness, and data quality.
 Useful RoT: Sample Size = (No. of Items x 5-10).
Bosede I.Edwards

16. What well-known/popular instruments are available?
What do you need to measure?: Study gap
Adopting: Use an instrument nearly verbatim
Adapting: Alter significantly (revalidation required)
Developing: Create from scratch
Degree requirements
Bosede I.Edwards

17. Analytical Goals
Types of measures
Type of respondents
Types/number of IVs
Types/number of DVs
Other types of variables
 Associations
 Predictions
 Group differences
 Data reduction
 Reliability
Bosede I.Edwards

18. Test Type Purpose Basic Requirements
Pearson's (product-
moment) correlation
Strength &
direction of
linear
relationship
2 continuous variables
Point-biserial
correlation
1 continuous, 1
dichotomous
Pearson's partial
correlation
2 continuous, controlled
for 1
Cochran-Armitage test
of trend
Ordinal IV and a
dichotomous DV
Bosede I.Edwards

19. Test Application Details/Comments
Linear
Regression
Predict
(magnitude &
direction of) linear
relationships
between 2
variables: 1 cont.
IV, 1 cont. DV; .
Predict value of DV based on value of an
IV (i.e. determine if there’s linear
relationship & how much of the variation
in the DV is explained by the IV; also
direction & magnitude of any
relationship; and values of the DV based
on diff. values of the IV.
Example: Predict lecturers' salaries based on the number of years they have graduated
from a bachelor’s (i.e., DV = "salary" and IV = "years after bachelors").
It is also possible to determine how much of the variation in lecturers' salaries can be
attributed to the number of years they have graduated from a bachelor’s programme.

20. Test Description Requirement
Independent-
samples T-test
Statistical significance of mean
difference between 2 two groups on a
continuous DV
Continuous DV
Paired-samples
T-test
Statistical significance of mean
difference between paired
observations
Cont. DV,
dichot. IV
NOTE: Paired-samples could be any of:
• Same subjects tested at 2 time points or under 2 different conditions on the same DV.
• Two groups of participants that have been matched (paired) on one or more
characteristics (e.g., IQ, age, gender, etc.) and tested on one DV.
• Also known as: Dependent or Repeated measures t-test, or simply the ‘paired t-test’.
Bosede I.Edwards

21. Research Goal
Associations
Predictions
Group
Differences
Other Goals
Variables &
Type
IV & Type
DV & Type
Other Var &
Types
Determin Test
Determine
Measures
Choose/ Develop
Instrument
Bosede I.Edwards

22. • Be sure of the actual measure
• Link to study goals: RQs/ROs, Litt. Review, Methodology
• When possible, work with Continuous DVs
• Where possible, work with continuous/categorical IV(s)
• If working with cat. IVs, consider work with 2 levels
• Reduce no of variables to what’s absolutely necessary
• Avoid complicated designs
• Choose scale carefully: Likert/Likert-type; X-point, etc.
Bosede I.Edwards

23. A study design for questionnaire development

24. You have won a RM2 million publicity contract to promote a
newly introduced University in Nigeria. The university
seeks to appeal to students from both the lower and middle
class equally. Your client requests proof that the advert you
have created appeals equally to members of the lower and
middle class. That is, you need to answer the Q: “is the way
that lower and middle class citizens engage with the advert
the same?”
Bosede I.Edwards

25. Bosede I.Edwards

26. To examine if the IV (SES)
has an effect on the DV
(engagement)
OR To answer the Q:
“Are there differences in
engagement between levels
of SES?”
Statistically…
 “Is the mean engagement score with the
advertisement different for males and
females?”
Hypothetically…
 This should be a null hypothesis, H0 (the
advert should be similarly engaging, so,
the expectation is that:) “there is no
significant difference in the mean
engagement score for persons in the
lower and middle SES classes”.
Bosede I.Edwards

27. Designing a questionnaire to measure engagement
 The questionnaire measures overall engagement score.
 Dependent Variable (DV) = Overall Engagement Score
 Independent Variable (IV) = SES (Socio-Economic Status)
 IV has 2 groups/levels:“Lower SES" and “Middle SES".
Bosede I.Edwards

28. Example of a study design for questionnaire development

29. Establish
conceptual
understanding
Develop
interview
guide
Collect &
analyze
data
Conceptualize:
Generate domains
through codes &
themes
Testing
Review of Literature;
Identify adoptable tools
Create Interview Q items
based on gap identified
Interview respondents &
gather useful data
Interview respondents &
gather useful data
Test, modify, retest
5
4
3
2
1
Bosede I.Edwards

30. For every 600 persons vaccinated…
VACCINE A
200 people will be saved
VACCINE B
There is ⅓ chance of saving 600
and ⅔ chance of saving nobody
VACCINE C
400 people will be killed.
VACCINE D
There’s 33% chance of killing
nobody.
VACCINE F
Over 60% of vaccinated persons
will die.
Choose Your Vaccine
VACCINE E
There is 63% chance of killing all
600 people.
Bosede I.Edwards

31. Litt. Rev; Identify concepts, constructs
(indicators), measures, etc.
Review the literature
Identify previous measures
Identify measures of your gap
Are there instruments that
address your objectives/RQs?
Literature on
Engagement
Measures
related to
ROs/RQs
Adopt Adapt
Measures
related to
ROs/RQs
Develop
Bosede I.Edwards

32. Create Interview Q items based on literature
findings or gaps identified
• Assumption: Available instruments do not address ROs:
• Create Qs whose answers will help in providing measures of the
constructs being investigated
• Example Objective: Ad appeal
• What factors do you use in judging the quality of advertisements?
• Do you have different attitudes to ads depending on where you see them?
• What factors are important in making you watch an ad to the end?
• If you really like an ad/do not like an ad, what do you do?
Bosede I.Edwards

33. Analyze feedback; identify themes, develop
items
• Analyze feedback and develop questionnaire items based on
the themes identified in the analysis.
• Example: Result shows persons across both SES engage with
advertisements based on:
• How interesting/clear/logical the advert is
• If they found the ad useful/practical, they share, download, save, etc.
• Offensive/discriminatory (sexist, racist, tribalistic, etc.) content.
Bosede I.Edwards

34. Analyze feedback; identify themes, develop
items
• Important themes are therefore identified as:
• Perception of Ad
• Interesting, Useful, Logical, etc.
• Ad Sharing
• Share with others, download, save, etc.
• Reaction to offensive/discriminatory content
• Sexism, racism, tribalism, stereotypical, etc.
Bosede I.Edwards

35. Theme-1: Perception of Ad
 I found the ad interesting
 I understand the message immediately
Theme-2: Ad Sharing
 I downloaded the ad
 I shared the ad with others
Theme-3: Sensitivity
 The ad is offensive to…
 The ad appeals to people at my income level
Bosede I.Edwards

36. Litt. Rev; Identify concepts, constructs
(indicators), measures, etc.
Create Interview Q items based on
literature findings or gaps identified
Interview selected respondents; collect
relevant data on concepts and constructs
Analyze feedback; identify themes,
develop items
Pilot-Test, Modify, Retest
Concepts: Income, Age, Educ. Level, No. of Kids
Constructs: Brand Loyalty, Purchase Intent,
Satisfaction, Engagement, Achievement, SES
Create Qs whose answers will help in
providing measures of the constructs
being investigated
Bear in mind measurements while
interviewing; stay focused on the finals
Develop questionnaire items based on
the themes identified in the analysis
Allow for expert review of the questionnaire items, update
and collect data for pilot testing.Validate, Modify & Retest
Bosede I.Edwards

37. Open with an introduction
Assure participants of confidentiality
Pay attention to language & wording
Testing & back-translating of translated
questionnaires with native speakers
Capture a single idea per question
Avoid leading items/double negatives
Close with appreciation
Bosede I.Edwards

38.  The psychometric literature suggests that having
more scale points is better but there’s a
diminishing return after ~11 points (Nunnally
1978).
 A 4-point Likert scale is observed to distort the
results.; 5-point Likert scale data is considered
more accurate than the 4-point data; 7-point Likert
scales are known to be the most accurate.
 A neutral point is preferable; better to know that
respondents were neutral rather than force an
answer.
Bosede I.Edwards

39. Gender: Male□ Female□ | Location: City/Town□ Village□ Age:
__________
S/N Indicators of Engagement 1 2 3 4 5 6 7
1 Perception of Ad
I found the ad interesting
I understand the message immediately
2 Ad Sharing
I downloaded the ad
I shared the ad with others
3 Sensitivity
The ad is offensive
The ad appeals to people at my income
level
Bosede I.Edwards

40. Pilot-test, Modify, Retest
• Start with peer-review of items & structure
• Get expert review of items and structure
• Update and collect data for pilot testing.
• Modify and/or Retest
Bosede I.Edwards

41. • Content/expert validation
• Reliability
• Construct validity
Bosede I.Edwards

42. Expert review
Pre-testing with colleagues
Identify misleading, ambiguous, biased questions
Determine completion time
Indicate the time in the introduction
Understand respondents’ interpretations
Revise, reorganize, pilot-test and modify
Track and report all changes.
Bosede I.Edwards

43. Reliability analysis for quantitative instruments
Test-retest reliability (tests)
Reliability analysis (questionnaires)
Correlations (questionnaires, ratings)
Covariances (questionnaires, ratings)
Cronbach’s Alpha, etc. (questionnaires, ratings)
Bosede I.Edwards

44. Face validity:
 Just the look. Anyone can provide a feedback
Content validation:
 Items are properly related to measures. Expert review.
Construct validation:
 Items/tests measure the intended constructs. Requires testing.
Criterion validation:
 Reflects the use of a criterion (well-established measurement procedure)
to create a new measurement procedure to measure a related construct.
 Can be done by establishing the concurrent or predictive validity.
Bosede I.Edwards

45. Academic study involving human subjects/minors.
You may need the approval to be able to conduct your study
Also required for publishing in some high-impact journal.
The research instrument, research procedure and
methodology is presented for the approval of the board
before conduct.
The board is known as:
 Ethics Committee
 Institutional Review Board (IRB)
 Independent Ethics Board (IEB)
Bosede I.Edwards

46. Issues to pay attention to may include:
The risk posed by the study to participants
Your plans to mitigate the risks
Level of intrusiveness of the instrument
Language and mode of administration
Type of demographic/clinical information to be collected
Bosede I.Edwards

47. Concurrent validity:
 Test participants on
instrument A (new) and
instrument B (well-
known) for measuring
construct XYZ.Test them
again after a short time.
Scores on both instrument
by low/med/high scorers
must be similar.
 Established through
correlation statistics.
Predictive validity:
 Strong, consistent
relationship (correlation)
between the scores from
a new measure, e.g. an
advanced intelligence
(pre-admission) test and
the scores from a well-
established measure, e.g.
GPA scores (1yr later).
Convergent validity:
 Scores on a measure should
corresponds to scores on
measures of related constructs:
e.g. a person’s score on IELTS
should be similar to their score
on TOEFL. If a person’s score on a
new English language skills test
correlates with IELTS/TOEFL
score, there is convergent, hence
construct validity. Divergent
validity is opposite (measure is
unrelated or negatively related).
Bosede I.Edwards

48. The use of a criterion to create a new measurement procedure.
Criterion: A well-established measurement procedure.
Measurement must be of a related construct.
Established through concurrent or predictive validity.
Bosede I.Edwards

49. Everyone loves good design
UI/UX Matters in all visuals
Avoid overcrowding
Use white space well
Include instructions
Choose appropriate typeface.
Bosede I.Edwards

50.  Everyone loves good design
 UI/UX Matters in all visuals
 Avoid overcrowding
 Use white space
 Include instructions
 Choose appropriate typeface.
Bosede I.Edwards

51. Respondents’ written consent
Personally-identifying data (name,
email address, age, income, etc.)
Intrusive data (blood pressure,
body fluid, etc.)
Clinical information (symptoms,
temperature, treatment data, etc.)

53. • Understand your research; what do you hope to achieve?
• What are your analytical goals?: Associations, etc.
• Understand variables and how they affect your research
• Choose concepts, constructs & measures carefully
• Stay simple; Keep categories minimal
• It mustn’t be complex to be good
Bosede I.Edwards

54. All the best!
Bosede I.Edwards

55. https://bosedeedwards.com