QUESTIONNAIRE DEVELOPMENT new.pptx

Developing questionnaires for medical
research
By: Dr Hanaa Bayomy
Assistant professor of Community Medicine

Introduction
• Questionnaire is a measuring device used to
query a population/sample in order to obtain
information for analysis.
• A questionnaire is simply a list of questions
that is completed by or for a respondent.
(Health Research Methodology 2nd ed. 2001
WHO)

Introduction
• Questionnaires are one of the most popular
survey methods. It is the most convenient
way to collect data from large numbers of
often geographically diverse individuals on a
wide range of issues.
• Questionnaire may be administered in a
variety of ways, including face to face,
telephone, post and online.

Introduction
• Questionnaires are good for gathering data about
constructs that are not directly observable e.g.,
student satisfaction with a new curriculum,
patients’ ratings of their physical discomfort, etc.
• Such constructs do not have a commonly agreed
upon metric.
• A survey scale comprises series of questions
designed to capture the different facets of a given
construct.

Introduction
• The questionnaire can influence the response rate
achieved in the survey, the quality of responses
obtained, and consequently the conclusions drawn
from the survey results.
• A well-designed questionnaire is the one ensuring
the collection of worthwhile data about the
phenomenon under study using a set of items that
every respondent will interpret the same way,
respond to accurately, and be willing and motivated
to answer.

Introduction
Definitions:
– A construct is A hypothesized concept or characteristic
that a survey or test is designed to measure.
– Key variables are those which describe the construct,
for example if the study is relative to cancer, key
variables will be relative to cancer.
– Explicative variables are all those variables which
might be correlated to the key variables, for example in
the case of cancer it could be the environment, stress,
food, etc.
– Structure variables are variables which are usually
used to describe the sample of the study such as age,
sex, education, and profession.

Types of survey
questionnaires
1. Exploratory
questionnaires
If the data to be collected is
qualitative for the
purpose of better
understanding or
generation of
hypotheses on a subject.
It may be that no formal
questionnaire is needed.
2. Formal standardized
questionnaires:
If the researcher is looking
to test and quantify
hypotheses and the data
is to be analyzed
statistically.
A formal standardized
questionnaire is needed.

Forms of questionnaires
Both structured and semi-structured forms can be
used in quantitative surveys.
• closed questions
Structured
• mixture of closed and
open questions
Semi-
structured
• open responses, suitable
for qualitative surveys
Unstructured

Questionnaire design
It should be borne in mind:
1. A well-designed questionnaire should meet the
research objectives.
2. It should obtain complete and accurate
information.
3. It should make it easy for respondents to give the
necessary information and for the interviewer to
record the answer, do sound analysis, and
interpretation.
4. It should be brief, to the point, and so arranged that
the respondent(s) remain interested throughout the
interview.

Questionnaire design for
medical education
research

AMEE (Association of Medical
Education in Europe) seven-step
design process:
1.Conduct a literature review,
2.Conduct interviews and/or focus groups,
3.Synthesize the literature review and
interviews/ focus groups,
4.Develop items,
5.Conduct expert validation,
6.Conduct cognitive interviews.
7.Conduct pilot testing.

AMEE seven-step survey
design process
Step 1: Conduct a literature review
To define the construct and identify
existing survey scales that might be
used.
Previously validated scales are
sensitive to target population, local
context, and intended use of the
scale.

design process
Step 2: Conduct interviews and/or
focus groups
When a new scale is used, it is
necessary to learn how the
population of interest conceptualizes
and describes the construct of
interest

design process
Step 3: Synthesize the literature review
and interviews/focus groups
By merging the results of the literature
review and interviews/focus groups
to ensure that the conceptualization
of the construct makes theoretical
sense to scholars in the field and
uses language that the population of
interest understands.

design process
Step 4: Develop items
To write survey items that adequately
represent the construct of interest in
a language that respondents can
easily understand and are written in
accordance with current best
practices in survey design.

Types of questions
• Closed-ended question – A survey question
with a finite number of response categories
from which the respondent can choose.
• Open-ended question – A survey question
that asks respondents to provide an answer
in an open space (e.g., a number, a list or a
longer in-depth answer).

Ideal qualities of a question
1. Yields a truthful, accurate answer.
2. Asks for one answer on one dimension.
3. Accommodates all possible responses.
4. Uses specific, simple language.
5. Has mutually exclusive and response
options.
6. Produces variability in response.
7. Minimizes social desirability.
8. Pretested

Avoiding five common pitfalls in survey design.
Artino et al. 2011. Lippincott Williams and Wilkins/Wolters Kluwer Health
Pitfalls
1- Creating a double-barreled item. e.g.,
How often do you talk to your nurses and
administrative staff when you have a
problem?
2- Creating a negatively worded item.
e.g., The chief resident should not be
responsible for denying admission to
patients.
3- Using statements instead of
questions. e.g., I am confident I can do
well in this course (not at all to completely
true).
4- Using agreement response anchors.
e.g., The high cost of health care is the
most important issue in America today.
(strongly disagree to strongly agree).
5- Using too few or too many response
anchors
Solutions
 Create multiple items for each
questions.
 Wording items positively.
Should the chief resident be responsible
for admitting patients?
 Formulate survey items as
questions.
How confident are you that you can do
well in this course?
 Use construct-specific response
anchors (not at all to extremely
important).
 Use 5-7 response anchors to
achieve stable participant responses.

Avoiding four visual-design pitfalls in survey
development.
Artino & Gehlbach (2012). Lippincott Williams and Wilkins/Wolters Kluwer Health
Pitfalls
1- Labeling only the end points of your
response options. E.g.,
How interesting did you find this clinical reasoning
course?
Not at all Extremely
interesting interesting
2- Labeling response options with both
numbers and verbal labels. E.g.,
How much did you learn in today’s workshop?
-2 -1 0 1 2
almost a little some quite a great
nothing bit a bit amount
3- Unequally spacing your response options.
E.g.,
How much did you learn from your peers in this
course?
4. Placing non-substantive response options together
with substantive response options. E.g.,
How satisfied are you with the quality of the library services?
not at all slightly moderately quite extremely not
Satisfied Satisfied Satisfied Satisfied Satisfied applicable
Solutions
 Verbally label each response
option.
How interesting did you find this clinical
reasoning course?
Not at all slightly moderately quite Extremely
interesting interesting interesting interesting interesting
 Use only verbal labels.
How much did you learn in today’s workshop?
 Maintain equal spacing between
response options.
How much did you learn from your peers in this
course?
 Use additional space to separate non-
substantive response options from the
substantive options.
How satisfied are you with the quality of the library
services?

Different Likert-type response options
Construct being
assessed
Five-point, unipolar
response scales
Seven-point, bipolar
response scales
Confidence • Not at all confident
• Slightly confident
• Moderately confident
• Quite confident
• Extremely confident
• Completely unconfident
• Moderately unconfident
• Slightly unconfident
• Neutral
• Slightly confident
• Moderately confident
• Completely confident
Interest • Not at all interested
• Slightly interested
• Moderately interested
• Quite interested
• Extremely interested
• Very uninterested
• Moderately uninterested
• Slightly uninterested
• Neutral
• Slightly interested
• Moderately interested
• Very interested
Effort • Almost no effort
• A little bit of effort
• Some effort
• Quite a bit of effort
• A great deal of effort

Different Likert-type response options
Construct being
assessed
Five-point, unipolar
response scales
Seven-point, bipolar
response scales
Importance • Not important
• Slightly important
• Moderately
important
• Quite important
• Essential
Satisfaction • Not at all satisfied
• Slightly satisfied
• Moderately satisfied
• Quite satisfied
• Extremely satisfied
• Completely dissatisfied
• Moderately dissatisfied
• Slightly dissatisfied
• Neutral
• Slightly satisfied
• Moderately satisfied
• Completely satisfied
Frequency • Almost never
• Once in a while
• Sometimes
• Often
• Almost always

Putting questions into a meaningful
order and format
• Opening questions should be easy, not embarrassing, and
within knowledge and experience of respondents to
encourage them to continue.
• Questions flow logically, items on one aspect are grouped
together, flow from general to specific, from least sensitive
to more sensitive, from factual and behavioral to attitudinal
and opinion questions.
• Question variety, using open-ended questions here and
there, showing cards/pictures to increase interest of
respondents.
• Closing questions: important questions should be in the
earlier part of the questionnaire and sensitive questions
should be left to the end.

Choose the methods of reaching target
respondents
• Face-to-face interview
– Pros: high response rates, can clarify questions, suitable
for longer questionnaires.
– Cons: high costs, time-consuming, need training of
interviewers, transportation, and respondents may give
socially acceptable answers.

respondents
• Telephone interviews
– Pros: high response rates, fast, suitable for short and not
complex questionnaire.
– Cons: interruptions, unsuitable timing of telephone calls,
require interviewers training, might be difficult to target
specific geographical locations.

respondents
• Mail questionnaires
– Pros: high response rates, easy, low cost, can cover
geographical area and large samples, and avoid
interviewer bias.
– Cons: difficult to control respondents and responses, and
time consuming.

respondents
• Internet questionnaire
– Pros: easy, fast, avoid interviewers bias and distortion,
low cost, and avoid socially influenced responses.
– Cons: poor control over respondents' selection, difficult
to follow up, and need special design.

design process
Step 5: Conduct expert validation
(content validation)
This step involves collecting data from
content experts to establish that individual
survey items are relevant to the construct
of interest and that key items or indicators
have not been omitted.
Both quantitative and qualitative assessment
methods are used to improve the content
validity of a new questionnaire/scale.

design process
Step 6: Conduct cognitive interviews
(cognitive pre-testing/ response
process validity)
An evidence-based qualitative method
specifically designed to investigate
whether a survey question satisfies
its intended purpose.
To ensure that respondents interpret
items in the manner that survey
designer intends.

Cognitive processes when
responding to a survey
Comprehension
Retrieval of
information
judgment
Selection of a
response

Cognitive interview
1. The think-aloud technique requires
respondents to verbalize every
thought that they have while
answering each item.
2. The verbal probing is a more active
form of data collection where the
interviewer administers a series of
probe questions designed to elicit
specific information.

Commonly used verbal probes.
Willis & Artino (2013). J grad Med Edu.

design process
Step 7: Conduct pilot testing
The data obtained from the pilot testing
are used to ascertain the internal
structure of the questionnaire (scale
uni-dimensionality) using factor
analysis, and reliability (internal
consistency) using the Cronbach’s
alpha coefficient.

Reliability
Reliability – The extent to which the scores
produced by a particular measurement
procedure or instrument (e.g., a survey)
are consistent and reproducible.
Reliability is a necessary but insufficient
condition for validity.
If a questionnaire is unreliable, then it can’t
be valid.

Aspects of reliability
• Done to ensure that same results are obtained
when used consecutively for two or more times
• Test-retest method is used
STABILITY
• To ensure all subparts of an instrument measure
the same characteristic (Homogeneity)
• Split-half method
INTERNAL
CONSISTENCY
• Used when two observers study a single
phenomenon simultaneously.
• Inter-rater reliability
EQUIVALENCY

Test-retest reliability
• It is used to assess the consistency of response to
the items in a questionnaire from one time to
another.
• Test administered twice to the same participants at
different times.
• Disadvantages:
– Practice effect
– Too short intervals
– Some traits may change with time.

Test-retest reliability
• The reliability coefficient is the correlation
coefficient (Pearson; r)
– Ranges from -1 to +1
– Correlation coefficient 0.7-1.0 is considered strong, 0.3-
0.69 is moderate.
• The test-retest reliability can be evaluated by using
intra-class correlation coefficient (ICC).
– ICC ranges between 0 and 1.
– The value of ICC is considered appropriate if it is at
least 0.70

Internal consistency reliability
• Here one should judge the reliability of the tool by
estimating how well the items that reflect the same
construct yield similar results i.e., how consistent
the results are for different items for the same
construct.
• Internal consistency reliability is expressed using:
– Split-half reliability
– Coefficient alpha

Split-half Reliability
• Split the contents of the questionnaire into two
equivalent halves; either odd/even number,
first/second half, or randomly selected contents.
• Then, we treat the halves as alternate forms.
• Correlation between the two halves indicates the
reliability of one half.
• Problems with how we split the test.

Spearman-Brown formula
Spearman-Brown formula is used to express reliability
of the entire test.

Cronbach’s alpha
• Most commonly used.
• It is acceptable if value is 80% or above.
• The result says to what extent all items represent the
same construct

Inter-rater reliability
• Assesses how two or more scorers differ in
their evaluation of a test.
• Intraclass Correlation Coefficient (ICC):
used for continuous data and captures both
correlation and agreement among raters
• Kappa: used for categorical data and takes
into account possibility of chance agreement
between examiners

Concept of Validity
The degree to which a questionnaire
measures what it was intended to
measure.
Degree to which the researcher has
measured what he has set out to
measure. (Smith, 1991)
Are we measuring what we think we are
measuring? (Kerlinger, 1973)

Types of Validity
Validity
Face
validity
Content
validity
Criterion
related
Concurrent Predictive
Construct
validity

1- Face validity
• The extent to which a measuring instrument
appears valid on its surface
• Each question must have a logical link with the
objective
E.g.: Questionnaire about domestic violence should have
questions related to that issue.
• Not a validity in technical sense because it does
not refer to what is being measured rather what
it appears to measure.
• It has more to do with rapport and public
relations than with actual validity.

Face validity
Evaluate in terms of:
Face validity can be done
using 2 methods:
1.Interview/prob method: The
investigator discuss with participants
each item, and whether it is
appropriate to elicit accurate
responses.
2.Bilingual method: It is employed
if the tool is translated into a regional
language. A bilingual expert is
employed to assess the face validity.
Readability
Layout
& style
Feasibility
Clarity
of
wording

2- Content validity
• The degree to which the items on the
instrument are representative of the knowledge
being tested or the characteristic being
investigated.
• Expert judgment is the primary method used to
determine whether a test has content validity.
– Coverage of issue should be balanced
– Each aspect should have similar and adequate
representation in questions.
• No statistical test is employed here.

How do experts evaluate
validity
 Average Congruency
Percentage (ACP)
• Average percentage of
relevant questions as
determined by
different experts.
• If the value >90 ……
valid.
 Content Validity Index
(CVI)
• Content Validity
Index for Individual
items (I-CVI).
• Content Validity
Index for the Scale
(S-CVI).

CVI: a panel of 3-10 experts are invited to
review the relevance of each question.

Content Validity Index
I-CVI
• Five or fewer experts: all must
agree (I-CVI = 1.0)
• Six or more: (I-CVI should not be
less than 0.78)
S-CVI
• The proportion of items on an
instrument that achieved relevance
by all the content experts
• S-CVI/UA – Universal agreement
• S-CVI/Ave - Average

3- Criterion validity
• The instrument’s capacity to predict behaviour or
ability in a given area.
• The measuring instrument is called ‘criteria’.
• The responses on the questionnaire being
developed are checked against a gold standard tool
which is direct and independent measure of what
the new questionnaire is designed to measure.
• In the absence of such a gold standard one, can use
proxy measures like clinical examination or direct
questions to respondents.

Criterion validity
1. Concurrent validity: the measurement and the
criterion refer to the same point in time.
– E.g.: visual inspection of a wound for evidence of
infection validated against bacteriological
examination of a specimen taken at the same time.
2. Predictive validity: If the test is used to predict
future performance.
– Academic aptitude test that is validated against
subsequent academic performance.

4- Construct validity
• Most important type of validity.
• It refers to the extent to which the new
questionnaire accurately measures an existing
ideas/hypothesis concerning the concepts/
constructs that are being measured.
• Construct validity can further be subdivided into:
1. Convergent validity: It is a general agreement
between an item and its own scale.
2. Discriminate validity: It is a general disagreement
between an item and other scales.

Construct validity
Factor analysis
• For newly developed tools.
• Factor analysis is a complicated statistical
procedure used to estimate where each item
in the questionnaire is correctly reflecting
the corresponding construct.
• If the tool has high construct validity, then
shows increased correlation with the
corresponding domains.

Construct validity
Factor Analysis
 Various items are gathered
into common factors
 Common factors are
synthesized into fewer factors
and then relation between each
item and factor is measured
 Unrelated items are eliminated

Questionnaire
translation
• A questionnaire translation
process should focus in
achieving the conceptual
equivalence instead of
achieving linguistic
equivalence.
• In view of this, the forward
–backward-forward
translation technique
should be applied

Questionnaire translation
1- Forward translation : two independent
professional bilingual translators are hired
to translate the original English version of
the questionnaire into the relevant language.
2- Backward translation: translate back
into English by another two independent
bilingual translators.
3- The final product of this process is
known as the finalized forward
translation.

Questionnaire translation
• VALIDATION STUDY
• It is highly recommended to conduct a validation
study on the translated questionnaire to examine its
psychometric properties such as the validity and
reliability of the questionnaire.
• The appropriate study design for validation study is
cross-sectional with at least 100 participants.
• These participants should be selected from various
social demographic as well as socio economic
background, so that the sample is more
representative of the population.

Conclusion
• It is necessary that data collection tools are
constructed systematically based on a sound
scientific method since research outcome is directly
dependent on the quality and the completeness of
the data collected.
• Reduced response rates and incomplete responses
to self administered questionnaires make it
mandatory that the tool be developed as simple and
respondent friendly as possible.

References
Artino Jr, A. R., La Rochelle, J. S., Dezee, K. J., & Gehlbach, H. (2014).
Developing questionnaires for educational research: AMEE Guide No. 87.
Medical teacher, 36(6), 463-474.
Trivedi, C. (2020, December 16). Reliability coefficients. ConceptsHacked.
Retrieved from https://conceptshacked.com/reliability-coefficients
Morrison Jo. Assessing questionnaire validity. 2022. Select statistical services
Ltd. https://select-statistics.co.uk/blog/assessing-questionnaire-reliability/
Morrison Jo. Assessing questionnaire validity. 2022. Select statistical services
Ltd. https://select-statistics.co.uk/blog/assessing-questionnaire-validity/

QUESTIONNAIRE DEVELOPMENT new.pptx

More Related Content

What's hot

Similar to QUESTIONNAIRE DEVELOPMENT new.pptx

Recently uploaded

QUESTIONNAIRE DEVELOPMENT new.pptx