How to assess questionnaires

QUESTIONNAIRE EVALUATION
Dr. Ahmed-Refat AG Refat
December 12, 2014
1

Contents
December 12, 2014
Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat
2
Main Concepts & terms ( Pre-test)
Sequence of questionnaire development & testing
Questionnaire Testing Methods
Errors Detection
Questionnaire Evaluation Criteria
Criteria for a Good Questionnaire
Using Questionnaire Appraisal System (QAS)-99

December 12, 2014
3
Main Concepts
& Terms
( Pre-test :10 terms)
Define the underlined terms in the following slide…….

It has to be established that the Qes. is valid and reproducible in the context in which it is going to be employed.
The validity of the questionnaire should have been investigated;
The questionnaire are a correct and comprehensive reflection of the concept the questionnaire is intended to measure (content validity).
the dimensionality (factor structure) and internal consistency of the questionnaire should have been investigated;
the test-retest reliability of the questionnaire should have been investigated;
Responsiveness will need to be determined for questionnaires being to be used to measure changes within individuals .
Information regarding the interpretation of questionnaire scores should be available
Floor effects or ceiling effects less than 15%
Accepted score 7+
December 12, 2014
4

Recommended Source-1
December 12, 2014
5

December 12, 2014
6

December 12, 2014
7
Questionnaire Appraisal System
(QAS-99)
QAS-99 is based on a system that developed for Behavioral Surveillance Branch of the Centers for Disease Control and Prevention for use in evaluating questions for the Behavioral Risk Factor Surveillance System (BRFSS).
http://www.cdc.gov/nccdphp/brfss

Questionnaire development approach
December 12, 2014
8
I. Determine Analytic Objectives
• What types of data will answer the research question?
II. Develop general concepts to be covered
• List areas to be covered by questions
III. Translate concepts into questions
IV. “Appraise” questions for common pitfalls
V. Evaluate questions empirically

December 12, 2014
9

Sequence of questionnaire development & testing
December 12, 2014
10

1- Conceptualization
December 12, 2014
11
Conceptual frame of the Questionnaire is more important for new surveys, whereas in existing surveys concepts might be already well established.
An integral part in every change in a questionnaire.
The main output are :
An entities/relationships scheme,
An area tree about the structure of the targeted questionnaire,
List of target variables

2- Questionnaire design.
December 12, 2014
12
First draft of a questionnaire
 Appropriate wording,
 Order of questions and
 Definition of answering categories

3- Testing.
December 12, 2014
13
The questionnaire needs to be tested regarding:
Wording of questions/answers, order and structure of the questionnaire;
Problems related to translation, and cultural background ;and
Data collection mode and the involvement of an interviewer ……….V&R

Testing the Questionnaire-Cont’
December 12, 2014
14
Pre-field and field methods.
It is recommended to involve respondents.
A combination of different methods is advisable

4- Revision.
December 12, 2014
15
After testing, making revisions to the questionnaire
Afterwards, a new round of testing is often essential.
This process may be repeated through two, three or even more phases of testing.
The aim is to check if the changes are really resulting in a higher validity and reliability of data in relation to the specific objectives of the survey.

5- Data collection
December 12, 2014
16
With the implementation of the survey (either by a pilot study or as the real survey) the process of development and revision is terminated,
but the process of observation should be continued via
monitoring the interviewers

5-Data collection-cont’
December 12, 2014
17
Monitoring can be regarded as :
Continuous tool of evaluation .
The monitoring of fieldwork can be essential for the further phase of post survey evaluation when conducting ongoing surveys or having implemented a new survey on full scale.

December 12, 2014
18
?
Questionnaire Testing Methods

December 12, 2014
19
Aims
Phase of testing
Methods
1- Focus groups (FGs)
(respondent group
discussion)

December 12, 2014
20
Aims
Phase of testing
Methods
Early stage of
questionnaire design
(respondent group
discussion)

December 12, 2014
21
Aims
Phase of testing
Methods
To gain a reflection of
the target population
perspective
• To check terms
Early stage of
questionnaire design
(respondent group
discussion)

Questionnaire Testing Methods -Cont
December 12, 2014
22
Aims
Phase of testing
Methods
2- Informal test
(evaluation by
colleagues)

December 12, 2014
23
Aims
Phase of testing
Methods
Possible in each
phase, preferably at
the beginning
2- Informal test
(evaluation by
colleagues)

December 12, 2014
24
Aims
Phase of testing
Methods
• To detect all kinds of
mistakes: wording,
layout, skips etc.
Possible in each
phase, preferably at
the beginning
2- Informal test
(evaluation by
colleagues)

December 12, 2014
25
Aims
Phase of testing
Methods
3- Expert group
Group
discussion among
design and matter
experts, sometimes
with users
******************** ********************

December 12, 2014
26
Aims
Phase of testing
Methods
Initial phase of
questionnaire
development
3- Expert group
Group
discussion among
design and matter
experts, sometimes
with users
******************** ********************

December 12, 2014
27
Aims
Phase of testing
Methods
• To check concepts,
definitions,
vocabulary against
survey’s objectives
• To discuss data
processing
requirements
Initial phase of
questionnaire
development
3- Expert group
Group
discussion among
design and matter
experts, sometimes
with users
******************** ********************

December 12, 2014
28
Focus Specifically on
Methods
• Terms and wording of questions;
• Structure of questions;
• The response alternatives;
• Order of questions;
• Navigational rules
• Instructions to interviewers .
• Confusing layout;
• Typographical errors
3- Expert group
Group
discussion among
design and matter
experts, sometimes
with users
******************** ********************

December 12, 2014
29
Aims
Phase of testing
Methods
4- In-depth or
qualitative
interviews
(interviews with
respondents)

December 12, 2014
30
Aims
Phase of testing
Methods
Early stage of
development and
testing the
questionnaire
4- In-depth or
qualitative
interviews
(interviews with
respondents)

December 12, 2014
31
Aims
Phase of testing
Methods
• To evaluate
respondents’
viewpoint and
understanding of the
questionnaire
• Explorative nature
Early stage of
development and
testing the
questionnaire
4- In-depth or
qualitative
interviews
(interviews with
respondents)

December 12, 2014
32
Aims
Phase of testing
Methods
5- Cognitive
interviews :(one- to-one
in-depth,
structured interviews
with specially trained
interviewers and
researchers)

December 12, 2014
33
Aims
Phase of testing
Methods
Middle of the
development
process, once a draft
questionnaire has
been developed
5- Cognitive
in-depth,
interviewers and
researchers)

December 12, 2014
34
Aims
Phase of testing
Methods
• To gain qualitative
information on how a
questionnaire is
understood and
answered
Middle of the
development
process, once a draft
questionnaire has
been developed
5- Cognitive
in-depth,
interviewers and
researchers)

December 12, 2014
35
Aims
Phase of testing
Methods
6- Observational
Interviews
(observation of
respondents while
completing a
questionnaire)

December 12, 2014
36
Aims
Phase of testing
Methods
Middle of
development, when a
tested questionnaire
exists
6- Observational
interviews
(observation of
respondents while
completing a
questionnaire)

December 12, 2014
37
Aims
Phase of testing
Methods
• To check self completed
questionnaire by
observing potential
respondents in the lab
Middle of
development, when a
tested questionnaire
exists
6- Observational
Interviews
(observation of
respondents while
completing a
questionnaire)

Questionnaire Testing Methods -Cont’
December 12, 2014
38
Aims
Phase of testing
Methods
7- Behaviour coding
(coding behaviour
and interaction of
interviewers and
respondents )

December 12, 2014
39
Aims
Phase of testing
Methods
After a set of pre-field
methods have been
conducted
7- Behaviour coding
(coding behaviour
and interaction of
interviewers and
respondents )

December 12, 2014
40
Aims
Phase of testing
Methods
To evaluate the
question- answering
process by
standardized methods
and coding scheme
After a set of pre- field
methods have been
conducted
7- Behaviour coding
(coding behaviour
and interaction of
interviewers and
respondents )

Error Detection
December 12, 2014
41
Completely identify the errors is virtually impossible;
therefore, the goal should be to design a testing procedure capable to catch as many errors as possible

Errors Detection
December 12, 2014
42
Q-by –Q
Analysis of single questions .
Testing by task
Assignment of specific tasks to different testers, so that each of them focuses on given issues.
Scenario testing
Hypothesizing some real situations, entering them in the questionnaire and checking the performance and the results.

December 12, 2014
43
Questionnaire Evaluation Criteria

The best-known and most comprehensive criteria are those from the Scientific Advisory Committee (SAC) of the Medical Outcomes Trust*….. The SAC defined
eight attributes
that deserve considerations in evaluation.
December 12, 2014
44

(SAC)
Eight Attributes
Of the good Questionnaire
?
December 12, 2014
45

(1) Conceptual and measurement model,
 (2) Validity, (……,……,……)
(3) Reliability,
(4) Responsiveness,
 (5) Interpretability,
(6) Respondent and administrative burden,
 (7) Alternative forms, and
 (8) Cultural and language adaptations (translations).
December 12, 2014
46

Specific Criteria
Within each attributes, specific criteria were defined :
(1) content validity,
 (2) construct validity,
(3) criterion validity,
(4) internal consistency
(5) reproducibility,
(6) responsiveness,
 (7) floor and ceiling effects, and
(8) interpretability.
December 12, 2014
47

?
December 12, 2014
48
Validity
&
Reliability

Validity
December 12, 2014
49
Validity addresses the amount of systematic or "built-in" error contained in the measure.

Reliability
Reliability refers to random error in measurement.
Reliability indicates the accuracy or precision of the measuring instrument .
The pilot test seeks to answer the question,
Does the questionnaire consistently measure whatever it measures?
December 12, 2014
50

Reliability
Several methods ……………all involve administering the instrument to a small sample during a pilot test.
 A common procedure………is the test/re-test
The instrument is given to the same group of individuals twice (about one week apart) and the two sets of scores are correlated, resulting in a coefficient of stability.
A correlation above 0.7 would indicate acceptable reliability.
Other techniques …….Cronbach's Alpha.
December 12, 2014
51

Specific Criteria
Within each attributes, specific criteria were defined :
(1) content validity,
 (2) construct validity,
(3) criterion validity,
(4) internal consistency
(5) reproducibility,
(6) responsiveness,
 (7) floor and ceiling effects, and
(8) interpretability.
December 12, 2014
52

1-Content Validity
December 12, 2014
53
Content validity examines
the extent to which the concepts of interest are comprehensively represented by the items in the questionnaire

1- Content Validity-cont;
December 12, 2014
54
To evaluate content validity the following aspects should be considered :
1.Measurement aim of the questionnaire
2.Target population
3.Concepts that intended to measure.
4.Item selection and item reduction
5.Interpretability of the items.
Completing the questionnaire should not require reading skills beyond that of a 12-year-old to avoid missing values and unreliable answers

2- Internal Consistency
December 12, 2014
55
Internal consistency is a measure of the extent to which items in a questionnaire (sub)scale are correlated (homogeneous), thus measuring the same concept.
Internal consistency is an important measurement property for questionnaires that intend to measure a single underlying concept (construct) by using multiple items.

3- Construct Validity
December 12, 2014
56
Construct Validity is used to ensure that the measure is actually measure what it is intended to measure (i.e. the construct), and not other variables. Using a panel of “experts” familiar with the construct is a way in which this type of validity can be assessed.
The experts can examine the items and decide what that specific item is intended to measure.

4- Reproducibility
December 12, 2014
57
Reproducibility concerns the degree to which repeated measurements in stable persons (test-retest) provide similar answers.

5- Criterion Validity
December 12, 2014
58
Criterion validity refers to the extent to which scores on a particular instrument relate to a gold standard

6- Responsiveness
December 12, 2014
59
The ability of a questionnaire to detect important changes over time, even if these changes are small
Responsiveness is measure of longitudinal validity.

6- Responsiveness-cont,
A longitudinal study with pre- and post- testing is required for determining responsiveness.
 It is important in this process that the design and potential intervention being used represent the situation in which the questionnaire will be used in the future.
December 12, 2014
60

7- Floor or Ceiling Effects
December 12, 2014
61
The number of respondents who achieved the lowest (floor/ground/basement) or highest (ceiling) possible score.
Calculation : The % of subjects who achieved the maximum score (ceiling) or the minimum score (floor).
These effects are considered when 15% of respondents reach the ceiling or floor scores.
Their effects :implications on the questionnaire’s reproducibility and responsiveness

7- Floor or Ceiling effects
December 12, 2014
62
The number of respondents who achieved the lowest or highest possible score.
If floor or ceiling effects are present, it is likely that extreme items are missing in the lower or upper end of the scale, indicating limited content validity. As a consequence, subjects with the lowest or highest possible score cannot be distinguished from each other, thus reliability is reduced.
Furthermore, the responsiveness is limited because changes cannot be measured in these patients.

A floor effect
December 12, 2014
63
A Floor Effect is when most of your subjects score near the bottom. There is very little variance because the floor of your test is too high.
The question is too hard for the group you are testing.

A ceiling effect
December 12, 2014
64
A Ceiling Effect :Most of subjects score near the top. There is very little variance because the ceiling of your test is too low.
The question is too easy for the group you are testing.

8- Interpretability
December 12, 2014
65
Interpretability is defined as
the degree to which one can assign qualitative meaning to quantitative scores.

Systematic Evaluation of Quest.
December 12, 2014
66
How do we find questionnaire problems????????
By conducting a s tructured expert review
( Technical Review )
Example: QAS-99

December 12, 2014
67
(QAS-99)
QAS-99 is based on a system that developed for Behavioral Surveillance Branch of the Centers for Disease Control and Prevention for use in evaluating questions for the Behavioral Risk Factor Surveillance System (BRFSS).

December 12, 2014
68
For structured expert reviews a set of criteria (coding schemes ) by which each question is to be examined are implemented.
The coding scheme needs to be filled in for each question and consequently the draft questionnaire needs to be completed by the standardized coding categories .

QAS-99 ( Items )
December 12, 2014
69
1. PROBLEMS WITH READING:.
2. PROBLEMS WITH INSTRUCTIONS
3. PROBLEMS WITH ITEM CLARITY:
4. PROBLEMS WITH ASSUMPTIONS
5. PROBLEMS WITH KNOWLEDGE/MEMORY:
6. PROBLEMS WITH SENSITIVITY/BIAS:
7. PROBLEMS WITH RESPONSE CATEGORIES

1. PROBLEMS WITH READING:
December 12, 2014
70
Determine if it is difficult for the interviewers to read the question uniformly to all respondents.
1a – What to read: Interviewers may have difficulty determining what parts of the question are to be read.
1b – Missing information: Information the interviewer needs to administer the question is not contained in the question.
1c – How to read: Question is not fully scripted and therefore difficult to read.

2. PROBLEMS WITH INSTRUCTIONS:
December 12, 2014
71
Look for problems with any introductions, instructions, or explanations from the respondent’s point of view.
2a – Conflicting or inaccurate instructions, introductions, or explanations.
2b – Complicated instructions, introductions, or explanations

3. PROBLEMS WITH ITEM CLARITY:
December 12, 2014
72
Identify problems related to communicating the intent or meaning of the question to the respondent
3a – Wording: The question is lengthy, awkward, ungrammatical, or contains complicated syntax.
3b – Technical terms are undefined, unclear or complex.
3c – Vague: The question is vague because there are multiple ways in which to interpret it or to determine what is to be included and excluded.
3d – Reference periods are missing, not well specified, or are in conflict.

4. PROBLEMS WITH ASSUMPTIONS:
December 12, 2014
73
Determine there are problems with assumption if s made or the underlying logic.
4a – Inappropriate assumptions are made about the respondent or his/her living situation.
4b – Assumes constant behaviour: The question inappropriately assumes a constant pattern of behaviour or experience for situations that in fact vary.
4c – Double-barrelled question that contains multiple implicit questions.

5. PROBLEMS WITH KNOWLEDGE/MEMORY:
December 12, 2014
74
Check whether respondents are likely to not know or have trouble remembering information.
5a – Knowledge: The respondent is unlikely to know the answer.
5b – An attitude that is asked about may not exist.
5c – Recall failure.
5d – Computation or calculation problem

6. PROBLEMS WITH SENSITIVITY/BIAS
December 12, 2014
75
: Assess questions for sensitive nature or wording, and for bias.
6a – Sensitive content: The question is on a topic that people will generally be uncomfortable talking about.
6b – A socially acceptable response is implied.

7. PROBLEMS WITH RESPONSE CATEGORIES:
December 12, 2014
76
Asses the adequacy of the range of responses to be recorded.
7a – Open-ended question that is inappropriate or difficult.
7b – Mismatch between question and answer categories.
7c – Technical terms are undefined, unclear, or complex.
7d – Vague response categories.
7e – Overlapping response categories.
7f – Missing response categories.
7g –Illogical order of response categories

QAS-99 ( Reference)
December 12, 2014
77
 Question Appraisal System QAS-99 . By:
Gordon B.Willis and Judith T. Lessler . Research Triangle Institute Suite 420 6110 Executive Blvd. Rockville, MD 20852 August, 1999
Biemer, P.P. and Lyberg, L.E. (2003). Introduction to Survey Quality. Hoboken, New Jersey: John Wiley&Sons

Cited References
December 12, 2014
78
http://appliedresearch.cancer.gov/areas/cognitive/qas99.pdf
http://www.joe.org/joe/2007february/tt2.php
https://readability-score.com/
http://www.joe.org/joe/1990summer/tt2.php
http://www.emgo.nl/kc/preparation/research%20design/ 8%20Questionnaires%20selecting,%20translating%20and%20validating.htm
Scientific Advisory Committee of the Medical Outcomes Trust. Assessing health status and quality-of-life instruments: attributes and review criteria. Qual Life Res 2002;11:193e205.
http://ec.europa.eu/eurostat/ramon/statmanuals/files/Handbook_of_Practices_for_Quest.pdf
http://appliedresearch.cancer.gov/areas/cognitive/qas99.pdf

December 12, 2014
79
Thank You

How to assess questionnaires

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to How to assess questionnaires

Similar to How to assess questionnaires (20)

More from Ahmed-Refat Refat

More from Ahmed-Refat Refat (20)

Recently uploaded

Recently uploaded (20)

How to assess questionnaires