Basic concepts of validation

1. Image from www.expertbriefings.com
How to validate a questionnaire
Dr Tan Chai Eng
Introduction to Research
Dr Fam Med
2014

2. Objectives
• What type of validation do I need to do?
• The process of content validation
• The process of linguistic validation
• The process of using exploratory factor
analysis for determining construct validity

3. Types of validation
• Content validation?
• Newly developed
questionnaire / survey
• Linguistic validation?
• Questionnaire in non-local
language
• Construct validation by
exploratory factor analysis?
• Checking whether the tool
measures factors consistent
with theoretical concepts
• Criterion validation?
• Can it correctly predict /
discriminate
conditions?

4. Content Validation
•Identify topic of interest
•Gather a collection of relevant experts
•Brainstorm items / qualitative study to
provide items related to the topic of
interest
Vet the items proposed in detail:
•Are the contents relevant?
•Are the items representative of the
many aspects of the intended aspect
to be measured?

5. Why content validation?
• Before you proceed to
construct validation,
you need to make sure
that you have covered
all aspects that could
potentially be related to
the topic of interest!

6. Process of linguistic validation
Forward
Translation
• Ideally 2 independent translation to target language
• Language experts – conceptual vs semantic equivalence
Reconciliation of
translated version
• Committee to review both translations
• Decide on the most appropriate wording
Back translation
• Reconciled translation sent to another 2 independent
translators

7. Process of linguistic validation
Comparing with
original tool
• Compare the back-translation with original tool
• Check for conceptual equivalence
Expert review
• Psychologist and clinician review
Cognitive
debriefing
• Do they understand the questionnaire?
• Comments on wording.

8. Linguistic validation
• Saya berasa susah
untuk menurunkan
angin
• Saya sedar tentang
kekeringan mulut
• Saya nampaknya tidak
dapat mengalami apa-apa
perasaan yang
positif langsung
DASS 21

9. Linguistic Validation
• Kesal atau marah di atas
benda-benda kecil
• Mulut terasa kering
• Tidak dapat mengalami
perasaan positif
DASS 21
Malay version of DASS 21 by Dr Ramli
Musa

10. Linguistic Validation
• If linguistic validation is not done, respondents
may interpret the items differently and give
inaccurate responses.
• Even English versions need to be adapted and
validated locally to ensure conceptual
equivalence and acceptability.

11. Process of linguistic validation
• Any inappropriately worded items  need to
repeat the process until an acceptable version
is achieved
• Desired outcome – conceptually equivalent
and suitable for local setting

12. Construct validation
• Exploratory factor analysis
• Confirmatory factor analysis
• Rasch model analysis
• Structural equation modelling
• Contrast analysis
Will not be discussed in detail during this lecture

13. Exploratory factor analysis
• Data reduction, can be done using SPSS
• Explores the underlying factor structure
• Determines the correlation between each
item in the questionnaire
• Items that belong to the same factor should
be highly correlated with each other

14. Exploratory factor analysis
Administer tool to a wide
variety of samples
•Heterogeneous sample is desirable.
•Not generalisable.
•Purposive sampling
Test for suitability for principal
components analysis
Determine number of factors
to extract
•Kaiser-Meyer-Olkin > 0.6
•Bartlett’s test of sphericity - significant
•Monte Carlo parallel analysis
•Kaiser’s criterion
•Scree plot

15. Exploratory factor analysis
Run principal components analysis
based on pre-determined number
of factors
Interpret the factor structure
Choice of rotation depends on whether
theoretically the factors are correlated
or not.
High factor loading means item is
correlated with the factors
Name the factors, test internal
consistency of each factor

16. Example

17. Exploratory factor analysis
• May reveal that the constructs measured by
the questionnaire is different from the original
questionnaire
• Confirmatory factor analysis to test how well
the items fit into the proposed structure
• Ideally construct should be similar to original
to allow cross-cultural comparison

18. Exploratory factor analysis
Parker et al
(original MPP –
US)
Chiu et al
(Singapore)
Fujimori et al
(Japan)
Mauri et al
(Italy)
Tan et al
(Malaysia)
3 factors:
•Content
•Facilitation
•Support
2 factors:
•Content and
facilitation
•Support
5 factors:
•Emotional
support
•Medical
information
•Clear
explanation
•Encouraging
question asking
•Setting
3 factors:
•Information
•Support
•Care
3 factors:
•Content and
facilitation
•Emotional
support
•Structural and
informational
support

19. Criterion validation
• Predictive validity - the scores from the new measure
to predict performance on a criterion measure
administered at a later time.
Mislevy, J (2010). doi:
http://dx.doi.org/10.4135/9781412961288.n67
– MoCA vs neuropsychological testing (gold
standard) in frontotemporal dementia
Freitas et al, 2012

20. Criterion validation
• Concurrent validity -the extent to which scores on a
new measure are related to scores from a criterion
measure administered at the same time.
– QoL in obesity vs SF-36
Mislevy, J (2010). doi:
http://dx.doi.org/10.4135/
9781412961288.n67
Moorehead et al, 2003

21. Good luck…
Image from b2bprblog.marxcommunications.com