Postgraduate psychiatry critical appraisal (Journal Club) Master of Medicine (Psychiatry) Universiti Sains Malaysia 26/12/2017

1. Presenter : Afiq Fikri bin Azmi
Supervisor : Professor Madya Dr Zahiruddin Othman

2.  Validation
 Reliability
 EFA/CFA in brief
 Introduction
 Objective / Research gap
 Methodology
 Results
 Discussion

3.  Validity - extent to which the instrument measures what it purports to measure.
 Classical models divided the concept into various "validities” (such as content,
criterion and construct validity)
 currently dominant view is that validity is a single unitary construct

4.  Content Validity: degree to which all aspects of the relevant subjects are assessed
Example : a depression scale may lack content validity if it only assesses
the affective dimension of depression but fails to take into account
the behavioral dimension
 FaceValidity : whether the intended characteristics appears to be measured
(subjective, superficial appraisal)
 Criterion Validity: ability of the test to distinguish between subjects already
known to differ on the basis of external, validated test
1. Concurrent validity: comparison of the test result with those of another of
prevalidated measure
2. Predictive validity: ability of test to predict the outcome as determined later by
another established scale

5.  Cross validity: extent to which the validity of a measure is retained when applied
to a new set of subjects
 Construct validity
1. Convergent validity: degree of association between measures that are
expected to be closely correlated
Example : If a measure of general happiness had convergent validity, then
constructs similar to happiness (satisfaction, contentment, cheerfulness, etc.)
should relate closely to the measure of general happiness
2. Divergent validity: degree to which a measure discriminates between that
which is being assessed from unrelated measures
constructs that are not supposed to be related to general happiness (sadness,
depression, despair, etc.) should not relate to the measure of general happiness

6.  Concurrent validity differs from convergent validity in that it focuses on the
power of the focal test to predict outcomes on another test or some outcome
variable.
 Convergent validity refers to the observation of strong correlations between two
tests that are assumed to measure the same construct.
 It is the interpretation of the focal test as a predictor that differentiates this type of
evidence from convergent validity, though both methods rely on simple
correlations in the statistical analysis.

7.  Some guidelines:
– Min.: N > 5 cases per variable
e.g., 12 variables, should have > 60 cases (1:5)
– Ideal: N > 20 cases per variable
e.g., 12 variables, ideally have > 240 cases
(1:20)
– Total N > 240 preferable
 Comrey and Lee's (1992) guidelines:
– 50 = very poor,
– 100 = poor,
– 200 = fair,
– 300 = good,
– 500 = very good
– 1000+ = excellent

8.  Reliability is defined as the extent to which a questionnaire, test, observation or any
measurement procedure produces the same results on repeated trials.
 Inter-rater reliability assesses the degree of agreement between two or more
raters in their appraisals.
 Test-retest reliability assesses the degree to which test scores are consistent from
one test administration to the next. Measurements are gathered from a single rater
who uses the same methods or instruments and the same testing conditions.[4] This
includes intra-rater reliability.

9.  internal consistency is typically a measure based on the correlations between
different items on the same test (or the same subscale on a larger test).
 It measures whether several items that propose to measure the same general
construct produce similar scores

10.  Sampling adequacy (Keiser-Meyer-Olkin value of more than 0.6) and factorability
of data (Bartlett’s test of sphericity) needs to be assessed before proceeding for
factor analysis
 exploratory factor analysis (EFA) is a statistical method used to uncover the
underlying structure of a relatively large set of variables. EFA is a technique
within factor analysis whose overarching goal is to identify the underlying
relationships between measured variables
 Confirmatory factor analysis (CFA) - used to test whether measures of
a construct are consistent with a researcher's understanding of the nature of that
construct (or factor). As such, the objective of confirmatory factor analysis is to test
whether the data fit a hypothesized measurement model

12.  PTSD can be found all over the world and its lifetime prevalence ranges from
0.35% in China and 6.8% in the United States of America
 Research has concentrated on wars and conflicts being the major traumatic event,
but recently, other events such as physical and sexual violence, disasters and motor
vehicle accidents (MVA) are receiving more attention.
 MVAs are a daily occurrence in the country, and it is estimated that there will be
more than 8000 fatalities due to MVAs in the year 2015
 approximately 10-15% of all MVA victims according to Australian Centre for
Posttraumatic Mental Health will develop PTSD

13.  There are numerous validated screening tools available for PTSD but to the best of
our knowledge, none has been validated to be use in the Malaysian population.
 The PTSD Checklist for Civilians (PCL-C) is a validated instrument to screen for
PTSD in the general public
 This study aims to translate the PCL-C into Malay language and validate the
translated version for use in the Malaysian MVA victim population

15.  Cross sectional study design
 Duration of study : January to May 2014
 Location : Emergency Department, UKMMC
 subjects were asked to complete the Post- Traumatic Stress Disorder Checklist for
Civilians (PCL-C) at least one month following their motor vehicle accident (MVA).
 Approval obtained from UKMMC ethics’ committee

16.  Sampling method : convenience sampling
 subjects were recruited from those attending the ED of UKMMC for MVA-related
injuries
 Everyone over the age of 18 was included
 Patients who have major language problems, significant head injury during the
accident or those under the age of 18 were excluded
 Calculated sample size = 51

17.  The PCL-C is a self-rated questionnaire, used as a screening tool for PTSD.
 It is chosen for its good validity, brevity and the fact that it is self-rated, which
appeals to the busy ED setting.
 Permission obtained from the main author (Frank Weathers) and current owner
(The National Centre for PTSD) before commencement of the study.

20.  First translated into Malay, and then reviewed by an independent content expert whose
mother tongue is Malay
 amended according to the comments made by the reviewer, and this second version is
then back-translated into English by a person who is both content and language expert.
 The process above was repeated by a different set of experts.
 All this was done to ensure that the translated version is terminologically and
grammatically sound while preserving the content and meaning of the original.
 The language expert employed in the study is a psychiatrist and qualifies as a language
expert due to prior works, including the development a Malay medical dictionary.

21.  By reviewing, revising and refining the translations a final version of the Malay
Post-Traumatic Stress Disorder Checklist Civilian version (MPCL-C) was produced.
 Pre testing exercise performed on a group of 15 first-line responders in the
emergency department consisting of healthcare assistants, nurses and paramedics
who frequently deal with the victims of MVA.
 It was from their input that the layout was altered for the finalized version.
 We found that the MPCL-C is easily understood in terms of language and message

22.  Face validity was determined by piloting the MPCL-C, on seven subjects who
presented to the ED following MVA and fulfilling the inclusion and exclusion criteria to
the study. Feedback - easily understood and at face value it fitted its purpose.
 content validity - translated version of the questionnaire presented to a panel of
Content experts consisting of psychiatrists and clinical psychologists reviewed the
translated version and were satisfied that the contents were preserved and terms used
were correct
 Construct validity – EFA to ascertain the construct validity of the questionnaire. A
factor loading of 0.3 is considered acceptable (Nunally and Bernstein)
 Reliability - two methods of estimation were used,
1. internal consistency - Chronbach’s Alpha (coefficient α) values. A modest value of at
least .70 is deemed sufficient
2. Test-pretest reliability - a randomly selected number of subjects were given the
MPCL-C again after 2 weeks

24.  construct validity was ascertained through exploratory factor analysis (EFA).
 It was found that it is still best to group PTSD symptoms in three domains, as is in the
original PCL-C
 15 out of 17 factors were grouped accordingly.
 Item 4 (“feeling upset when reminded”) had a higher factor loading in “avoidance”
domain rather than “re- experiencing” domain.
 Items 10 (“feeling distant”) and 12 (“feeling as if future will be cut short”) both had low
factor loadings in its rightful places, which would be in the “avoidance” domain of PTSD
symptoms.

26. • test-retest reliability after 2 weeks is very high. It had a 0.98 Pearson
correlation value (p=0.02), which is significant at p<0.05 level

27.  main strength of this study – translation process which was methodical, followed the
guidelines with involvement of many experts at different stages and level
 The sample size satisfied the calculation and representative of the population
being studied.
 Good validity and reliability suggest that the Malay version of the PCL-C is
comparable to versions in other languages.
 Findings of this study support the use of the PCL-C for Malay speaking patients

28.  The construct of the scale remains mostly intact and cross-culturally sensitive.
 Some terms such as “upset," “feeling distant” and “foreshortened future” have no
counterpart in Malay and indeed as a concept, may be unfamiliar to Malaysians.
 these inaccurate translations were reflected through the low factor loading in these
three variables.
 Ease of administration - ease of administration, brief content and user-friendly format
and language, makes it reliable as a screening tool for PTSD among MVA victims even
in a hectic clinical setting.
 Limitation - does not employ other forms of validity processes. For example, the
concurrent validity due to absence of an alternative measure.
 Ideally, the gold standard tool, in this case the Clinician Administered PTSD Scale
(CAPS), but it is lengthy, requires a trained professional and is not available in Malay

29. Is the study's research objective relevant?
 Yes.There is a lack of instruments in Malay for PTSD.Thus this research aims to
validate PCL-C for the purpose of use in the local setting.

30.  Was the sample size adequate enough?
 Some guidelines:
– Min.: N > 5 cases per variable
e.g., 12 variables, should have > 60
cases (1:5)
– Ideal: N > 20 cases per variable
e.g., 12 variables, ideally have > 240
cases
(1:20)
– Total N > 240 preferable
Comrey and Lee's (1992) guidelines:
– 50 = very poor,
– 100 = poor,
– 200 = fair,
– 300 = good,
– 500 = very good
– 1000+ = excellent
Based on these guidelines, an ideal sample size would be 340 (20 x 17)

31.  Were the results clearly presented?
The results for EFA were not clearly presented. Usually each factor in the model
should include its Eigenvalue (usually set to 1 and above) and also its Percent of
variance.

32.  Was the data suitable for exploratory factor analysis?
We should initially measure how suitable our data is for factory analysis.
Sampling adequacy using Kaiser-Meyer-Olkin (KMO) Test
Factorability of the correlation matrix using Barlett’s test of sphericity

33.  How would you improve this study?
As a screening tool, the ability to predict a diagnosis of PTSD should be compared
with a gold standard – Clinician Administered PTSD Scale (CAPS) the assess for
concurrent validity
The correlation between these two tools should be measured