Validating An Instrument: CTT (EFA) versus IRT (Rasch Partial Credit)

©drtamil@gmail.com 2019
Malaysian Rasch Association
Validating an instrument:
Classical Test Theory (EFA) versus
Item Response Theory
(Rasch Partial Credit).
Azmi Mohd Tamil

Sample of
an
instrument
to measure
QOL

Basic Concept
• In psychometry, an instrument (i.e.
questionnaires) with items are created to
measure a latent trait that is not usually
measurable in the normal physical way.
• For example, what if we want to measure
physical fitness? So we come up with items
that is related to measuring physical fitness.
• The following are sample measures of fitness
in ascending difficulty;

Sample Measures of Fitness
Physical Activity 1=Limit
ed a lot
2=A bit
limited
3=Not
limited
Bathing or dressing yourself
Bending, kneeling or stooping
Lifting or carrying groceries
Walking one block
Climb one flight of stairs
Walking several blocks
Climb several flight of stairs
Moderate activities such as moving a table
Walking a mile (1.6 km) or more
Vigorous activities such as running or strenuous sports.
i
n
c
r
e
a
s
i
n
g
d
i
f
f
• Minimum score 10 – very unfit, maximum score 30 – very fit.
• These items combined to measure a single factor/latent trait -> Fitness.

Sample of a Fitness Score
Physical Activity 1=Limit 2=A bit 3=Not Score
Bathing or dressing yourself  3
Bending, kneeling or stooping  2
Lifting or carrying groceries  2
Walking one block  3
Climb one flight of stairs  2
Walking several blocks  2
Climb several flight of stairs  1
Moderate activities such as moving a table  2
Walking a mile (1.6 km) or more  1
Vigorous activities such as strenuous sports.  1
Total Score 19
i
n
c
r
e
a
s
i
n
g
d
i
f
f
• Cut-off=(Maximum-Minimum)/2+Minimum=(30-10)/2+10=20.
• Score of 19 is below the cut-off point. Is the respondent unfit?
• The reliability of a `composite score’ can be checked by Cronbach α.

Issues in Scoring
• The responses are ordinal in nature. Can we
sum up the scores to come up with one global
score?
• Is the global scores ordinal or interval in
nature?
• Each item measures a different level of fitness.
Should we give each item equal or different
weightage/loading?
• Will the instrument be reliable and valid?

Why Validate?
• There is an increasing demand for
psychometric scales in the health field for
measuring things that cannot be measured in
the usual way.
• Therefore scales in the health field need to
have excellent psychometric properties,
including reliability and validity.

Why Use Rasch?
• The usual statistical methods to develop,
evaluate & refine scales was Factor Analysis.
• But Rasch Analysis (part of Item Response
Theory) is increasing in its popularity.
• Rasch converts the ordinal scores into interval.
• We already covered Rasch Model for
dichotomous data earlier.
• Rasch has many kinds of models.

Family of Rasch models:
• Dichotomous Rasch Model
• Partial Credit Model
• Rating Scale Model
• Binomial and Poisson Models
• Many-Facet Rasch Model
• Multidimensional Rasch Model

Rasch Family Models
Model Description Example
Dichotomous Probability of person n with ability Bn succeeding
on item i which has difficulty level Di.
Yes/No (or success/failure) answers to
questions.
Binomial
trials
Several independent attempts are
made at an item and the number of
successes is counted.
In shooting contests, the shooter is allowed to take
several, say m, attempts at a target and the total number
of hits, say x, within m attempts is counted. The
probability of a shooter with ability Bn aiming at a target
with difficulty level Di and getting x hits in m attempts
Poisson
Count
If the number of trials in the binomial model is
infinite and the probability of success is small
Counting the number of customers buying a certain
product at the supermarket in some given time period.
Rating Scale Outcomes in response categories in Likert
questionnaires may include ordered ratings such
as “Strongly Disagree/Disagree/ Agree/ Strongly
Agree”
Questionnaire type survey
Partial
Credit
Similar to the rating scales model except that
now each item has its own threshold parameters
Writing assesment with different scoring
Ranks
Model
Respondents are asked to rank order a
group of objects instead of giving a rating
to each object
Judge ordering pianists from the strongest to the
weakest, or a worker sequencing jobs from most
to least urgent
Many facets Outcomes from interaction between elements,
e.g., a student, an item and a rater
Get desireable scores from several judges observing
performances/test etc.

Aim of the current exercise
• To assess the psychometric properties of the
Mental & Physical Scale using EFA & Rasch
Partial Credit analysis.
• To ensure continuity, I shall use the same
dataset for both EFA and Rasch.
• 10 items on Physical Function.
• 5 items on Mental Function.

15 Variables, 117 Complete Dataset.

Physical Scores

Scoring for Physical
“Limited A Lot” “Limited A Little” “Not Limited At All”
1 2 3
Higher score indicate higher physical capability.
• Respondents scored 1 to 3 only.

Physical Function Questions
1
Vigorous activities, such as running, lifting heavy objects,
participating in strenuous sports
2
Moderate activities, such as moving a table, pushing a
vacuum cleaner, bowling, or playing golf
3 Lifting or carrying groceries
4 Climbing several flights of stairs
5 Climbing one flight of stairs
6 Bending, kneeling, or stooping
7 Walking more than a mile
8 Walking several blocks
9 Walking one block
10 Bathing or dressing yourself

Physical Scores
Score 1 2 3 4 5 6 7 8 9 10
1 37 22 9 13 10 12 26 20 21 11
2 61 49 24 52 15 38 52 34 44 14
3 24 47 86 55 92 68 39 63 52 93
Total 122 118 119 120 117 118 117 117 117 118
Higher Score, Better Physical Function
• Minimum score 10 – very unfit, maximum score 30 – very fit.
• These items combined to measure a single factor/latent trait -> Fitness.
Response
Limited a lot
Limited a bit
Not limited

Mental Scores

Scoring for Mental
• Negative 1, 4, 5, 8. Positive 2, 3, 6, 7, 9.
• The 5 questions used in this exercise are the 5
positive ones. The scores are not reversed.
• So higher score indicate better mental health.
All the time Most time A good bit Some time A little time None at all
1 2 3 4 5 6
1 Did you feel full of pep?
2 Have you been a very nervous person?
3 Have you felt so down in the dumps that nothing could cheer you up?
4 Have you felt calm and peaceful?
5 Did you have a lot of energy?
6 Have you felt downhearted and blue?
7 Did you feel worn out?
8 Have you been a happy person?
9 Did you feel tired?
+
+
+

Mental Function
Score 1 2 3 4 5
1 2 2 2 1 6
2 6 6 8 2 7
3 17 6 31 13 30
4 48 35 9 48 15
5 27 27 36 27 31
6 12 37 26 19 22
Total 112 113 112 110 111
Items Nervous In Dump Blue Worn Out Tired
Response
All the time
Most time
A good bit
Some time
A little time
None at all
• Minimum score 5 – poor mental health, maximum score 30 – good mental health.
• These items combined to measure a single factor/latent trait -> Mental Health.

VALIDATION OF INSTRUMENT
Reliability

Reliability Analysis
• Kuder–Richardson Formula 20 (KR-20), first
published in 1937, is a measure of internal
consistency reliability for measures with
dichotomous choices. Covered already.
• Cronbach α (1951) is a measure of internal
consistency for measures with ordinal ratings
such as Likert scale.

Cronbach α
• All these items measures something similar;
i.e. fitness, therefore they are highly
correlated with one another and therefore
pointing in the same “direction”.
• Cronbach α can be used to check reliability of
a `composite score’ and will identify which
`items’ have problem in forming the
composite score; i.e. different direction.

Data for this lesson.
• https://wp.me/p4mYLF-sB

Internal Consistency RA
• Use PF-MH.sav: Analyze >>> Scale >>>
Reliability Analysis…
• Do physical & mental health separately.

Internal
Consistency RA

Cronbach α
• Cronbach α value is “very good”.

Interpreting the Mean
• “2” is the
neutral score.
• The mean is
above the
neutral score
for item PF2
until PF10. Item
PF1 is below
neutral score.
• The order of easiest to endorse “3” to most difficult;
PF5>PF10>PF3>PF6>PF8>PF4>PF9>PF2>PF7>PF1

PF5>PF10>PF3>PF6>PF8>PF4>PF9>PF2>PF7>PF1
Physical Activity Mean Endorse
5. Climb one flight of stairs 2.70 1
10. Bathing or dressing yourself 2.69 2
3. Lifting or carrying groceries 2.65 3
6. Bending, kneeling or stooping 2.49 4
8. Walking several blocks 2.37 5
4. Climb several flight of stairs 2.36 6
9. Walking one block 2.26 7
2. Moderate activities such as moving a table 2.21 8
7. Walking a mile (1.6 km) or more 2.11 9
1. Vigorous activities such as running or strenuous sports. 1.91 10
i
n
c
r
e
a
s
i
n
g
d
i
f
f
• The order do not exactly match our expectation.

Terminology
• Item Total Correlation (ITC) - Pearson
correlation between the scores for that item
and the average of the scores of the remaining
items.

Item Total Correlation (ITC)
• The higher the ITC, the higher is the “association” of
that item towards the fitness composite score.
• If there is any value larger than 0.869 in this column,
then the reliability is better without that item. Nope!

Item Total Correlation (ITC)?
• Item Total Correlation just mean the correlation of each
item against the Fitness Total Score.
• The higher the correlation, the higher the association of
each item against the Fitness Total Score.
This is Pearson
correlation
between the
Total Score of
All Items and
the ordinal
response of the
respective item.

Internal Consistency RA
• Use PF-MH.sav: Analyze >>> Scale >>>
Reliability Analysis…
• Now do for mental health.

Cronbach α
• Cronbach α value is “minimally
acceptable”.
• Need to add items to improve reliability.

Interpreting the Mean
• “3.5” is the
neutral score.
• The mean is
above the
neutral score
for all items.
• The order of easiest to endorse “6” to most difficult;
MH2>MH4>MH3>MH1>MH5

Item Total Correlation (ITC)
• The ITC for MH5 is too low at 0.197.
• Removing MH5 will improve the Cronbach α, from 0.658
to 0.719.
• Need to see if MH5 requires rewording or changes. Does
“MH5:Did you feel tired?” has the same theme as the
rest of the items?..................... No!

First 4 Items Are Symptoms of Depression
Mental Health Mean Endorse
2. Have you felt so down in the dumps that nothing could cheer
you up?
4.70 1
4. Did you feel worn out? 4.41 2
3. Have you felt downhearted and blue? 4.29 3
1. Have you been a very nervous person? 4.17 4
5. Did you feel tired? 4.10 5
l
e
s
s
• The last item MH5 is too general, not specific to depression or mental health.
Therefore MH5 should be be changed to something more specific for depression
or mental health.

Exploratory Factor Analysis

Analyze->Dimension Reduction>Factor

• KMO need to be 0.6 or
higher.
• Bartlett’s Test
significant means that
there is more than one
dimension.
Factor Analysis
Amount of variance from 15 items,
extracted by each factor is called
‘eigenvalue’ of each factor.

Factor Analysis - Options
• Easier to view the factor loadings of each
component once we removed the smaller values.
• Both PF & MH clearly split into two groups.

How many factors?
1. Factors eigenvalue ≥ 1 (Kaiser’s
criterion or rule)
2. Scree plot: The Cattell rule is to
pick all factors prior to where
the plot levels off. So, 4 factors
should be taken.
3. Comprehensibility: Though not
a strictly mathematical
criterion, we should limit the
number of factors to those
whose dimension of meaning is
readily comprehensible.
4. We should try with 2 factors
because we design the
questionnaire as PF & MH, 2
domains.
Level off here, so
take earlier 4.

Limit Factor to 2
• The two factors support our design as
two domains. Items in each domain
are correlated with items in the same
domain, but they are not correlated
with items in the other domain.
Loading is
good if it is
≥ 0.4, and is
serious
problem if
below <0.2.

How to present the result?
Mean
Std.
Dev.
Item-
Total
Corr.
Cronbach
α
Factor
1
Factor
2
Physical Functioning 01 1.91 .695 .450 .540
Mental Health 1 4.17 1.074 .408 .704
Mental Health 2 4.70 1.223 .541 .804
Mental Health 3 4.29 1.377 .474 .638
Mental Health 4 4.41 1.025 .519 .758
Mental Health 5 4.10 1.440 .197 .319
0.87
0.66
• Both reliability and
factor analysis reveal an
acceptable reliability
and a good validity of
the scale.
• Factor analysis
indicated the presence
of 2 separate
dimensions.

How to present the result?
Mean
Std.
Dev.
Item-
Total
Corr.
Cronbach
α
Factor
1
Factor
2
Mental Health 1 4.17 1.074 .408 .704
Mental Health 2 4.70 1.223 .541 .804
Mental Health 3 4.29 1.377 .474 .638
Mental Health 4 4.41 1.025 .519 .758
Mental Health 5 4.10 1.440 .197 .319
0.87
0.66
• Both reliability and
factor analysis reveal an
acceptable reliability
and a good validity of
the scale.
• Factor analysis
indicated the presence
of 2 separate
dimensions.
This is heresy. Since the responses are
ordinal in nature, calculating the mean and
standard deviation of the responses would
be wrong. Maybe it should be omitted.

Rasch Partial Credit

Steps
• Response Format
• Model Fit
• Internal Consistency
• Identify Item-Bias/DIF
• Targeting
• Dimensionality - Mental & Physical analysed
together at start. If PCA indicate more than
one dimension, then analysed separately.
Mental &
Physical
items done
separately.

Data for this lesson.
• https://wp.me/p4mYLF-sB
• Introduction to Rasch & Dichotomous Rasch
Model covered earlier at
https://www.slideshare.net/drtamil/difficulty-
index-discrimination-index-reliability-and-
rasch-measurement-analysis

Person=Ln(Score/Max-Score)

Item=Ln(Max-Score/Score)

Mental & Physical Function

Terminology
• infit - an information-weighted statistic.
• outfit - an unweighted statistic.
• MNSQ - a mean-square statistic. This is a chi-
squared statistic divided by its degrees of
freedom. It is expectated to be 1.0.
• ZSTD - standardized to approximate a theoretical
"unit normal", mean 0 and variance 1,
distribution. Expected mean to be 0.
• RMSE – Root Mean Squared Error. Summary
index of differences between predicted and
observed values.

TABLE 3.2
Step 1: Response Format Physical
Expected values are 1.0. Greater than 1.5
are problematic. These are OK.
S=separation;
1.4<s<5.0, in
this case;
1.12-(-1.12)
=2.24, so ok. If
>5 = add rating
<1.4 = merge
rating.
Endorsement frequencies

Category Characteristics Curve
Andrich Threshold of 1.12-(-1.12) =2.24 logits, showing
empirical distinction between the categories of 1, 2 & 3.
There is adequate option with Likert rating of 1 to 3.
>5 = add rating
<1.4 = merge
rating
TABLE 3.2

Step 2: Assess Model Fit
– If InFit Zstd close to 0 & MNSQ close
to 1, the data fits the model.
– If not, using Winsteps, drill down to
each person or item, and check fit-
residual. Consider deleting
items/people that have fit residuals
>2.0.

Fit Statistics
InFit Zstd Person & Item
are close to 0 & MNSQ are
close to 1, so the data fits
the model.

Scan the InFit Zstd for values
larger than 2.0. Such items
are considered erratic and
should be removed or
changed.
Here, all are less than 2.0
No items requires review.
-2 < Z < +20.5 < y < 1.5 0.32 < x < 0.8

Scan the InFit
Zstd for values
larger than 2.0.
Such items are
considered
erratic and
should be
removed or
changed.

Step 3a: Assess Internal
Consistency (Person)
Internal consistency of the constructs are examined using the
Sample Reliability of Person Separation (R) – a Rasch
based version of Cronbach's alpha.
R=G2/(1+G2); & Person Separation Index G=True S.D./RMSE
Person Separation Index (G) provides an indication of the
power of the measure to discriminate amongst respondents with
different levels of the trait being measured.
Separation index = 𝐺 = √((𝑅/(1−𝑅)) ) , Strata = (4𝐺+1)/3
r value of .8 is considered acceptable, representing the ability to
statistically differentiate at least 3 ability groups.
r value of .9 would indicate the ability to discriminate between 4
or more groups.

Example r=0.8
𝐺 = √((𝑅/(1−𝑅)))
= √((0.8/(1−0.8)))
= 2
Strata = (4x2+1)/3
= 3.
r value of .8 is considered
acceptable, representing the ability
to statistically differentiate at least 3
ability groups.

Example r=0.9
𝐺 = √((𝑅/(1−𝑅)))
= √((0.9/(1−0.9)))
= 3
Strata = (4x3+1)/3
= 4.3
r value of .9 would indicate the
ability to discriminate between 4 or
more groups.

G=True S.D./RMSE
G = 1.65/0.99 = 1.67
r = G2/(1+G2);
r=1.672/(1+1.672)
=2.7889/3.7889
=0.736
Strata=(4x1.67+1)/3
=2.56
Can have 3 groups.

From Table 3.1
Summary
Statistics
Separation index
𝐺=√((𝑅/(1−𝑅)))
=√((0.74/(1−0.74)))
=1.71
Strata=(4x1.71+1)/3
=2.61
Can have 3 groups.

0.96; ‘Very Good’ item
measurement reliability in
measuring fitness of patients.
Summary Statistics
(minus 16)
+ve Person mean
μ = 1.13 logit
P[Ɵ] LOi= e(1.13-0)/(1+e(1.13-0))
= 3.096/4.096
= 0.75 (easy to be fit)
InFit Zstd close to 0 & MNSQ
close to 1, so data fit the
model.
G=1.71
Separation index=(4G+1)/3=2.61
‘Good’ Person separation of 3
groups. So can have 3 levels!
0.74 ‘Fair’ person reliability
Cronbach-α not available here.
In earlier slide Cronbach-α=1.0
Interpretation of Person & item measurement reliability;
-<0.67 is Poor
- 0.67 – 0.80 Fair
- 0.81 – 0.90 Good
- 0.91 – 0.94 Very Good
e (βn – δi )
P(Ɵ) =
1 + e (βn – δi )
where;
e= Euler’s Number, 2.7183
βn= Person’s ability measure
δi= item difficulty measure

Step 3b: Assess Internal
Consistency (Item)
Internal consistency of the constructs are examined
using the Sample Reliability of Item Separation
(R) – a Rasch based version of Cronbach's alpha.
R=G2/(1+G2); &
Item Separation Index G=True S.D./RMSE
Item Separation Index (G) provides an indication
of the power of the measure to discriminate
amongst the items with different levels of the trait
being measured.
Item Separation Index = 𝐺 = √((𝑅/(1−𝑅)) ) ,
Strata = (4𝐺+1)/3

G=True S.D./RMSE
G = 1.00/0.20 = 5.00
r = G2/(1+G2);
r=52/(1+52)
=25/26
=0.961
Strata=(4x5+1)/3
=7
Can have 7 levels.

Step 4: Identify DIF
• Occurs when subgroups respond differently to the specific
item.
• Subgroups examples:
• Cross-cultural differences
• Language barriers
• Gender differences
• Differences in fitness states (e.g. stage of disease)
• Utilizing the scale in another disease (e.g. Validated in
this population & extended to the Stroke population)
• Winsteps provides the ability to assess for differences
in responses of subgroups. Uses DIF & Rasch-Welch t-
test to test item-bias.
• The Physical Score cannot be tested for differential item
functioning for gender or age, since we don’t have such data.

How to DIF? Table 30.

Step 5: T1. Assess Targeting
M=mean for Persons
Can we see 3 strata for
Persons?
M=mean for Items
Can we see 7 levels for Items?
Good physical health
Easiest items to endorse. Easy to give 3.
1. Fitness level poorer
than scale can assess,
patients responded
“1” to most items.
2.Fitness level better than
scale can assess; patients
scored “3” on most items.
n=49 (37%)
On target. Between
the mean + 1sd.
6/10 = 60%
Item most difficult to endorse. Hard to give 3.
1
Vigorous activities, such as ru
participating in strenuous spor
2
Moderate activities, such as m
vacuum cleaner, bowling, or p
4 Climbing several flights of stai
8 Walking several blocks
9 Walking one block
10 Bathing or dressing yourself
1
Vigorous activities, such as ru
participating in strenuous spo
2
Moderate activities, such as m
vacuum cleaner, bowling, or p
4 Climbing several flights of sta
Conclusion: Good targeting by the 10 items.

Conclusion – Physical OK
• Response Format – Separation of 2.24 logits, so
adequate Likert Scale selection.
• Model Fit - InFit Zstd close to 0 & MNSQ close to
1, the data fits the model.
– Persons – Fair reliability of 0.73, able to stratify
persons into 3 groups.
– Items – Very good reliability of 0.96, able to stratify
items into 7 levels.
• Identify Item-Bias/DIF – not possible to test.
• Targeting – items and persons well targeted.

Terminology
• Item Total Correlation (ITC) - Pearson
correlation between the scores for that item
and the average of the scores of the remaining
items.
• Point Measure Correlation (PTMA) - Pearson
correlation between the raw scores and the
item measures, as estimated from the raw
scores.

Compare SPSS with Rasch (Physical)
SPSS Rasch
Mean
Std.
Dev.
Item-
Total
Corr.
Cronbach
α
Factor
1
Factor
2
ning 01 1.91 .695 .450 .540
ning 02 2.21 .737 .625 .689
ning 03 2.65 .620 .478 .607
ning 04 2.36 .675 .587 .652
ning 05 2.70 .620 .619 .761
ning 06 2.49 .665 .631 .740
ning 07 2.11 .740 .622 .693
ning 08 2.37 .761 .638 .700
ning 09 2.26 .747 .651 .719
ning 10 2.69 .636 .560 .703
0.87
NAMEMEASURE SE PMCorr R
PF01 1.77 0.17 0.62
PF02 0.72 0.17 0.72
PF03 -1.12 0.21 0.55
PF04 0.15 0.18 0.67
PF05 -1.42 0.23 0.63
PF06 -0.37 0.19 0.68
PF07 1.05 0.17 0.72
PF08 0.12 0.18 0.70
PF09 0.5 0.18 0.73
PF10 -1.39 0.22 0.59
0.96

Step 1: Response Format Mental
Expected values are 1.0. Lesser than 0.5
S=separation;
1.4<s<5.0,
-1.27-(-1.20) =-0.07
-0.13-(-1.27) = 1.14
0.86-(-0.13) = 0.99
1.74-0.86 = 0.8
Need to merge some
scales.
TABLE 3.2

Results indicate there is too much option with scoring of 1 to 6.
Need to drop to only 1 to 3.
-1.27-(-1.20) =-0.07
-0.13-(-1.27) = 1.14
0.86-(-0.13) = 0.99
1.74-0.86 = 0.8
Need to merge some
scales.
1 = 1
2 = 1
3 = 2
4 = 2
5 = 3
6 = 3

Need to recode
Original Response New Coding
1 All the time
1 All the time
2 Most time
3 A good bit
2 Some time
4 Some time
5 A little time
3 Never
6 None at all
-1.27-(-1.20) =-0.07
-0.13-(-1.27) = 1.14
0.86-(-0.13) = 0.99
1.74-0.86 = 0.8

Step 1: Response Format Mental
Expected values are 1.0. Lesser than 0.5
S=separation;
1.4<s<5.0,
1.7-(-1.7) =3.4
New scales are OK.
TABLE 3.2

Andrich Threshold of 1.7-(-1.7) =3.4 logits, showing
empirical distinction between the categories of 1, 2 & 3.
There is adequate option with Likert rating of 1 to 3.

Likert 3 or Likert 6?
• Not only Rasch detected the problem, it also
provided the solution.
• If this is a pilot study, we shall proceed to the
full study using the MH questionnaire with
Likert 3 response format.
• If this is the full study, we shall proceed with
further analysis using the new recoded Likert
3 response dataset.
• Following analysis is based on recoded data.

Step 2: Assess Model Fit
– If InFit Zstd close to 0 & MNSQ
close to 1, the data fits the model.
– If not, using Winsteps, drill down to
each person or item, and check fit-
residual. Consider deleting
items/people that have fit residuals
>2.0.

Scan the InFit Zstd for values
larger than 2.0. Such items
are considered erratic and
should be removed or
changed.
Here, MH5 is larger than 2.0
MH5 requires review.
-2 < Z < +20.5 < y < 1.5 0.32 < x < 0.8

Step 3a: Assess Internal
Consistency (Person)
Internal consistency of the constructs are examined
using the Sample Reliability of Person
Separation (R) – a Rasch based version of
Cronbach's alpha.
R=G2/(1+G2); &
Person Separation Index G=True S.D./RMSE
Person Separation Index (G) provides an
indication of the power of the measure to
discriminate amongst respondents with different
levels of the trait being measured.
Separation index = 𝐺 = √((𝑅/(1−𝑅)) ) ,
Strata = (4𝐺+1)/3

G=True S.D./RMSE
G = 1.17/1.24 = 0.94
r = G2/(1+G2);
r=0.942/(1+0.942)
=0.8836/1.8836
=0.469
Strata=(4x0.94+1)/3
=1.58
Can have 2 groups.

G=True S.D./RMSE
G = 0.26/0.21 = 1.21
r = G2/(1+G2);
r=1.212/(1+1.212)
=1.4641/2.4641
=0.594
Strata=(4x1.21+1)/3
=1.95
Can have 2 levels.

Step 4: T30. Identify DIF
• Occurs when subgroups respond differently to the specific
item.
• Subgroups examples:
• Cross-cultural differences
• Language barriers
• Gender differences
• Differences in fitness states (e.g. stage of disease)
• Utilizing the scale in another disease (e.g. Validated in
this population & extended to the Stroke population)
• Winsteps provides the ability to assess for differences
in responses of subgroups. Uses DIF & Rasch-Welch t-
test to test item-bias.
• The Mental Health Score cannot be tested for differential item
functioning for gender or age, since we don’t have such data.

Step 5: T1. Assess Targeting
M=mean for Persons
Can we see 2 strata for
Persons?
M=mean for Items
Can we see 2 levels for Items?
Good mental health
Easiest item to endorse. Easy to give 6.1. Fitness level poorer
than scale can assess,
patients responded
“1” to most items.
2.Fitness level better than
scale can assess; patients
scored “3” on most items.
n=49 (37%)
On target. Between
the mean + 1sd.
3/5 = 60%
Conclusion: Poor targeting by the 5 items.
Need more items.
Item most difficult to endorse. Hard to give 6.

Conclusion – Mental KO
• Response Format – Separation of 3.34 logits after
recoding, so adequate Likert Scale selection.
• Model Fit - InFit Zstd close to 0 & MNSQ close to 1, the
data fits the model. But MH5 ZSTD > 2.0.
– Persons – Poor reliability of 0.47, able to stratify persons
into 2 groups.
– Items – Poor reliability of 0.6, able to stratify items into 2
levels.
• Identify Item-Bias/DIF – not possible to test.
• Targeting – items and persons poorly targeted. Need
more items.

Compare SPSS with Rasch (Mental)
SPSS Rasch
Mean
Std.
Dev.
Item-
Total
Corr.
Cron
bach
α
MH1 4.17 1.074 .408
MH2 4.70 1.223 .541
MH3 4.29 1.377 .474
MH4 4.41 1.025 .519
MH5 4.10 1.440 .197
0.66
NAMEMEASURESE PTMA-E R
MH1 0.48 0.2 0.678
MH2 -0.46 0.21 0.6322
MH3 -0.25 0.21 0.6245
MH4 0.04 0.2 0.6415
MH5 0.2 0.2 0.6531
0.6

Principal Component Analysis of Residuals

Step 6: Assess Dimensionality
The QOL scale should only measure
one concept, therefore can be
classed as unidimensional.
Winsteps assesses dimensionality
using Principal Components Analysis
of the residuals.

Dimensionality
• To check that all items share the same
dimension. This identifies sub-structures,
"secondary dimensions", in the data by
performing a principal components/contrast
decomposition of the observation residuals. If
there are large sub-structures, then it may be
wiser to divide the data into two
measurement instruments.

PCA of Residuals (All)

PCA of Residuals (All)
In the past, we
prefer this to be at
least 40% .
Larger than 2. Need to look
at items in cluster 1 versus
items in cluster 3.
15 because we
have 15 items.

Cluster 1 versus Cluster 3 (All)
Cluster 1 Items
Cluster 3 Items
Contrast
This vertically define the
contrast. One cluster is
close to the intended
dimension. The other
cluster is off-
dimensional, in a major
way.

Table 23.2 1st Contrast (All)
Cluster 1
Items
Cluster 3 Items

Disattenuated Person
Measure Correlation (All)
Less than 0.3. Proven
to have secondary
dimension.

Next Step
• There are two dimensions (PF & MH) in this
instrument, therefore further analysis should
be done separately.

PCA of Residuals (Physical)

PCAR (Physical)
In the past, we
least 40% . This is
49.7%
Larger than 2. Need to look
at items in cluster 1 versus
items in cluster 3.
15 because we
have 15 items.

Cluster 1 versus Cluster 3 (Phys)
Cluster 1 Items
Cluster 3 Items
Contrast
cluster is off-
dimensional, in a minor
way.

Table 23.2 1st Contrast (Phys)
Cluster 1 Items Cluster 3 Items

Measure Correlation
Larger than 0.3.
Unidimensional.

Conclusions (Physical)
• Dimensionality testing suggests the PF
scale is unidimensional and could be
used as an individual subscales for
fitness.

PCA of Residuals (Mental)

PCAR (Mental)
In the past, we
least 40% . This is
32.1%
Less than 2. No need to look at items
in cluster 1 versus items in cluster 3.
Accept unidimensionality.
5 because we
have 5 items.

Cluster 1 versus Cluster 3 (Mental)
Cluster 1 Items
Cluster 3 Items
Contrast
cluster is off-
dimensional, in a minor
way.

Table 23.2 1st Contrast (MH)
Cluster 1 Items Cluster 3 Items

Measure Correlation (MH)
Just larger than
0.3. MH scale is
Unidimensional.

Conclusions (MH)
• Dimensionality testing suggests the MH
scale is unidimensional and could be
used as an individual subscales for
mental health.

Overall Conclusion
On Dimensionality
The QOL as a total scale should
measure only one concept (QOL), and
each subscale should also have
unidimensionality (Physical Fitness &
Mental Health)
Results suggest that the subscales
(Physical Fitness & Mental Health) are
unidimensional but the total score
(QOL) is multidimensional.

CTT versus IRT
• EFA did an adequate job on validating the
instrument.
• Rasch Partial Credit Analysis did an even
better job. Not only in detecting the problems
within the data and items but also on the
response format (Likert Scale).
• It also provides the solution to the problem,
enabling the researcher to complete the
analysis.

References
• Lin Naing 2010. First UKMMC Intermediate To
Advance Biostatistics In Medical Sciences
Workshop Lecture Notes.
• Zali Mohd 2019. Training of Trainers on Rasch
Measurement Analysis Lecture Notes.
• John M. Linacre 2019. A User's Guide to
WINSTEPS®/MINISTEP Rasch-Model Computer
Programs Program Manual 4.4.0.
• Benjamin D. Wright & Geoff Masters 1982. Rating
Scale Analysis (Rasch Measurement Series).

Thank You!
• Rasch Malaysian Association (PERAMAL) for
the training and feedback on these slides.

Validating An Instrument: CTT (EFA) versus IRT (Rasch Partial Credit)

Recommended

Recommended

More Related Content

What's hot

What's hot (8)

Similar to Validating An Instrument: CTT (EFA) versus IRT (Rasch Partial Credit)

Similar to Validating An Instrument: CTT (EFA) versus IRT (Rasch Partial Credit) (20)

More from Azmi Mohd Tamil

More from Azmi Mohd Tamil (20)

Recently uploaded

Recently uploaded (20)

Validating An Instrument: CTT (EFA) versus IRT (Rasch Partial Credit)