1) The document discusses challenges in modeling situational judgement tests (SJTs) using item response theory (IRT) models due to their multidimensional nature. 2) It compares using a multidimensional IRT model that accounts for item dependencies versus a testlet model that treats clusters of related items as single polytomous items. 3) The results show that while the multidimensional model had better model fit, the testlet model performed reasonably well and avoided loss of measurement precision at lower proficiency levels.

1. IRT Modelling Situational judgment tests:
Items vs. Testlets
Tao Li, Hogrefe Ltd. UK
BACKGROUND
Surge of interest in utilising Situational judgment Tests (SJTs) in personnel selection and development
Despite its popularity, little is known about the psychometric properties of SJTs
SJTs pose challenges for measurement: classical test theory methods are often inadequate to deal with its complexity
This study addresses the applications of item response theory (IRT) to SJTs and seeks for a greater understanding of SJTs
sTJS gnilledoM ni segnellahC sTJS gnilledoM TRI rof seigetartS
Multidimensional both at the item and test level Multidimensional Model: Explicitly model and account for
multidimensionality at the items level
Items are nested within scenarios and response to
items are context dependent, introducing local dependency
These features violate key assumptions of dominantly Testlet Model: treat each scenario and related items as a single
employed unidimensional IRT model polytomous item to estimate an unidimensional IRT model
METHOD
Instrument: Leadership judgment Indicator® (Hogrefe). A theory-based, construct-driven SJT that measures accuracy of
judgment in using four styles of leadership (directive, consultative, consensual, delegative) when dealing with leadership decisions
Sample: 1345 UK managers across industry sectors and organisational hierarchies.
ledoM lanoisnemiditluM ledoM teltseT
Account for conditional dependency by incorporating specific Testlet is a collection of clustered items that are all related to a
dimensions in addition to the general dimension . common stimulus
Using this strategy, a testlet is modelled as a scoring unit and local
The multidimensional IRT Model can be implemented in item dependency is partial out across testlets
a latent variable modelling framework (e.g. Mplus) Fitting an unidimensional polytomous IRT model to bundled
scenario scores
RESULTS
TRI lanoisnemiditluM :noitcnuF noitamrofnI tseT TRI teltseT :noitcnuF noitamrofnI tseT
Model Fit: CFI=0.91; RMSEA=0.021 Model Fit: CFI=0.95; RMSEA=0.039
DISCUSSION
Treating the data as testlets resulted in loss of information (precision of measurement)
The incremental test information gained from a multidimensional IRT model seems to
be located on the lower end of the latent proficiency scale. Depending on assessment
For more information,
purpose, this may not be desirable in an occupational setting.
please contact Tao Li at
The testlet model performed reasonably well in modelling LJI.
The multidimensionality nature of SJT item responses does not necessitate the use of tao.li@hogrefe.co.uk
complex multidimensional models.