This is the second of a series of powerpoints presented at a CAT/IRT workshop at the University of Brasilia in 2012. It provides a discussion on how IRT is applied to developing better assessments, including item and test information functions, standard error of measurement, and use of Xcalibre. Learn more at www.assess.com.

1. Day 1 PM: Using IRT
Item and test information
Comparison of IRT to Classical Test Theory
How to do IRT analysis

2. Part 1
Item and test information

3. Information
 Information is the tool that IRT uses
to build tests
 It is a statistical term that quantifies
how much something “adds” to a
procedure
 Or, alternatively, how much
uncertainty (error) it decreases
 A good test has a lot of information!

4. Item information
 IRT calculates information for each
item and test at each level of q
 It is therefore not a single number –
it is a function across ability
 Each item has an item information
function
 Each test has an test information
function

5. Item information
 Some items provide information for
high students, some for low
 Same is true for tests: a test can be
more accurate for certain score
ranges – and IRT will tell you which

6. Information
 Item information is summative, that
is, it can be added up to obtain the
test information function (TIF)
 Then we know where to add/subtract
items
 Bonus: The TIF can also be inverted
to obtain a predicted SEM curve

7. Item information
 With CTT, “information” can be
conceptualized by jointly considering
the P and rpbis
◦ Obviously, a higher rpbis is better
 Definitely don’t want negative!
◦ P represents which examinees it is most
appropriate for
 P = 0.95 is easy, good for low examinees
 P = 0.50 is hard, good for high examinees

8. Item information
 But since items and examinees are
not on the same scale, there is no
direct connection
 With IRT, there is
 Item with b = 0.7 is good for person
with q = 0.7
◦ This is the basis of adaptive testing –
doing this continually

9. Item information
 Item information takes this idea and
quantifies it across the spectrum
 It is therefore a function of q as well as
the item parameters
 Where P(q) is the probability of a
correct answer for a given value and
Q(q) is 1-P
 
2
2 2 ( ) ( )
I
( ) 1
i i i
i
i i
Q P c
D a
P c
 q q 
q   
q  

10. Item information
 That is the computational equation
 Conceptual version that is seen in the
literature is
 Or the slope squared over the
conditional variance
   
2
I ( ) / ( ) (1 ( ) )i i iP P Pq q q q 

11. Graphing info
 So what does this mean?
 We calculate with that equation, and
it will be higher wherever the slope
of the IRF is higher (for a given value
of q)
 This is the item information function
(IIF)

12. Graphing info
 So the location of the item
determines the location of the IIF
 The discrimination of the item
determines the spread/peakedness of
the IIF
 Information decreases as the guessing
parameter increases

13. Some example items
Seq a b c
1 1.00 -2.00 0.26
2 0.70 -1.00 0.21
3 0.40 -0.50 0.30
4 0.50 1.00 0.00
5 0.80 0.00 0.22

14. Example item IRFs

15. IIFs – example items

16. Graphing info functions
 Note that a lower slope is not ALL
bad
 Even though Item 3’s peak is lower, it
provides some info at a much wider
range
 So items like that are quite useful
when info is needed across a wide
range

17. Using item info
 Item information is inversely related
to error in measurement
 If the item provides more info, it
reduces error
 The equation:
   2/1
1 qq ISEM 

18. Using item info
Key point: an item has less
error where it has more
information
--> where it has more slope
A test has less error where
it has more information
(items)

19. Using item info
 IIFs are another way to examine
items individually
 They are also what adaptive testing
utilizes for item selection
 But the best use of item info: test
information and test assembly…

20. Test information
 As a result of the assumption of local
independence, IIFs can be summed to
obtain a test information function
(TIF)
 Same is true for IRFs – they can be
summed into a TRF
◦ This converts thetas to estimated raw
score

21. Test information
 Test information, like item
information, shows how well a test
measures at each value of q
 Also inverts to CSEM
 This is extremely useful for test
assembly (aka construction, design,
or building)

22. Test information
 Consider the 5 IRFs…

23. Test information
 The TRF is…

24. Test information
 Consider the 5 IIFs…

25. Test information
 The TIF is…

26. Test information
 The CSEM curve is…

27. Test assembly
 Form building is more efficient and
better directed with IRT
 Reason: we can predict measurement
error (SEM) at each level of θ, not
just overall reliability

28. Test assembly
 This then allows you to build test
forms with specific TIFs or CSEMs in
mind
 Or multiple forms with the same TIF
 The following figures have the same
average a (0.9) but differ in where
they provide information

29. TRFs

30. TIFs

31. CSEMs

32. Test development
 You can build your test with specific
TRF/TIF/SEM graph in mind
 Peak at cutscore?
 This can be done inside item bankers
(FastTEST & FT Web) or in separate
spreadsheets (my Form Building Tool)

33. Bank development
 You can also build the bank for a
testing program with the desired TIF
in mind
 If you know you want it to be peaked,
write items at the desired level of
difficulty to build an adequate bank

34. Bank development
 Otherwise you risk overexposure
 Don’t use all your best items at once
to make a peaked TIF – or any TIF for
that matter
 In the theoretical IRT world, we don’t
have to worry about that, but
exposure is a real issue

35. Bank development
 That is the reason linear-on-the-fly
(LOFT )was developed – to massively
reduce exposure and increase
security
◦ Every person gets an very similar TIF, but
a completely different test
◦ These tests are parallel, from an IRT
point of view
◦ Tests are conventional fixed-form

36. Part 2
A brief comparison of CTT and IRT

37. CTT and IRT Assumptions
 IRT:
◦ Unidimensionality and local independence
◦ Responses modeled by IRF
◦ Parameters, not statistics (sample
independence)
 CTT:
◦ X = T + E
◦ (1) true scores and error scores are
uncorrelated; (2) the average error score in
the sample is zero
◦ Statistics (not parameters) are sample-based

38. Comparing CTT and IRT
 CTT is said to have weaker assumptions
◦ Does not explicitly assume
unidimensionality
 But if not there, statistics will be iffy, and rpbis
and reliability suffer
 Sum scoring implicitly assumes items are
equivalent, which means unidimensional (all
items count equally on one total score)

39. Comparing CTT and IRT
 CTT is said to have weaker assumptions
◦ Does not explicitly assume IRF
 But if the idea of an IRF is not working, then the
item isn’t either
 And if you use rpbis, you assume a linear IRF,
which is actually impossible!

40. Comparing CTT and IRT
 CTT item statistics are at odds with
each other
◦ P says that there is one common
probability of a correct response
(binomial)
◦ But rpbis says that P increases with total
score (~ability)

41. Comparing CTT and IRT
 Classical SEM: same for everyone
 IRT SEM: different for everyone –
depends on the items you see and
your ability
 Which is more realistic?

42. Comparing CTT and IRT
 Direct comparison of item statistics
◦ We still use “difficulty” and
“discrimination”
◦ How different are they from CTT?
◦ Difficulty correlates highly (>0.90)
◦ Discrimination does not – because Rpbis
is linear and IRT is not

43. Comparing CTT and IRT
 IRT and CTT scores also correlate
>0.95
 So why use IRT?
 There are distinct advantages…

44. Advantages of IRT
 IRT has parameters, not statistics
 Sample-independent… within a linear
transformation
 Huh? This means that if you have two
calibration groups of different levels,
we can convert parameters/scores
with a simple y = mx + b
 (Linking)

45. Advantages of IRT
 Items and people are on the same
scale
 Easier to interpret, and allows
adaptive testing

46. Advantages of IRT
 Information provides an important
tool for test building and bank
development
 Better match the purposes of a test
 IRT CSEM allows far better
description of precision

47. Advantages of IRT
 More precise scores
 CTT number correct scoring is limited
to k + 1 scores
 3PL has 2k scores
 Compare with 10 items:
◦ 11 vs 1024 possible scores

48. Advantages of IRT
 Scores take item difficulty into
account
 Allows direct comparison of
examinees that saw different sets of
items
 Scores also account for guessing

49. Advantages of IRT
 Nonlinear IRF – the linear IRF
assumed by CTT is impossible
 Allows for different SEM for every
examinee
 Not realistic to assume they are all
the same

50. Disadvantages of IRT
 Sample size
 CTT: 50 is OK, 100 is great
◦ It is much easier to fit a straight line
“model” than an IRF because it is an
oversimplification
 IRT: 100 is bare minimum for 1PL
◦ 3PL? ~500
◦ Puts it out of reach of small testing
programs

51. Disadvantages of IRT
 No “native” distractor analysis unless
polytomous models
 Can adapt the CTT idea of
quantile/distractor plot with IRT
◦ IRT programs will also give you option P
and Rpbis

52. Disadvantages of IRT
 Complexity
◦ Not only do you have to understand it
yourself, but…
◦ You also have to explain it to
stakeholders!

53. Disadvantages of IRT
 However, note that these are not big
problems
◦ Many places have plenty of sample size
◦ You can still use CTT for distractor
analysis (always use both!!!!)
◦ The complexity is not too bad unless
using complex models
◦ Often, the biggest issue is the
stakeholders!

54. IRT Analysis
How do I go about doing this?

55. IRT Analysis
 Xcalibre 4 for IRT
 CTT analysis with Iteman 4 (not
necessary, but sometimes helps)
 Also:
◦ Scoring and graphing tool
◦ Form building tool
◦ Empirical IRFs in Excel
◦ Have we covered these sufficiently?

56. IRT Analysis
 I’m assuming here we are analyzing
just one sample of one test
 What would I look for? Basic…
◦ Items with good parameters (keep/clone)
◦ Items with bad parameters (retire)
 Evaluate their CTT option statistics
◦ TIF/CSEM – meet our needs? (not
good/bad in absolute sense)

57. IRT Analysis
 What would I look for? Advanced…
◦ Dimensionality assessment (reliability,
any items/sections “off on their own”)
◦ Item fit (also dimensionality, and possible
item issues)
◦ Test sections – any stand out for being
hard, easy, low discriminations, poor
precision, etc?
◦ CSEM/TIF for sections: anything under-
measured?

58. IRT Analysis
 What would I look for? Advanced…
◦ Finally: what do you want to see in the
data, and how will the test be used?
 Later, we’ll talk about more
advanced uses like:
◦ Linking and equating multiple forms
◦ Test assembly
◦ Adaptive testing
◦ Dimensionality evaluation

59. Iteman 4.1
 Performs comprehensive classical
analysis
 Quantile plots allow broad evaluation
of IRF shape
 Advantages:
◦ Easily understandable – can use with SMEs
◦ Includes distractors

60. Xcalibre 4.1
 Provides a comprehensive and user-
friendly IRT analysis
 Allows evaluation of individual items
and test as a whole
 All major graphs
 Many summary graphs (freqs etc.)
 Classical analysis too

61. Reasons for Xcalibre 4.1
 Current available software (Parscale,
Bilog, Multilog, ConQuest, WinSteps,
ICL) still require programming skills
 Some still run on DOS!
 If IRT is to be more widely used, it
needs a user-friendly system
◦ Input and output

62. Reasons for Xcalibre 4.1
 Better input
◦ Yes: Point and click buttons
◦ No: DOS programming quasi-language
 Better output
◦ Yes: Word docs (RTF), spreadsheets (CSV)
◦ No: DOS txt files with ugly tables

63. Reasons for Xcalibre 4.1
 Advanced users with programming
skills and need for customized analysis
can still utilize previous software
 Xcalibre 4.1 is designed for a wider
range of users
 The following description is of Xcalibre
4, but also applies to Iteman 4

64. Xcalibre 4.1 Interface
 Divided into tabs
 Move left to right…

65. Xcalibre 4.1 Interface
 All options are specified with buttons
or simple entry boxes
 No code based on keywords
◦ Best example: IRT models (you’ll see)
 Also: usable error messages

66. Specify files/input; choose options
 I’ll now show how to use X4, and do
some analysis of real data…