Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
×

ABCs of IRT

825 views

Published on

Presentation given at ICE 2010 (Atlanta) regarding basic concepts and vocabulary of Item Response Theory

• Full Name
Comment goes here.

Are you sure you want to Yes No
Your message goes here
• Be the first to comment

• Be the first to like this

ABCs of IRT

1. 1. November 18, 2010 Diane M. Talley, MA Stephen B. Johnson, PhD James A. Penny, PhD Castle Worldwide
2. 2. Psychometrics as Science and Art 2010 ICE Educational Conference
3. 3.  IRT and Classical  Concepts of IRT – A logit – The abc’s  Benefits – Pre-equating – immediate scoring – Population invariance  Assumptions  Implications 2010 ICE Educational Conference
4. 4. The right tools for the job  Data  Program  Tool 2010 ICE Educational Conference Versus
5. 5. Classical versus IRT model 2010 ICE Educational Conference
6. 6. Classical versus IRT Classical Model IRT Model  Traditional  Modern  Requires less strict adherence to assumptions  Requires stricter adherence to assumptions  Sample dependent  Population invariant  Statistics (p – diff, p-biserial – disc)  Probability-based statistics (b-diff, a-disc, c-guessing)  Simple scoring model (raw score)  Scoring is more complex 2010 ICE Educational Conference
7. 7. What’s a logit? Ability The Performance Standard Probability 2010 ICE Educational Conference
8. 8. b (difficulty) 2010 ICE Educational Conference 0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00 -3 -2.8 -2.5 -2.3 -2 -1.8 -1.5 -1.3 -1 -0.8 -0.5 -0.3 0 0.25 0.5 0.75 1 1.25 1.5 1.75 2 2.25 2.5 2.75 THETA P(u=1|THETA) Paint by Numbers Leonardo 1 4 3 2 5
9. 9. a (discrimination) and b 2010 ICE Educational Conference 0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00 -3 -2.75 -2.5 -2.25 -2 -1.75 -1.5 -1.25 -1 -0.75 -0.5 -0.25 0 0.25 0.5 0.75 1 1.25 1.5 1.75 2 2.25 2.5 2.75 THETA P(u=1|THETA) Paint by Numbers Leonardo 1 2 3
10. 10. a, b, and c (guessing) 2010 ICE Educational Conference 0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00 -3 -2.75 -2.5 -2.25 -2 -1.75 -1.5 -1.25 -1 -0.75 -0.5 -0.25 0 0.25 0.5 0.75 1 1.25 1.5 1.75 2 2.25 2.5 2.75 THETA P(u=1|THETA) Paint by Numbers Leonardo 1 2 3
11. 11. Fit statistics Comparison of Infitand Outfit 0 1 2 3 4 5 6 Infit Outfit ItemOrder ICE 2010 Conference Atlanta Georgia Outfit Mean Square Plot 0 0.2 0.4 0.6 0.8 1 1.2 0 5 10 15 20 25 30 Item Order MSQ Infit Mean Square Plot 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 0 5 10 15 20 25 30 Item Order MSQ
12. 12. Population Invariance Low Performing High Performing Item 1 .15 .50 Item 2 .60 .80 Item 3 .70 .92 Classical Difficulty Values IRT Difficulty Values Low Performing High Performing Item 1 1.50 1.50 Item 2 0.00 0.00 Item 3 -.75 -.75 2010 ICE Educational Conference
13. 13. IRT Pre-Equating  What does it mean?  Why would you want to do it?  What does it mean for building item banks and forms? 2010 ICE Educational Conference
14. 14. Test Information Function (TIF) Comparison of Test Information Functions 0.0 10.0 20.0 30.0 40.0 50.0 60.0 70.0 -3 -2.75 -2.5 -2.25 -2 -1.75 -1.5 -1.25 -1 -0.75 -0.5 -0.25 0 0.25 0.5 0.775 1.025 1.275 1.525 1.775 2.025 2.275 2.525 2.775 3.025 Theta Information Form A Form B 2010 ICE Educational Conference
15. 15. Assumptions  Unidimensionality  Local Independence 2010 ICE Educational Conference
16. 16. Implications  Item writing – Leave those scored items alone! – Focused item writing targeting the performance standard  Assembly – Items selected for a form should be around the standard  Testing and Reporting – Field test items for pre-equating/on-demand scoring – Form assignment – Scoring – Recalibration – Harder to explain to stakeholders 2010 ICE Educational Conference
17. 17. Does IRT make sense for you?  What is the size and maturity of your program and item bank?  Do you like to tinker with items?  Do your program requirements change frequently?  How experienced/capable are your item writers?  How do you score candidates?  IRT or number correct  Do you hold scores or do immediate scoring?  Can you afford a psychometrician? 2010 ICE Educational Conference
18. 18. Questions? Diane M. Talley dtalley@castleworldwide.com James A. Penny jpenny@castleworldwide.com Stephen B. Johnson sjohnson@castleworldwide.com 919.572.6880