Recommended
More Related Content
Viewers also liked
Recently uploaded
Recently uploaded (20)
Waop presentation 2016 a basket case
- 1. eelloo.nl A basket case Using baskets and verification in unproctored testing Jozien Silvester Liesbeth van Leeuwen Jelle Geertsma WAOP Presentation 25 november 2016
- 2. eelloo.nl Unproctored testing • Stakes are high in selection • Possibility of cheating • Proctoring not always possible or efficient • Benefits for candidates of unproctored testing: time and place of choice
- 3. eelloo.nl Unproctored testing • Validity proctored /unproctored equal if: • Informing candidates about optimal test conditions • Taking measures to prevent fraud • Items that measure reasoning (fluid) • Time limit • Basket method
- 4. eelloo.nl Basket method • Large item bank • Items of equal difficulty are placed in a basket • Item is randomly picked from basket • Same number of baskets for every candidate • Four subtests: Figure series, Analogies, Diagrams, Number series
- 5. eelloo.nl Benefits of basket method • Easy calculation • Easy monitoring, maintaining, extension • Easy retesting (system remembers which items were already administered) • Prevention of item leakage
- 6. eelloo.nl Validity • High correlation with proctored extended test (.75) • Test-retest correlation (.77) • About 8% scores lower than expected, possible cheaters
- 8. eelloo.nl Basket monitoring • Some items with equal difficulty are more equal than others Basket Item Difficulty (n) Difference Average 1017 Hcyf106b 0.66 (287) 0.10 0.68 Hcyf106c 0.65 (266) HCYF01b 0.74 (294) 1018 HCYF112 0.62 (409) 0.02 0.63 HCYF112b 0.64 (438)
- 9. eelloo.nl Difference in p-value • Even if items are carefully balanced, there will still be differences, although they may be small • For most candidates this will average • Some will have all the ‘easy’ or all the ‘difficult’ items • Effect on reliability and score • Comparison of correction methods on two measures
- 10. eelloo.nl Score calculation improvement - reliabilities (split half) Education level N CTT IRT1pl IRT2pl Elo Glicko MBO (Vocational) 743 0.85 0.80 0.81 0.79 0.86 HBO (Bachelor) 1397 0.82 0.82 0.83 0.79 0.85 WO (Master) 706 0.80 0.76 0.77 0.74 0.81 • Glicko has best performance • CTT has good results as well
- 11. eelloo.nl Effects Glicko on z-score Item difficulty group regular score Glicko score z-score
- 12. eelloo.nl Concluding remarks • Valid unproctored test possible with basket method • Glicko shows slight improvement in score calculation, but CTT also works well
Editor's Notes
- Measures are: Controletest dreigen met Vragen vooral op fluid, opzoeken op internet heeft geen zin Verification test Question type reasoning rather than knowledge
- Testhelften Acap