Your SlideShare is downloading. ×
0
Upcoming Caveon Events• Caveon Webinar Series: Next session, June 19Protecting your Tests Using Copyright Law• Presenters ...
Latest Publications• Handbook of Test Security – Now available forpurchase! We’ll share a discount code before endof sessi...
Caveon Online• Caveon Security Insights Blog– http://www.caveon.com/blog/• twitter– Follow @Caveon• LinkedIn– Caveon Compa...
“Using Decision Theory to ScoreAccurate Pass/Fail Decisions”Lawrence M. Rudner, Ph.D., MBAVice President and Chief Psychom...
Agenda for today• Role of decision theory• Examples• Logic• Tools• Adaptive Testing
Goal of Measurement Decision TheoryClassify an examinee into one of K groups– mastery/non-master– below basic / basic / pr...
Poll #1Are you involved with any classificationtests as part of your work?Attendee Responses:Yes – Pass/Fail – 49%Yes - Ye...
Poll #2How familiar are you with Item ResponseTheory?Attendee Responses:Very – I understand and routinely apply IRT formul...
Poll #3What is your primary job function?Attendee Responses:Teacher or Content Expert -6%Item Writer – 8%Psychometrician –...
Usual ApproachPopulation Distribution
Usual ApproachPopulation Distribution
New ThinkingProbability of being a Masteror a Non-Master0.00.10.20.30.40.50.60.70.80.91.0Non-Master Master
A Different QuestionOld: Your score was 76 which is above thepassing score of 72. You passed.vsNew: Probability of this re...
IRT ApproachProbability of a correct response to Question 123 given ability levelQuestion 1230.00.10.20.30.40.50.60.70.80....
0.00.10.20.30.40.50.60.70.80.91.0Non-Master MasterNew ThinkingProbability of a correct response to Question 123for Masters...
Advantages• Simple framework• Small number of items• Small calibration sample sizes• Classifies as well as or betterthan I...
Applications• Intelligent Tutoring Systems• Diagnostic Testing• Personality Assessment• Automated Essay Scoring• Certifica...
Examples
A Certification Examination
MDT
Logic
Notation• K - # of mastery states• P(mk) - Prob of a randomly drawn examinee beingin each mastery state k• z - an individu...
WantP(mk | z )The probability of each mastery state k, mk, given theresponse vector z.The probability of being a master gi...
Do you recognize these people?
Bayes Theorem• P(a|b)*P(b) = P(b|a)*P(a)k k kP(m | ) P( )= P( |m ) P(m )cz z z
Mastery state(using Bayes Theorem)P(m | ) = P( |m ) P(m )k k kz zcBut there are too many possible responsevectors z
Mastery state(using Bayes Theorem)P(m | ) = P( |m ) P(m )k k kz zcBut there are too many possible responsevectors zP( |m )...
Basic ConceptConditional probabilities of a correct response,P(zi=1|mk)Item 1 Item 2 Item 3Masters (m1) .8 .8 .6Non-master...
Probability of the response vector z for eachmastery state is:P(z| m1) =.8 * .8 * (1-.6) = .26Conditional probabilities of...
Probability of the response vector z for eachmastery state is:P(z| m1) =.8 * .8 * (1-.6) = .26P(z| m2) =.3 * .6 * (1-.5) =...
Probability of the response vector z for eachmastery state is:P(z| m1) =.8 * .8 * (1-.6) = .26P(z| m2) =.3 * .6 * (1-.5) =...
Probability of the response vector z for eachmastery state is:P(z| m1) =.2 * .2 * .6 = .024P(z| m2) =.7 * .4 * .5 = .14Con...
Probability of the response vector z for eachmastery state is:P(z| m1) =.2 * .2 * .6 = .024P(z| m2) =.7 * .4 * .5 = .14Nor...
Conditional probabilities of a correct response,P(zi=1|mk)Item 1 Item 2 Item 3Masters (m1) .8 .8 .6Non-masters (m2) .3 .6 ...
Probability of the response vector z for each masterystate is:P(z| m1) =.8 * (1-.8) * .6 = .096P(z| m2) =.3 * (1-.6) * .5 ...
Decision Criteria
Decision Rule – Maximum Likelihood00.050.10.150.20.250.3P(z|mk)MasterNon-Master• Probability of the response vector, z, fo...
Decision Rule - Maximum a posterioriprobability• Probability of each mastery state isP(m1|z) = c * .26 *.7 = c* .52 = .87P...
Decision CriteriaBayes RiskGiven a set of itemresponses z and thecosts associatedwith eachdecision, select dk tominimize t...
Tools
Tools and Resourceshttp://edres.org/mdt• Paper• Java Applet• Download Excel tool• Tools for– Data Generation– Item Calibra...
http://bit.ly/pareonline
Example
Adaptive Testing
1. Sequentially select items to maximizecertainty,2. Administer and score item,3. Update the estimated mastery stateclassi...
Claude Shannon
EntropyA measure of the disorder of a system.How many bits of information are needed to senda) 1,000,000 random signalsb) ...
Less peaked = more uncertainty= more entropy0.00.20.40.60.81.0Non-Master Master0.00.20.40.60.81.0Non-Master MasterH(s) = 1...
Adaptive Testing0.20.40.60.810 5 10 15 20 25 30 35 40 45 50Max No of itemsProportionAccuracyClassifiedPercent classified v...
Recap• Simple framework• Small number of items• Classifies as well as or better thanmuch more complicated IRT• Effective f...
Option For• Small certification programs• Large certification programs• Embedded in instructional systems• Test preparation
HANDBOOK OF TEST SECURITY• Editors - James Wollack & John Fremer• Published March 2013• Preventing, Detecting, and Investi...
Questions?Please type questions for our presenters in theGoToWebinar control panel on your screen
THANK YOU!- Follow Caveon on twitter @caveon- Check out our blog…www.caveon.com/blog- LinkedIn Group – “Caveon Test Securi...
Caveon Webinar Series: Using Decision Theory for Accurate Pass/Fail Decisions
Caveon Webinar Series: Using Decision Theory for Accurate Pass/Fail Decisions
Caveon Webinar Series: Using Decision Theory for Accurate Pass/Fail Decisions
Caveon Webinar Series: Using Decision Theory for Accurate Pass/Fail Decisions
Caveon Webinar Series: Using Decision Theory for Accurate Pass/Fail Decisions
Caveon Webinar Series: Using Decision Theory for Accurate Pass/Fail Decisions
Caveon Webinar Series: Using Decision Theory for Accurate Pass/Fail Decisions
Caveon Webinar Series: Using Decision Theory for Accurate Pass/Fail Decisions
Caveon Webinar Series: Using Decision Theory for Accurate Pass/Fail Decisions
Upcoming SlideShare
Loading in...5
×

Caveon Webinar Series: Using Decision Theory for Accurate Pass/Fail Decisions

323

Published on

The traditional approach to classification testing is extremely inefficient and often difficult to implement in applied settings. Typically, examinees are rank ordered either through Item Response Theory or Classical Test Theory, and then scores are compared to a difficult-to-define cut score.

This webinar will introduce the use of decision theory which basically asks: “Does this response pattern look like the response pattern of a master or a non-master?” This simpler model has major advantages over IRT and CTT:

1. Only a small sample of clear masters and a small sample of clear non-masters are needed to calibrate questions.
2. There are no assumptions for unidimensionality, and normal distribution or requirement for monotonically increasing probabilities of correct responses.

This model is attractive and a natural for end-of-unit examinations, adaptive testing, and as the routing mechanism for intelligent tutoring systems.

This webinar will explain the model, identify current applications, and introduce free tools for generating, calibrating and scoring data.

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
323
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
18
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Are you involved with any classification tests as part of your work?Yes – Pass/FailYes – Multiple categories, e.g. A,B,C,D,FNo
  • Are you involved with any classification tests as part of your work?Yes – Pass/FailYes – Multiple categories, e.g. A,B,C,D,FNo
  • Are you involved with any classification tests as part of your work?Yes – Pass/FailYes – Multiple categories, e.g. A,B,C,D,FNo
  • Abraham Wald (October 31, 1902(1902-10-31) - December 13, 1950) was a mathematician born in Cluj, in the then Austria–Hungary (present-day Romania) who contributed to decision theory, geometry, and econometrics, and founded the field of statisticalsequential analysis.[1]was thus home-schooled by his parents until college.[1] His parents were quite knowledgeable and competent as teachers.[2]Emigrated to US to avoid the nazi’sThomas Bayes (pronounced: ˈbeɪz) (c. 1702 – 17 April 1761) was an Englishmathematician and Presbyterian minister, known for having formulated a specific case of the theorem that bears his name: Bayes' theorem, which was published posthumously.
  • Shannon is famous for having founded information theory with a landmark paper that he published in 1948. However, he is also credited with founding both digital computer and digital circuit design theory in 1937, when, as a 21-year-old master's degree student at the Massachusetts Institute of Technology (MIT), he wrote his thesisdemonstrating that electrical applications of boolean algebra could construct and resolve any logical, numerical relationship. It has been claimed that this was the most important master's thesis of all time.[3] Shannon contributed to the field of cryptanalysis for national defense during World War II, including his basic work on codebreaking and secure telecommunications.
  • Transcript of "Caveon Webinar Series: Using Decision Theory for Accurate Pass/Fail Decisions "

    1. 1. Upcoming Caveon Events• Caveon Webinar Series: Next session, June 19Protecting your Tests Using Copyright Law• Presenters include Intellectual Property Attorney Kenneth Horton and amember of the Caveon Web Patrol team• Register at: http://bit.ly/protectingip• NCSA – June 19-21 National Harbor, MD– Dr. John Fremer is co-presenting Preventing, Detecting, and Investigating TestSecurity Irregularities: A Comprehensive Guidebook On Test Security For States– Visit the Caveon booth!
    2. 2. Latest Publications• Handbook of Test Security – Now available forpurchase! We’ll share a discount code before endof session.• TILSA Guidebook for State AssessmentDirectors on Data Forensics – coming soon!
    3. 3. Caveon Online• Caveon Security Insights Blog– http://www.caveon.com/blog/• twitter– Follow @Caveon• LinkedIn– Caveon Company Page– “Caveon Test Security” Group• Please contribute!• Facebook– Will you be our “friend?”– “Like” us!www.caveon.com
    4. 4. “Using Decision Theory to ScoreAccurate Pass/Fail Decisions”Lawrence M. Rudner, Ph.D., MBAVice President and Chief PsychometricianResearch and DevelopmentGMAC®May 15, 2013Caveon Webinar Series:Jamie Mulkey, Ed.D.Vice President and General ManagerTest Development ServicesCaveon
    5. 5. Agenda for today• Role of decision theory• Examples• Logic• Tools• Adaptive Testing
    6. 6. Goal of Measurement Decision TheoryClassify an examinee into one of K groups– mastery/non-master– below basic / basic / proficient / advanced– A / B / C / D / F
    7. 7. Poll #1Are you involved with any classificationtests as part of your work?Attendee Responses:Yes – Pass/Fail – 49%Yes - Yes - Multiple categories, e.g. A,B,C,D,F – 39%No – 11%
    8. 8. Poll #2How familiar are you with Item ResponseTheory?Attendee Responses:Very – I understand and routinely apply IRT formulas – 37%Somewhat – I understand the logic and concepts – 38%A little – I have heard of it – 20%Not at all – I have never heard of it – 5%
    9. 9. Poll #3What is your primary job function?Attendee Responses:Teacher or Content Expert -6%Item Writer – 8%Psychometrician – 30%Manager and I am a non Psychometrician – 35%Manager and I am a Psychometrician – 21%
    10. 10. Usual ApproachPopulation Distribution
    11. 11. Usual ApproachPopulation Distribution
    12. 12. New ThinkingProbability of being a Masteror a Non-Master0.00.10.20.30.40.50.60.70.80.91.0Non-Master Master
    13. 13. A Different QuestionOld: Your score was 76 which is above thepassing score of 72. You passed.vsNew: Probability of this response pattern for amaster is 85% and the probability for a non-master is 15%. You passed.
    14. 14. IRT ApproachProbability of a correct response to Question 123 given ability levelQuestion 1230.00.10.20.30.40.50.60.70.80.91.0-3 -2 -1 0 1 2 3
    15. 15. 0.00.10.20.30.40.50.60.70.80.91.0Non-Master MasterNew ThinkingProbability of a correct response to Question 123for Masters and Non-MastersQuestion 123
    16. 16. Advantages• Simple framework• Small number of items• Small calibration sample sizes• Classifies as well as or betterthan IRT• Effective for adaptive testing• Well developed science
    17. 17. Applications• Intelligent Tutoring Systems• Diagnostic Testing• Personality Assessment• Automated Essay Scoring• Certification Examinations• End-of-course examinations
    18. 18. Examples
    19. 19. A Certification Examination
    20. 20. MDT
    21. 21. Logic
    22. 22. Notation• K - # of mastery states• P(mk) - Prob of a randomly drawn examinee beingin each mastery state k• z - an individual’s response vector z1,z2,…,zNzi ∈ (0,1) for N questions
    23. 23. WantP(mk | z )The probability of each mastery state k, mk, given theresponse vector z.The probability of being a master given zThe probability of being a non-master given z
    24. 24. Do you recognize these people?
    25. 25. Bayes Theorem• P(a|b)*P(b) = P(b|a)*P(a)k k kP(m | ) P( )= P( |m ) P(m )cz z z
    26. 26. Mastery state(using Bayes Theorem)P(m | ) = P( |m ) P(m )k k kz zcBut there are too many possible responsevectors z
    27. 27. Mastery state(using Bayes Theorem)P(m | ) = P( |m ) P(m )k k kz zcBut there are too many possible responsevectors zP( |m ) = P(z | m )k i ki=1NzSimplifying assumption
    28. 28. Basic ConceptConditional probabilities of a correct response,P(zi=1|mk)Item 1 Item 2 Item 3Masters (m1) .8 .8 .6Non-masters (m2) .3 .6 .5Response Vector [1,1,0]
    29. 29. Probability of the response vector z for eachmastery state is:P(z| m1) =.8 * .8 * (1-.6) = .26Conditional probabilities of a correct response, P(zi=1|mk)Item 1 Item 2 Item 3Masters (m1) .8 .8 .6Non-masters (m2) .3 .6 .5Response Vector [1,1,0]Examinee 1
    30. 30. Probability of the response vector z for eachmastery state is:P(z| m1) =.8 * .8 * (1-.6) = .26P(z| m2) =.3 * .6 * (1-.5) = .09Conditional probabilities of a correct response, P(zi=1|mk)Item 1 Item 2 Item 3Masters (m1) .8 .8 .6Non-masters (m2) .3 .6 .5Response Vector [1,1,0]Examinee 1
    31. 31. Probability of the response vector z for eachmastery state is:P(z| m1) =.8 * .8 * (1-.6) = .26P(z| m2) =.3 * .6 * (1-.5) = .09NormalizedP(z| m1) = .26 / (.26 + .09) = .74P(z| m2) = .09 / (.26 + .09) = .26Conditional probabilities of a correct response, P(zi=1|mk)Item 1 Item 2 Item 3Masters (m1) .8 .8 .6Non-masters (m2) .3 .6 .5Response Vector [1,1,0]Examinee 1
    32. 32. Probability of the response vector z for eachmastery state is:P(z| m1) =.2 * .2 * .6 = .024P(z| m2) =.7 * .4 * .5 = .14Conditional probabilities of a correct response, P(zi=1|mk)Item 1 Item 2 Item 3Masters (m1) .8 .8 .6Non-masters (m2) .3 .6 .5Response Vector [0,0,1]Examinee 2
    33. 33. Probability of the response vector z for eachmastery state is:P(z| m1) =.2 * .2 * .6 = .024P(z| m2) =.7 * .4 * .5 = .14NormalizedP(z| m1) = .024 / (.024 + .14) = .15P(z| m2) = .14 / (.024 + .14) = .85Conditional probabilities of a correct response, P(zi=1|mk)Item 1 Item 2 Item 3Masters (m1) .8 .8 .6Non-masters (m2) .3 .6 .5Response Vector [0,0,1]Examinee 2
    34. 34. Conditional probabilities of a correct response,P(zi=1|mk)Item 1 Item 2 Item 3Masters (m1) .8 .8 .6Non-masters (m2) .3 .6 .5Response Vector [1,0,1]Poll1. Master2. Non-masterCheck YourselfExaminee 3
    35. 35. Probability of the response vector z for each masterystate is:P(z| m1) =.8 * (1-.8) * .6 = .096P(z| m2) =.3 * (1-.6) * .5 = .06NormalizedP(z| m1) = .096 / (.096 + .06) = .62P(z| m2) = .06 / (.096 + .06) = .38Response Vector [1,0,1]Conditional probabilities of a correct response, P(zi=1|mk)Item 1 Item 2 Item 3Masters (m1) .8 .8 .6Non-masters (m2) .3 .6 .5Examinee 3
    36. 36. Decision Criteria
    37. 37. Decision Rule – Maximum Likelihood00.050.10.150.20.250.3P(z|mk)MasterNon-Master• Probability of the response vector, z, for each mastery state is:P(z| m1) = .8 * .8 * (1-.6) = .26P(z| m2) = .3 * .6 * (1-.5) = .09
    38. 38. Decision Rule - Maximum a posterioriprobability• Probability of each mastery state isP(m1|z) = c * .26 *.7 = c* .52 = .87P(m2|z) = c * .09 *.3 = c* .08 = .1300.10.20.30.40.50.60.70.80.9P(mk|z)MasterNon-Master
    39. 39. Decision CriteriaBayes RiskGiven a set of itemresponses z and thecosts associatedwith eachdecision, select dk tominimize the totalexpected cost.
    40. 40. Tools
    41. 41. Tools and Resourceshttp://edres.org/mdt• Paper• Java Applet• Download Excel tool• Tools for– Data Generation– Item Calibration– Scoring– CAT simulation (in progress)
    42. 42. http://bit.ly/pareonline
    43. 43. Example
    44. 44. Adaptive Testing
    45. 45. 1. Sequentially select items to maximizecertainty,2. Administer and score item,3. Update the estimated mastery stateclassification probabilities,4. Evaluate whether there is enough informationto terminate testing,5. Back to Step 1 if needed.Sequential Testing
    46. 46. Claude Shannon
    47. 47. EntropyA measure of the disorder of a system.How many bits of information are needed to senda) 1,000,000 random signalsb) 1,000,000 zero’sH S p pkkKk( ) log12
    48. 48. Less peaked = more uncertainty= more entropy0.00.20.40.60.81.0Non-Master Master0.00.20.40.60.81.0Non-Master MasterH(s) = 1.00H(s) = 0.72
    49. 49. Adaptive Testing0.20.40.60.810 5 10 15 20 25 30 35 40 45 50Max No of itemsProportionAccuracyClassifiedPercent classified vs accuracy as a function of themaximum number of items administered (NAEP items)
    50. 50. Recap• Simple framework• Small number of items• Classifies as well as or better thanmuch more complicated IRT• Effective for adaptive testing• Small sample sizes• Well developed science
    51. 51. Option For• Small certification programs• Large certification programs• Embedded in instructional systems• Test preparation
    52. 52. HANDBOOK OF TEST SECURITY• Editors - James Wollack & John Fremer• Published March 2013• Preventing, Detecting, and Investigating Cheating• Testing in Many Domains– Certification/Licensure– Clinical– Educational– Industrial/Organizational• Don’t forget to order your copy at www.routledge.com– http://bit.ly/HandbookTS (Case Sensitive)– Save 20% - Enter discount code: HYJ82
    53. 53. Questions?Please type questions for our presenters in theGoToWebinar control panel on your screen
    54. 54. THANK YOU!- Follow Caveon on twitter @caveon- Check out our blog…www.caveon.com/blog- LinkedIn Group – “Caveon Test Security”Lawrence M. Rudner, Ph.D. MBAVice President and Chief PsychometricianResearch and DevelopmentGMAC®Jamie Mulkey, Ed.D.Vice President and General ManagerTest Development ServicesCaveon
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×