Caveon Webinar Series: Using Decision Theory for Accurate Pass/Fail Decisions
Upcoming SlideShare
Loading in...5
×
 

Caveon Webinar Series: Using Decision Theory for Accurate Pass/Fail Decisions

on

  • 374 views

The traditional approach to classification testing is extremely inefficient and often difficult to implement in applied settings. Typically, examinees are rank ordered either through Item Response ...

The traditional approach to classification testing is extremely inefficient and often difficult to implement in applied settings. Typically, examinees are rank ordered either through Item Response Theory or Classical Test Theory, and then scores are compared to a difficult-to-define cut score.

This webinar will introduce the use of decision theory which basically asks: “Does this response pattern look like the response pattern of a master or a non-master?” This simpler model has major advantages over IRT and CTT:

1. Only a small sample of clear masters and a small sample of clear non-masters are needed to calibrate questions.
2. There are no assumptions for unidimensionality, and normal distribution or requirement for monotonically increasing probabilities of correct responses.

This model is attractive and a natural for end-of-unit examinations, adaptive testing, and as the routing mechanism for intelligent tutoring systems.

This webinar will explain the model, identify current applications, and introduce free tools for generating, calibrating and scoring data.

Statistics

Views

Total Views
374
Views on SlideShare
374
Embed Views
0

Actions

Likes
0
Downloads
17
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Are you involved with any classification tests as part of your work?Yes – Pass/FailYes – Multiple categories, e.g. A,B,C,D,FNo
  • Are you involved with any classification tests as part of your work?Yes – Pass/FailYes – Multiple categories, e.g. A,B,C,D,FNo
  • Are you involved with any classification tests as part of your work?Yes – Pass/FailYes – Multiple categories, e.g. A,B,C,D,FNo
  • Abraham Wald (October 31, 1902(1902-10-31) - December 13, 1950) was a mathematician born in Cluj, in the then Austria–Hungary (present-day Romania) who contributed to decision theory, geometry, and econometrics, and founded the field of statisticalsequential analysis.[1]was thus home-schooled by his parents until college.[1] His parents were quite knowledgeable and competent as teachers.[2]Emigrated to US to avoid the nazi’sThomas Bayes (pronounced: ˈbeɪz) (c. 1702 – 17 April 1761) was an Englishmathematician and Presbyterian minister, known for having formulated a specific case of the theorem that bears his name: Bayes' theorem, which was published posthumously.
  • Shannon is famous for having founded information theory with a landmark paper that he published in 1948. However, he is also credited with founding both digital computer and digital circuit design theory in 1937, when, as a 21-year-old master's degree student at the Massachusetts Institute of Technology (MIT), he wrote his thesisdemonstrating that electrical applications of boolean algebra could construct and resolve any logical, numerical relationship. It has been claimed that this was the most important master's thesis of all time.[3] Shannon contributed to the field of cryptanalysis for national defense during World War II, including his basic work on codebreaking and secure telecommunications.

Caveon Webinar Series: Using Decision Theory for Accurate Pass/Fail Decisions Caveon Webinar Series: Using Decision Theory for Accurate Pass/Fail Decisions Presentation Transcript

  • Upcoming Caveon Events• Caveon Webinar Series: Next session, June 19Protecting your Tests Using Copyright Law• Presenters include Intellectual Property Attorney Kenneth Horton and amember of the Caveon Web Patrol team• Register at: http://bit.ly/protectingip• NCSA – June 19-21 National Harbor, MD– Dr. John Fremer is co-presenting Preventing, Detecting, and Investigating TestSecurity Irregularities: A Comprehensive Guidebook On Test Security For States– Visit the Caveon booth!
  • Latest Publications• Handbook of Test Security – Now available forpurchase! We’ll share a discount code before endof session.• TILSA Guidebook for State AssessmentDirectors on Data Forensics – coming soon!
  • Caveon Online• Caveon Security Insights Blog– http://www.caveon.com/blog/• twitter– Follow @Caveon• LinkedIn– Caveon Company Page– “Caveon Test Security” Group• Please contribute!• Facebook– Will you be our “friend?”– “Like” us!www.caveon.com
  • “Using Decision Theory to ScoreAccurate Pass/Fail Decisions”Lawrence M. Rudner, Ph.D., MBAVice President and Chief PsychometricianResearch and DevelopmentGMAC®May 15, 2013Caveon Webinar Series:Jamie Mulkey, Ed.D.Vice President and General ManagerTest Development ServicesCaveon
  • Agenda for today• Role of decision theory• Examples• Logic• Tools• Adaptive Testing
  • Goal of Measurement Decision TheoryClassify an examinee into one of K groups– mastery/non-master– below basic / basic / proficient / advanced– A / B / C / D / F
  • Poll #1Are you involved with any classificationtests as part of your work?Attendee Responses:Yes – Pass/Fail – 49%Yes - Yes - Multiple categories, e.g. A,B,C,D,F – 39%No – 11%
  • Poll #2How familiar are you with Item ResponseTheory?Attendee Responses:Very – I understand and routinely apply IRT formulas – 37%Somewhat – I understand the logic and concepts – 38%A little – I have heard of it – 20%Not at all – I have never heard of it – 5%
  • Poll #3What is your primary job function?Attendee Responses:Teacher or Content Expert -6%Item Writer – 8%Psychometrician – 30%Manager and I am a non Psychometrician – 35%Manager and I am a Psychometrician – 21%
  • Usual ApproachPopulation Distribution
  • Usual ApproachPopulation Distribution
  • New ThinkingProbability of being a Masteror a Non-Master0.00.10.20.30.40.50.60.70.80.91.0Non-Master Master
  • A Different QuestionOld: Your score was 76 which is above thepassing score of 72. You passed.vsNew: Probability of this response pattern for amaster is 85% and the probability for a non-master is 15%. You passed.
  • IRT ApproachProbability of a correct response to Question 123 given ability levelQuestion 1230.00.10.20.30.40.50.60.70.80.91.0-3 -2 -1 0 1 2 3
  • 0.00.10.20.30.40.50.60.70.80.91.0Non-Master MasterNew ThinkingProbability of a correct response to Question 123for Masters and Non-MastersQuestion 123
  • Advantages• Simple framework• Small number of items• Small calibration sample sizes• Classifies as well as or betterthan IRT• Effective for adaptive testing• Well developed science
  • Applications• Intelligent Tutoring Systems• Diagnostic Testing• Personality Assessment• Automated Essay Scoring• Certification Examinations• End-of-course examinations
  • Examples
  • A Certification Examination
  • MDT
  • Logic
  • Notation• K - # of mastery states• P(mk) - Prob of a randomly drawn examinee beingin each mastery state k• z - an individual’s response vector z1,z2,…,zNzi ∈ (0,1) for N questions
  • WantP(mk | z )The probability of each mastery state k, mk, given theresponse vector z.The probability of being a master given zThe probability of being a non-master given z
  • Do you recognize these people?
  • Bayes Theorem• P(a|b)*P(b) = P(b|a)*P(a)k k kP(m | ) P( )= P( |m ) P(m )cz z z
  • Mastery state(using Bayes Theorem)P(m | ) = P( |m ) P(m )k k kz zcBut there are too many possible responsevectors z
  • Mastery state(using Bayes Theorem)P(m | ) = P( |m ) P(m )k k kz zcBut there are too many possible responsevectors zP( |m ) = P(z | m )k i ki=1NzSimplifying assumption
  • Basic ConceptConditional probabilities of a correct response,P(zi=1|mk)Item 1 Item 2 Item 3Masters (m1) .8 .8 .6Non-masters (m2) .3 .6 .5Response Vector [1,1,0]
  • Probability of the response vector z for eachmastery state is:P(z| m1) =.8 * .8 * (1-.6) = .26Conditional probabilities of a correct response, P(zi=1|mk)Item 1 Item 2 Item 3Masters (m1) .8 .8 .6Non-masters (m2) .3 .6 .5Response Vector [1,1,0]Examinee 1
  • Probability of the response vector z for eachmastery state is:P(z| m1) =.8 * .8 * (1-.6) = .26P(z| m2) =.3 * .6 * (1-.5) = .09Conditional probabilities of a correct response, P(zi=1|mk)Item 1 Item 2 Item 3Masters (m1) .8 .8 .6Non-masters (m2) .3 .6 .5Response Vector [1,1,0]Examinee 1
  • Probability of the response vector z for eachmastery state is:P(z| m1) =.8 * .8 * (1-.6) = .26P(z| m2) =.3 * .6 * (1-.5) = .09NormalizedP(z| m1) = .26 / (.26 + .09) = .74P(z| m2) = .09 / (.26 + .09) = .26Conditional probabilities of a correct response, P(zi=1|mk)Item 1 Item 2 Item 3Masters (m1) .8 .8 .6Non-masters (m2) .3 .6 .5Response Vector [1,1,0]Examinee 1
  • Probability of the response vector z for eachmastery state is:P(z| m1) =.2 * .2 * .6 = .024P(z| m2) =.7 * .4 * .5 = .14Conditional probabilities of a correct response, P(zi=1|mk)Item 1 Item 2 Item 3Masters (m1) .8 .8 .6Non-masters (m2) .3 .6 .5Response Vector [0,0,1]Examinee 2
  • Probability of the response vector z for eachmastery state is:P(z| m1) =.2 * .2 * .6 = .024P(z| m2) =.7 * .4 * .5 = .14NormalizedP(z| m1) = .024 / (.024 + .14) = .15P(z| m2) = .14 / (.024 + .14) = .85Conditional probabilities of a correct response, P(zi=1|mk)Item 1 Item 2 Item 3Masters (m1) .8 .8 .6Non-masters (m2) .3 .6 .5Response Vector [0,0,1]Examinee 2
  • Conditional probabilities of a correct response,P(zi=1|mk)Item 1 Item 2 Item 3Masters (m1) .8 .8 .6Non-masters (m2) .3 .6 .5Response Vector [1,0,1]Poll1. Master2. Non-masterCheck YourselfExaminee 3
  • Probability of the response vector z for each masterystate is:P(z| m1) =.8 * (1-.8) * .6 = .096P(z| m2) =.3 * (1-.6) * .5 = .06NormalizedP(z| m1) = .096 / (.096 + .06) = .62P(z| m2) = .06 / (.096 + .06) = .38Response Vector [1,0,1]Conditional probabilities of a correct response, P(zi=1|mk)Item 1 Item 2 Item 3Masters (m1) .8 .8 .6Non-masters (m2) .3 .6 .5Examinee 3
  • Decision Criteria
  • Decision Rule – Maximum Likelihood00.050.10.150.20.250.3P(z|mk)MasterNon-Master• Probability of the response vector, z, for each mastery state is:P(z| m1) = .8 * .8 * (1-.6) = .26P(z| m2) = .3 * .6 * (1-.5) = .09
  • Decision Rule - Maximum a posterioriprobability• Probability of each mastery state isP(m1|z) = c * .26 *.7 = c* .52 = .87P(m2|z) = c * .09 *.3 = c* .08 = .1300.10.20.30.40.50.60.70.80.9P(mk|z)MasterNon-Master
  • Decision CriteriaBayes RiskGiven a set of itemresponses z and thecosts associatedwith eachdecision, select dk tominimize the totalexpected cost.
  • Tools
  • Tools and Resourceshttp://edres.org/mdt• Paper• Java Applet• Download Excel tool• Tools for– Data Generation– Item Calibration– Scoring– CAT simulation (in progress)
  • http://bit.ly/pareonline
  • Example
  • Adaptive Testing
  • 1. Sequentially select items to maximizecertainty,2. Administer and score item,3. Update the estimated mastery stateclassification probabilities,4. Evaluate whether there is enough informationto terminate testing,5. Back to Step 1 if needed.Sequential Testing
  • Claude Shannon
  • EntropyA measure of the disorder of a system.How many bits of information are needed to senda) 1,000,000 random signalsb) 1,000,000 zero’sH S p pkkKk( ) log12
  • Less peaked = more uncertainty= more entropy0.00.20.40.60.81.0Non-Master Master0.00.20.40.60.81.0Non-Master MasterH(s) = 1.00H(s) = 0.72
  • Adaptive Testing0.20.40.60.810 5 10 15 20 25 30 35 40 45 50Max No of itemsProportionAccuracyClassifiedPercent classified vs accuracy as a function of themaximum number of items administered (NAEP items)
  • Recap• Simple framework• Small number of items• Classifies as well as or better thanmuch more complicated IRT• Effective for adaptive testing• Small sample sizes• Well developed science
  • Option For• Small certification programs• Large certification programs• Embedded in instructional systems• Test preparation
  • HANDBOOK OF TEST SECURITY• Editors - James Wollack & John Fremer• Published March 2013• Preventing, Detecting, and Investigating Cheating• Testing in Many Domains– Certification/Licensure– Clinical– Educational– Industrial/Organizational• Don’t forget to order your copy at www.routledge.com– http://bit.ly/HandbookTS (Case Sensitive)– Save 20% - Enter discount code: HYJ82
  • Questions?Please type questions for our presenters in theGoToWebinar control panel on your screen
  • THANK YOU!- Follow Caveon on twitter @caveon- Check out our blog…www.caveon.com/blog- LinkedIn Group – “Caveon Test Security”Lawrence M. Rudner, Ph.D. MBAVice President and Chief PsychometricianResearch and DevelopmentGMAC®Jamie Mulkey, Ed.D.Vice President and General ManagerTest Development ServicesCaveon