Caveon Webinar Series - Discrete Option Multiple Choice April 2014


Published on

It's not news that those of us in the testing world struggle with a variety of challenges. What's one of the largest? Test security - which is not likely to go away anytime soon. But, we also have measurement issues as our tests sometimes measure more than we want them to, and may not be as fair to certain subgroups in the population. In addition, costs to develop and administer tests have gone up with the use of technology, a fact opposite to how technology use has generally affected other industries. Finally, testing critics seem to be getting bolder in their claims, some of which are uncomfortably close to accurate.

This webinar looks at how, through technology, changing the traditional multiple choice question can help substantially with these challenges and others. You won't want to miss this webinar, hosted by Caveon CEO David Foster. Dr. Foster will present information from his recently published white paper Compelling Test Design Technology: Discrete Option Multiple Choice.

Published in: Education, Business, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Caveon Webinar Series - Discrete Option Multiple Choice April 2014

  1. 1. Caveon Webinar Series: Solving Big Problems with DOMC, a 21st Century Technology April 23, 2014 Dr. David Foster, CEO Caveon Test Security
  2. 2. Agenda for Today • What is DOMC? • DOMC effects on testing problems • Creating DOMC items • SUPER DOMC • Challenges with DOMC 2
  3. 3. Major Problems We Face Security is abysmal Quality of measurement can be better Costs are too high Every test is unfair to someone 3
  4. 4. DOMC can help… What is DOMC? Discrete Option Multiple Choice Simplest Description: Options for the multiple choice question are presented randomly one at a time with YES and NO buttons. Options stop appearing when the test taker answers the question correctly or incorrectly. 4
  5. 5. DOMC Examples The best way to understand DOMC is to try it out…going to We are going to look at some varieties: • simple multiple choice • multiple choice with more than one correct answer • multiple choice scored correct with responses to distractors only • DOMC with larger option pools (SUPER DOMC) 5
  6. 6. How can DOMC help solve these problems? 6
  7. 7. Major Problem #1: Tests Are Stolen and Used to Cheat • Tests are stolen routinely and shared quickly on the Internet. • New technologies (e.g., Google Glass) will be very effective. • Cheating by using pre-knowledge of questions is impossible to detect by proctors. • Old fashioned cheating remains a threat. • Security problems directly affect the test score and society’s trust in those scores. 7
  8. 8. Security Solution: How Does DOMC Help? • DOMC exposes much less of the item content • Makes it more difficult to steal accurate questions – SCIENTIFIC RESEARCH: • 2.5 options on the average are presented compared to 4 or 5 for the traditional MC item. • About a 50% reduction in the exposure of critical content. – Absolutely NO MEASUREMENT LOSS – SUPER DOMC reduces the exposure more dramatically (more on this later) 8
  9. 9. How Does Reduced Exposure Improve Test Security? • You can’t capture what you can’t see! • You can’t share what you didn’t see! • If you want to cheat, you can’t depend on what others tell you about the test! 9
  10. 10. RESEARCH: College Students 10
  11. 11. Expert Opinion “DOMC testing may also considerably reduce the danger of unwanted copying and sharing of test items because many answer options do not have to be presented at all.” Sonja Willing, Martin Ostapczuk, and Jochen Musch, University of Dusseldorf, in Advances in Health Science Education, 2014 11
  12. 12. Major Problem #2: Quality of a score from multiple choice tests is less than it should be What would happen to the quality of a cake if you added 10% of some totally unrelated ingredient? Chili Peppers? Vinegar? Asphalt? • Traditional multiple choice questions are like this cake. 12
  13. 13. Major Problem #2: Quality of a score from multiple choice tests is less than it should be • SCIENTIFIC RESEARCH: TESTWISENESS adds about 10% to 15% to a person’s score. • This is a large portion of a score (and time/money) devoted to measuring an irrelevant skill! • It is CRITICAL that we remove as much TESTWISENESS as possible if we want to more accurately measure what is important. 13
  14. 14. Measurement Solution: How Does DOMC Help? • SCIENTIFIC RESEARCH: – DOMC reduces TESTWISENESS effect • Test takers cannot compare options, a major tactic of TESTWISENESS • Use of TESTWISENESS is also a FAIRNESS issue. (more on this later) 14
  15. 15. Windfall Test takers soon realize that “being prepared” means learning the subject matter of the test, not learning how to game the question format. 15
  16. 16. RESEARCH: Student Reactions College students prefer the traditional MC format… …because DOMC removes test taking cues Of course, they would also prefer not taking tests at all, not going to the dentist, and not having their ID’s checked. 16
  17. 17. Major Problem #3: Tests cost too much to create and deliver  Programs create large numbers of items for security reasons (to foil thieves).  The cost to administer a computerized test is higher than it should be, in my opinion. This may not be a major concern to most programs, but who wouldn’t like more cash to spend? 17
  18. 18. Cost Solution: How Can DOMC Help? In at least two ways: 1. DOMC test questions remain in service, uncompromised, for much longer, perhaps indefinitely. Test development costs are cut significantly. 1. SCIENTIFIC RESEARCH: Testing seat time is reduced by 10% to 39%. 1. Less seat time means less money paid for test administration. 2. Opportunity for greater testing volumes. 18
  19. 19. Major Problem #4: Fairness FAIRNESS Facts:  The testing industry is getting hammered by critics—justifiably so in some cases 1. We allow TESTWISENESS in multiple choice tests 2. Our multiple choice tests are unfair to some sub- groups  These are, in my opinion, accurate criticisms  The most NOBLE thing we can do is make our tests as fair as possible. 19
  20. 20. How Does DOMC Help with Unfairness? Removes Some Inequality of Test Prep A major goal of test prep courses is to teach tactics to best answer multiple choice questions (TESTWISENESS). Test preparation courses costs money and can’t be afforded by many test takers, who are disadvantaged as a result. DOMC removes the need to take test prep course to learn test taking skills. 20
  21. 21. How Else Is DOMC More Fair? It Is Simple to Do! DOMC format is cognitively simpler making it easier for all, including: – young children – older adults – those with test anxiety – those where the language of the test is not their native language – those where the culture does not include MC tests – many others 21
  22. 22. Creating DOMC Items • Item banking and test administration systems need to be modified for DOMC, just as they need to be changed for any new item type. • Kryterion has supported a simple version of DOMC. • Excelsoft has built the more complete version of DOMC. • Here is a screen from their system, SARAS. 22
  23. 23. SARAS Test Development Screen #1: Stem 23
  24. 24. SARAS Test Development Screen #2: Options 24
  25. 25. SARAS Test Development Screen #3: Scoring/Stopping Rules 25
  26. 26. SUPER DOMC Item Definition: A SUPER DOMC item has more correct and incorrect options than would be normal for the traditional version. For example, having 3 correct and 7 incorrect options would enhance the advantages described in these slides. It is not always easy to do to add options, but almost always possible. 26
  27. 27. Example: SUPER DOMC Item Objective: Add two 2-digit numbers. MC Stem: Is this addition correct? Correct Options look like this: 12 + 79 = 91 There are hundreds of possible correct options Incorrect Options look like this: 33 + 17 = 40 There are thousands of possible incorrect options 27
  28. 28. Challenges with DOMC • Most test takers don’t like it. • Takes only a little bit of practice. • Item banking systems and test administration systems have to be modified. • Test administration must use a “forward-only” design (e.g., no re-viewing items). 28
  29. 29. To Learn More… There is no charge to the licensee to try out DOMC or for doing research. Caveon will provide you will the help you need to get started. Link to DOMC white paper – If questions, email 29
  30. 30. Questions? 30
  31. 31. Thank you! Dave Foster, CEO Caveon Test Security Follow Caveon on twitter @caveon Check out our blog LinkedIn Group “Caveon Test Security” 31