Opening up multiple choice - assessing with confidence

872 views
717 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
872
On SlideShare
0
From Embeds
0
Number of Embeds
12
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Opening up multiple choice - assessing with confidence

  1. 1. Opening up Multiple Choice: Assessing with confidence CALRG Conference, June 2011 Jon Rosewell [email_address]
  2. 2. Multiple Choice Questions (MCQ) <ul><li>Basic fare of e-assessment: </li></ul><ul><ul><li>robust </li></ul></ul><ul><ul><li>easy to implement </li></ul></ul><ul><ul><li>but pedagogically not ideal </li></ul></ul><ul><li>Open questions </li></ul><ul><ul><li>pedagogically preferable </li></ul></ul><ul><ul><li>but difficult to provide robust and valid marking </li></ul></ul>
  3. 3. Demo <ul><li>Standard MCQ </li></ul><ul><li>Standard CBM </li></ul><ul><li>Open CBM </li></ul>screen dump
  4. 5. Confidence-Based Marking (CBM) <ul><li>Student selects: </li></ul><ul><ul><li>answer and </li></ul></ul><ul><ul><li>level of confidence/certainty </li></ul></ul><ul><li>Incentive is scoring: </li></ul><ul><ul><li>Confidently correct: Full marks </li></ul></ul><ul><ul><li>A tentative correct answer: Some credit </li></ul></ul><ul><ul><li>Cocksure & wrong: Penalty </li></ul></ul><ul><li>Pays to be honest about own knowledge </li></ul>
  5. 6. CBM scoring Tentative & correct Confidently correct Cocksure – and wrong! Score -6 3 High -2 2 Medium 0 1 Low Wrong Correct Confidence
  6. 7. CBM motivations <ul><li>Rewards care and effort </li></ul><ul><li>Greater engagement </li></ul><ul><li>Encourages reflective learning </li></ul><ul><li>Encourages self-assessment </li></ul>
  7. 8. Demo <ul><li>Standard MCQ </li></ul><ul><li>Standard CBM </li></ul><ul><li>Open CBM </li></ul>screen dump
  8. 10. Open CBM <ul><li>Two stage question: </li></ul><ul><ul><li>Initially: </li></ul></ul><ul><ul><ul><li>Question (but no MCQ options) </li></ul></ul></ul><ul><ul><ul><li>Student sets confidence level </li></ul></ul></ul><ul><ul><li>Next: </li></ul></ul><ul><ul><ul><li>MCQ options revealed </li></ul></ul></ul><ul><li>Finally: </li></ul><ul><ul><li>Marking and feedback as for normal MCQ / CBM </li></ul></ul>
  9. 11. Demo <ul><li>Standard MCQ </li></ul><ul><li>Standard CBM </li></ul><ul><li>Open CBM </li></ul>screen dump
  10. 13. Open CBM advantages <ul><li>Mechanically, question remains simple MCQ </li></ul><ul><ul><li>answer matching is trivial </li></ul></ul><ul><ul><li>easy to implement </li></ul></ul><ul><ul><li>possible to reuse existing question banks </li></ul></ul><ul><li>But transformed closed to open question </li></ul><ul><ul><li>student must formulate answer to decide confidence </li></ul></ul><ul><ul><li>must decide answer without +ve or –ve clues. </li></ul></ul><ul><ul><li>cannot work backward </li></ul></ul><ul><ul><li>will not be led into misconceptions. </li></ul></ul><ul><li> must answer the question as set . </li></ul>
  11. 14. MCQ Open CBM <ul><li>Pros </li></ul><ul><ul><li>objective marking </li></ul></ul><ul><ul><li>reliable marking </li></ul></ul><ul><ul><li>easy to implement </li></ul></ul><ul><li>Cons </li></ul><ul><ul><li>distractors trivial </li></ul></ul><ul><ul><li>distractors engender misconceptions </li></ul></ul><ul><ul><li>working backwards </li></ul></ul><ul><li>Pros </li></ul><ul><ul><li>open question </li></ul></ul><ul><ul><li>reflection </li></ul></ul><ul><ul><li>...as MCQ </li></ul></ul><ul><li>Cons </li></ul><ul><ul><li>not always applicable </li></ul></ul><ul><ul><li>intimidating? </li></ul></ul><ul><ul><li>personality dependent? </li></ul></ul>
  12. 15. Possible benefits <ul><li>Students </li></ul><ul><ul><li>better engagement </li></ul></ul><ul><ul><li>improved learning </li></ul></ul><ul><ul><li>more reflective learners </li></ul></ul><ul><li>Staff </li></ul><ul><ul><li>easier question setting </li></ul></ul>
  13. 16. eSTEeM project plan (revised…) <ul><li>Implement open CBM question type </li></ul><ul><ul><li>in OU Moodle Quiz and/or OpenMark </li></ul></ul><ul><li>Trial in course assessment </li></ul><ul><ul><li>controlled experimental design </li></ul></ul><ul><ul><li>pilot (formative) on T216 CISCO networking </li></ul></ul><ul><li>Survey/interview to probe attitudinal aspects </li></ul><ul><li>User-lab trials </li></ul><ul><li>Log measures </li></ul><ul><ul><li>Assignment scores </li></ul></ul><ul><ul><li>Time spent on task </li></ul></ul>
  14. 17. Jon Rosewell [email_address] <ul><li>www.open.ac.uk </li></ul>

×