Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Confidence based marking: assessing with confidence

626 views

Published on

Jon Rosewell from The Open University gave a presentation about Confidence based marking as part of the online events by expert pool Assessment within EMPOWER.

Published in: Education
  • Be the first to comment

  • Be the first to like this

Confidence based marking: assessing with confidence

  1. 1. Assessing with confidence Jon Rosewell
  2. 2. Confidence-based question
  3. 3. Why ask students about certainty? Ask students: ● What do you know? ● How certain is your knowledge? Wikipedia: https://en.wikipedia.org/wiki/Confidence-based_learning How can we get students to honestly report their certainty?
  4. 4. Confidence ≡ certainty Confidence-based marking (CBM) NB terminology: ‘certainty’ would be better, but ‘confidence’ has stuck Confidence Score Correct Wrong Low 1 0 Medium 2 -2 High 3 -6 Tentative & correct Confidently correct Cocksure – and wrong! Gardner-Medwin & Curtin (2007) Certainty-Based Marking (CBM) for Reflective Learning and Proper Knowledge Assessment [http://www.ucl.ac.uk/lapt/REAP_cbm.pdf]
  5. 5. Some trial questions Question Taken by Mean score What is 2 + 2? 238 2.51 What is derivative of x³? 223 -0.47 Who painted the 'Mona Lisa'? 212 2.14 Who is the 'Mona Lisa'? 208 0.21 Uncertainty principle -- whose? 207 0.03 Uncertainty principle -- formula? 218 0.01 Easy Difficult Tricky!
  6. 6. Do students honestly assess confidence? Question High Medium Low What is 2 + 2? 210 5 23 What is derivative of x³? 96 46 81 Who painted the 'Mona Lisa'? 165 21 26 Who is the 'Mona Lisa'? 56 41 111 Uncertainty principle -- whose? 62 31 114 Uncertainty principle -- formula? 39 21 158
  7. 7. Medical students CBM – for learning and revision Moodle: https://docs.moodle.org/30/en/Using_certainty-based_marking
  8. 8. ‘CB bonus’
  9. 9. Under exam conditions…
  10. 10. Benefits to students CBM – motivations ● Rewards care and effort ● Greater engagement ● Encourages reflective learning ● Encourages self-assessment Thinkstock.com: 483452995
  11. 11. Do students like CBM? Yes – regard it as fair and challenging, helpful to learning and No – less likely to do CBM than MCQ when optional Schoendorfer, N., & Emmett, D. (2012). Use of certainty-based marking in a second-year medical student cohort: a pilot study. Advances in Medical Education and Practice, 3, 139–43. doi:10.2147/AMEP.S35972 Nix, I., & Wyllie, A. (2011). Exploring design features to enhance computer-based assessment: Learners’ views on using a confidence-indicator tool and computer-based feedback. British Journal of Educational Technology, 42(1), 101–112. doi:10.1111/j.1467-8535.2009.00992.x Barr, D. A., & Burke, J. R. (2013). Using confidence-based marking in a laboratory setting: A tool for student self-assessment and learning. The Journal of Chiropractic Education, 27(1), 21–26. doi:10.7899/JCE-12-018
  12. 12. Is CBM fair? ● No significant gender differences ● Very few students seem over-confident, but some were under-confident ‘In decision-rich occupations such as medicine, mis-calibration of reliability is a serious handicap’ Gardner-Medwin (2014, p.6) ● Scores will generally be lower when marked as CBM than MCQ but possible to scale to non-CBM marking to set grade boundaries Gardner-Medwin, A. R., & Gahan, M. (2003). Formative and Summative Confidence-Based Assessment. In Proc. 7th International Computer-Aided Assessment Conference (pp. 147–155). Retrieved from www.caaconference.com Gardner-Medwin, T. (2014). CBM selftests at UCL: The past and the future of LAPT. Retrieved from http://www.tmedwin.net/~ucgbarg/tea/SLMS2014_A4.pdf
  13. 13. Types of questions Open question Multiple-choice question Confidence-based
  14. 14. What is wrong with MCQ? MCQ Pros:  Objective marking  Reliable marking  Easy to implement Cons:  Distractors trivial  May engender misconceptions  Working backwards Open question Pros:  Numeric easy to mark  Tests deeper learning  Can find misconceptions Cons:  Free text still difficult to mark reliably
  15. 15. ‘Open’ CBM
  16. 16. ‘Open’ CBM – what benefit? MCQ Pros:  Objective marking  Reliable marking  Easy to implement Cons:  Distractors trivial  May engender misconceptions  Working backwards Open CBM Pros:  Open question  Reflection  …as MCQ Cons:  Not always applicable  Intimidating?  Personality dependent?
  17. 17. 17 www.indegene.com/lifesciences/ipace ipace ● ‘Global training partner to pharma companies’ ● Need to establish that reps are properly trained for compliance ● Platform that delivers: ● Regular questions from a bank ● CBM assessment ● Mastery = all questions correctly answered more than once
  18. 18. 18
  19. 19. 19
  20. 20. 20
  21. 21. 21
  22. 22. 22
  23. 23. 23
  24. 24. 24
  25. 25. Future of CBM? Pros: ● Fit with competency and mastery assessment is a good one ● Use in formative / revision contexts avoids issues to do with unconventional marking and assigning grade boundaries Cons: ● Dislike of negative marking ● Poor platform support – but improving ● Difficulty of marking more complex question types
  26. 26. 26

×