Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Assessing with confidence
Jon Rosewell
Confidence-based question
Why ask students about certainty?
Ask students:
● What do you know?
● How certain is your knowledge?
Wikipedia: https://en...
Confidence ≡ certainty
Confidence-based marking (CBM)
NB terminology: ‘certainty’ would be better, but ‘confidence’ has st...
Some trial questions
Question Taken by Mean score
What is 2 + 2? 238 2.51
What is derivative of x³? 223 -0.47
Who painted ...
Do students honestly assess confidence?
Question High Medium Low
What is 2 + 2? 210 5 23
What is derivative of x³? 96 46 8...
Medical students
CBM – for learning and revision
Moodle: https://docs.moodle.org/30/en/Using_certainty-based_marking
‘CB bonus’
Under exam conditions…
Benefits to students
CBM – motivations
● Rewards care and effort
● Greater engagement
● Encourages reflective learning
● E...
Do students like CBM?
Yes – regard it as fair and challenging, helpful to learning
and
No – less likely to do CBM than MCQ...
Is CBM fair?
● No significant gender differences
● Very few students seem over-confident, but some were under-confident
‘I...
Types of questions
Open question Multiple-choice question Confidence-based
What is wrong with MCQ?
MCQ
Pros:
 Objective marking
 Reliable marking
 Easy to implement
Cons:
 Distractors trivial
...
‘Open’ CBM
‘Open’ CBM – what benefit?
MCQ
Pros:
 Objective marking
 Reliable marking
 Easy to implement
Cons:
 Distractors trivia...
17
www.indegene.com/lifesciences/ipace
ipace
● ‘Global training partner to pharma companies’
● Need to establish that reps...
18
19
20
21
22
23
24
Future of CBM?
Pros:
● Fit with competency and mastery assessment is a good one
● Use in formative / revision contexts avo...
26
Upcoming SlideShare
Loading in …5
×

Assessing with confidence

250 views

Published on

Assessing with confidence

Jon Rosewell, The Open University

Confidence-based marking (CBM) is an assessment method which asks the student not only to provide the answer to a question, but also to report their level of confidence (or certainty) in the correctness of their answer. They need to consider this carefully because it affects the marks they are awarded: a student scores full marks for knowing that they know the correct answer, some credit for a tentative correct answer but are penalised if they believe they know the answer but get it wrong. There are several motivations for using CBM: it rewards care and effort so engendering greater engagement, it encourages reflective learning, and it promises accuracy and reliability.

CBM has had niche success in the past in the context of medical training and recently may have a found a new niche in the context of regulatory compliance; these are both areas where assessment of competency and mastery is expected. However, CBM has not been widely adopted in other areas of education.

In this talk I will review the CBM landscape and ask why CBM is not used more widely. What are the benefits claimed and how robust is the evidence? How should CBM be presented to the students? Do they need training to understand how the system works? Is it a fair method of assessment? Does it disadvantage any category of student? How does it fit with ideas around ‘assessment for learning’ and ‘reflective learning’?

Confidence-based marking could offer both the student and teacher greater insight into a student’s understanding than the standard fare of e-assessment, the multiple-choice quiz. It is a technique that we should therefore keep under consideration.

Published in: Education
  • Be the first to comment

  • Be the first to like this

Assessing with confidence

  1. 1. Assessing with confidence Jon Rosewell
  2. 2. Confidence-based question
  3. 3. Why ask students about certainty? Ask students: ● What do you know? ● How certain is your knowledge? Wikipedia: https://en.wikipedia.org/wiki/Confidence-based_learning How can we get students to honestly report their certainty?
  4. 4. Confidence ≡ certainty Confidence-based marking (CBM) NB terminology: ‘certainty’ would be better, but ‘confidence’ has stuck Confidence Score Correct Wrong Low 1 0 Medium 2 -2 High 3 -6 Tentative & correct Confidently correct Cocksure – and wrong! Gardner-Medwin & Curtin (2007) Certainty-Based Marking (CBM) for Reflective Learning and Proper Knowledge Assessment [http://www.ucl.ac.uk/lapt/REAP_cbm.pdf]
  5. 5. Some trial questions Question Taken by Mean score What is 2 + 2? 238 2.51 What is derivative of x³? 223 -0.47 Who painted the 'Mona Lisa'? 212 2.14 Who is the 'Mona Lisa'? 208 0.21 Uncertainty principle -- whose? 207 0.03 Uncertainty principle -- formula? 218 0.01 Easy Difficult Tricky!
  6. 6. Do students honestly assess confidence? Question High Medium Low What is 2 + 2? 210 5 23 What is derivative of x³? 96 46 81 Who painted the 'Mona Lisa'? 165 21 26 Who is the 'Mona Lisa'? 56 41 111 Uncertainty principle -- whose? 62 31 114 Uncertainty principle -- formula? 39 21 158
  7. 7. Medical students CBM – for learning and revision Moodle: https://docs.moodle.org/30/en/Using_certainty-based_marking
  8. 8. ‘CB bonus’
  9. 9. Under exam conditions…
  10. 10. Benefits to students CBM – motivations ● Rewards care and effort ● Greater engagement ● Encourages reflective learning ● Encourages self-assessment Thinkstock.com: 483452995
  11. 11. Do students like CBM? Yes – regard it as fair and challenging, helpful to learning and No – less likely to do CBM than MCQ when optional Schoendorfer, N., & Emmett, D. (2012). Use of certainty-based marking in a second-year medical student cohort: a pilot study. Advances in Medical Education and Practice, 3, 139–43. doi:10.2147/AMEP.S35972 Nix, I., & Wyllie, A. (2011). Exploring design features to enhance computer-based assessment: Learners’ views on using a confidence-indicator tool and computer-based feedback. British Journal of Educational Technology, 42(1), 101–112. doi:10.1111/j.1467-8535.2009.00992.x Barr, D. A., & Burke, J. R. (2013). Using confidence-based marking in a laboratory setting: A tool for student self-assessment and learning. The Journal of Chiropractic Education, 27(1), 21–26. doi:10.7899/JCE-12-018
  12. 12. Is CBM fair? ● No significant gender differences ● Very few students seem over-confident, but some were under-confident ‘In decision-rich occupations such as medicine, mis-calibration of reliability is a serious handicap’ Gardner-Medwin (2014, p.6) ● Scores will generally be lower when marked as CBM than MCQ but possible to scale to non-CBM marking to set grade boundaries Gardner-Medwin, A. R., & Gahan, M. (2003). Formative and Summative Confidence-Based Assessment. In Proc. 7th International Computer-Aided Assessment Conference (pp. 147–155). Retrieved from www.caaconference.com Gardner-Medwin, T. (2014). CBM selftests at UCL: The past and the future of LAPT. Retrieved from http://www.tmedwin.net/~ucgbarg/tea/SLMS2014_A4.pdf
  13. 13. Types of questions Open question Multiple-choice question Confidence-based
  14. 14. What is wrong with MCQ? MCQ Pros:  Objective marking  Reliable marking  Easy to implement Cons:  Distractors trivial  May engender misconceptions  Working backwards Open question Pros:  Numeric easy to mark  Tests deeper learning  Can find misconceptions Cons:  Free text still difficult to mark reliably
  15. 15. ‘Open’ CBM
  16. 16. ‘Open’ CBM – what benefit? MCQ Pros:  Objective marking  Reliable marking  Easy to implement Cons:  Distractors trivial  May engender misconceptions  Working backwards Open CBM Pros:  Open question  Reflection  …as MCQ Cons:  Not always applicable  Intimidating?  Personality dependent?
  17. 17. 17 www.indegene.com/lifesciences/ipace ipace ● ‘Global training partner to pharma companies’ ● Need to establish that reps are properly trained for compliance ● Platform that delivers: ● Regular questions from a bank ● CBM assessment ● Mastery = all questions correctly answered more than once
  18. 18. 18
  19. 19. 19
  20. 20. 20
  21. 21. 21
  22. 22. 22
  23. 23. 23
  24. 24. 24
  25. 25. Future of CBM? Pros: ● Fit with competency and mastery assessment is a good one ● Use in formative / revision contexts avoids issues to do with unconventional marking and assigning grade boundaries Cons: ● Dislike of negative marking ● Poor platform support – but improving ● Difficulty of marking more complex question types
  26. 26. 26

×