Opening up multiple choice - assessing with confidence

Opening up Multiple Choice: Assessing with confidence CALRG Conference, June 2011 Jon Rosewell [email_address]

Multiple Choice Questions (MCQ)
Basic fare of e-assessment:
robust
easy to implement
but pedagogically not ideal
Open questions
pedagogically preferable
but difficult to provide robust and valid marking

Demo
Standard MCQ
Standard CBM
Open CBM
screen dump

Confidence-Based Marking (CBM)
Student selects:
answer and
level of confidence/certainty
Incentive is scoring:
Confidently correct: Full marks
A tentative correct answer: Some credit
Cocksure & wrong: Penalty
Pays to be honest about own knowledge

CBM scoring Tentative & correct Confidently correct Cocksure – and wrong! Score -6 3 High -2 2 Medium 0 1 Low Wrong Correct Confidence

CBM motivations
Rewards care and effort
Greater engagement
Encourages reflective learning
Encourages self-assessment

Demo
Standard MCQ
Standard CBM
Open CBM
screen dump

Open CBM
Two stage question:
Initially:
Question (but no MCQ options)
Student sets confidence level
Next:
MCQ options revealed
Finally:
Marking and feedback as for normal MCQ / CBM

Demo
Standard MCQ
Standard CBM
Open CBM
screen dump

Open CBM advantages
Mechanically, question remains simple MCQ
answer matching is trivial
easy to implement
possible to reuse existing question banks
But transformed closed to open question
student must formulate answer to decide confidence
must decide answer without +ve or –ve clues.
cannot work backward
will not be led into misconceptions.
 must answer the question as set .

MCQ Open CBM
Pros
objective marking
reliable marking
easy to implement
Cons
distractors trivial
distractors engender misconceptions
working backwards
Pros
open question
reflection
...as MCQ
Cons
not always applicable
intimidating?
personality dependent?

Possible benefits
Students
better engagement
improved learning
more reflective learners
Staff
easier question setting

eSTEeM project plan (revised…)
Implement open CBM question type
in OU Moodle Quiz and/or OpenMark
Trial in course assessment
controlled experimental design
pilot (formative) on T216 CISCO networking
Survey/interview to probe attitudinal aspects
User-lab trials
Log measures
Assignment scores
Time spent on task

Jon Rosewell [email_address]
www.open.ac.uk

Opening up multiple choice - assessing with confidence

by Jon Rosewell

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 11

Statistics