SlideShare a Scribd company logo
Get chatting
Before the workshop starts at 14:00,
briefly introduce any previous
experiences using Moodle quizzes
•Contexts
•Positives
•Negatives
Join TEL Community of Practice on Teams
This session is being recorded
Mute Chat
tel@bath.ac.uk
Writing Effective
Multiple Choice Questions
Bertie Squire & Josh Lim
Technology Enhanced Learning (TEL) team
Instructional Designers Team
teachinghub.bath.ac.uk
Today’s Session
At the end of this workshop you will learn:
 The benefits of using MCQs both for both
assessment & learning
 How to write effective MCQs and pitfalls to avoid
 How to construct, analyze and alter MCQs
questions
 The experiences, ideas and tips of others!
Recording session & will make slides (with links) available
Please get in touch afterwards tel@bath.ac.uk for any follow-up queries
Good quiz practices = link to learning
Empower students as self-regulated learners
Activity linked to learning objectives
Aims for learners
• Impact through quality
feedback = students act on
feedback
• Bitesize, low stakes  build
motivation & self-esteem
• Combine with other learning &
teaching strategies
D. J. Nicol & D. Macfarlane‐Dick (2006)
For authors
Planning • Learning objectives + cognitive skills
• Feedback  next steps, dialogue (e.g.
tutorials, signposting, peer
support, forums, problems classes)
Writing • See following guidance...
• Continuously
Managing • Build question bank over time
• Use data to inform
• Review questions
Multiple Choice Questions
Advantages Disadvantages
• Different cognitive levels – fact
recall/apply/evaluate
• Finding “good” distractors hard
• Dominance of “recall” questions?
• “Higher order” vs. surface learning?
• Wide coverage (less “sampling”)
• Fast feedback for students
• Easy to mark/score - can apply
frequently
• Test analytics
• Different Uses (diagnostic, revision,
Assessment for/as Learning)
• Many accessible features “built-in”
• Difficult to “show working”
• Not well suited to maths
• Accuracy/suitability of assessment
• Engagement
• Students guessing
• Poor questions -> teachers work
harder than students
• Fairness/accuracy
"Non-googleable MCQs"
Learning objectives + cognitive skill = MCQ test
1. Shift focus from recall to application
– Understand (explain)
– Apply (recall in context)
– Analyse (explain in context)
– Evaluate (best fit)
2. Use novel context
3. Use common mistakes/misconceptions as
distractors
4. Multiple select (1 or 2 correct)*
5. Embrace open book exams…
Bloom’s Taxonomy by Nicoguaro is licensed under CC BY-SA
Constructing Effective Questions
Stem should be
meaningful on
its own
Don’t include
irrelevant
material
Use negative
phrasing
carefully
1
2
3
Writing Good Multiple Choice Test Questions – C. Brame (2013)
Constructing Effective Distractors
Plausible
Clear and concise
Mutually exclusive
Homogenous in nature
Free from having clues
to correct answer
All the distractors should be:
The distractors aren’t functional if they are not plausible, as students who don’t
understand the learning outcomes can still be rewarded. Use common student errors
– these make the best distractors.
Students who know the correct answer should not have to worry about unclear and
wordy answers and grasping what the answer is actually trying to say
Writing Good Multiple Choice Test Questions – C. Brame (2013)
A ‘trick’ question can sometimes lead to multiple answers being correct. Don’t have
this ambiguity – make the answers mutually exclusive
This means the answers should all be in the same topic area. If one answer is focused
on a different subject, it can lead students to the correct answer.
Savvy test takers can spot clues to correct answers such as differences in grammar,
length, formatting, and language choices.
MCQ pitfalls to avoid
1) Select the correct option
MCQs are not…
A) Unlikely to be never
ineffective
B) somewhat easy
C) Difficult to write, particularly
to make them valid
D) All of the above
2) Select the errors
demonstrated in question 1 from
the list
(Answers: 2,4,6,7,8,9)
Cognitive load
1. Superfluous information (minimise reading)
2. Unfocused question stems (avoid lots of text or open ended) and blanks
3. Unimportant content/minutia – instead test main points
4. Negative stem or answer options (if necessary, be clear to emphasise the
negative element “Which of this is NOT…”)
5. Confusingly ordered options (e.g. by size)
Wording Clues
6. Absolute terms (easy to dismiss) or vague terms (open to interpretation)
7. Grammatical clues (options deduced from stem)
Systematic
8. Unequal option length (correct answers often take more info)
9. “All/none of the above” option (e.g. trivial if 2+ match)
10. Logical clues (avoid mutually exclusive pair options, use multi select question
instead)
11. Convergent option – correct answer has similarities with others
Webb et al., 2015
Limiting student collusion
Some strategies (with collusion disincentive in brackets), listed from more extrinsic to intrinsic student motivation:
A. Police/enforce/invigilate (the “stick” approach)
B. Restrict duration (limit opportunities)
C. Don’t have marks associated with assessment – separate formative/summative assessment (remove incentive)
D. Question randomisation: mix up the order in the test or set question variants (remove ease of colluding)
E. Use confidence based marking to score students and encourage students to reflect (dissuade blind copying)
F. Follow up assessment e.g. create consequences: pick students to explain answer in tutorial, random to provide
written worked solution justification to accompany course material (create incentives to be honest & seek help)
G. Use variable/context randomisation – this could be used to promote peer collaboration/discussion (avoid simple
copying)
H. Educate and promote ethos, purpose of assessment– make it clear to students (from all backgrounds) what
collusion is, aims of assessment and disadvantages to learning (educate learners)
I. Broaden assessment aims to allow collaboration and group/peer instruction (remove incentive)
J. High intrinsic value of the activity and feedback (make motivation to collude counterproductive)
Disclaimers!
• This list isn’t exhaustive and the exact and extent of the solution(s) will be context specific (the above is not a hierarchy), but may be useful to consider!
• Many of these issues are not unique to e-Quizzes, but digitising does give some solutions!
Workshop
Activity
Improve an MCQ question
Open WORKSHOP ACTIVITY
Powerpoint
Write an improved question
• Improve question stem?
• Improve distractors?
• Shift question emphasis?
One person per group >
Feedback on what you did
Groups: 3-4
10 mins
Feel free to add text box
comments to your own slides!
MCQ Experiences Dicussion
• Think about past experiences both taking and
delivering MCQs
 What went well?
 Do you refine and revisit questions?
 Did you vary question types?
 Were then un-googleable?
 Any tips which you found helped?
 Did it help students’ understanding?
Key points
• Start with learning objectives & the focus
for the quiz
• Write effective questions!
• Review and revisit your question bank
References and further reading
• Quick read: Writing Good Multiple Choice Test Questions – C. Brame (2013)
• Good summary with links to literature: The Importance of Writing Effective Distractors – C Malamed (2020)
• Nicol, D. J & Macfarlane‐Dick, D., 2006. Formative assessment and self‐regulated learning: a model & 7
principles of good feedback practice, Studies in Higher Education 31(2) p199
• Carneson J, Delpierre G, Masters K., 2016. Handbook: Designing and Managing Multiple Choice Questions
(2nd Ed) [online only]
• Gierl, M et al., 2017. Developing, Analyzing, and Using Distractors for Multiple-Choice Tests in Education: A
Comprehensive Review. Review of Educational Research, 87: 6, pp. 1082 –1116.
• Rodriguez, M. (2005). Three Options Are Optimal for Multiple‐Choice Items: A Meta‐Analysis of 80 Years of
Research, Educational Measurement: Issues and Practice, 24:2.
• Haladyna, T.M. et al., 2002. A Review of Multiple-Choice Item-Writing Guidelines for Classroom
Assessment. Applied Measurement in Education 15(3) p309
• Webb, E et al., 2015. Does Educator Training or Experience Affect the Quality of Multiple-Choice
Questions? Academic Radiology 22(10) pp. 1317
• McCoubrie, P. 2004. Improving the fairness of multiple-choice questions: a literature review. Medical
Teacher 26(8) p709
Moodle Quiz Settings and Setup FAQ on the Teaching Hub
How to get engagement in formative
assessment and feedback?
Link to general engagement strategies: employability & real world context, develop skills, student voice/ownership, peer
assisted, student generated feedback, stakeholder engagement, social learning, diversify/integrate with other teaching
activities, educate students on benefits, combine with summative, students reflect on learning, reflect & adapt teaching strategy
….
Strategies specific to online quizzes:
• Vary quiz behaviour
• low stakes & “fast fail” approach: multiple attempts (with penalties), instant feedback
• Use social aspects as a lever with technology
• Variable/context randomisation to promote peer collaboration
• Social learning – forums, MS Teams etc to support discussion
• Student generated feedback – students as assessors (e.g. Moodle workshop)
• Vary feedback delivery
• instant timely feedback, selective feedback (reveal feedback only, hide score etc), blended feedback (in tutorial)
• Stress High intrinsic value of the activity and feedback
• Assess aspects that would be take to long to assess traditionally (e.g. wide coverage, calculations with real data set
etc).
• Timely feedback can be more effective
• Create quality bespoke feedback (this becomes scalable)
Before you start
• Guidelines for setting up
quizzes have been created
for the current assessment
period
• Read the guidance (even if
familiar with Moodle quizzes)
1. Dos and Don'ts of online
assessments PDF
2. Quiz settings Setup and FAQ
3. Creating a Quiz Assessment in
Moodle as an alternative
assessment (PDF)
Some settings will be different
from the ones you normally use
for formative quizzes e.g.
visibility of feedback
Moodle quiz settings
Step by step
instructions with
screenshots:
Creating a Quiz
Assessment in Moodle
as an alternative
assessment
(PDF)
Quiz settings (rubric,
availability, timing etc)
Extra time
Edit question set
View, amend,
export grades
Quiz settings Setup and FAQ
Setting up the Moodle quiz
Key things to remember:
• Read the guidance and follow carefully
• Use default settings unless otherwise specified
• Name your quiz clearly
• Give clear instructions to students (University Rubric,
Academic Integrity Declaration)
• Consider using grade transfer
• Adjust availability or duration for students with DAPs
• Test: Preview your quiz, check availability

More Related Content

What's hot

MEU Workshop, writing objectives, Developing objectives from competencies, li...
MEU Workshop, writing objectives, Developing objectives from competencies, li...MEU Workshop, writing objectives, Developing objectives from competencies, li...
MEU Workshop, writing objectives, Developing objectives from competencies, li...
DrGaneshPrasadSingh
 
Scientific Presentations
Scientific PresentationsScientific Presentations
Scientific Presentations
KellyHester
 
Problem Based Learning In Medical Education
Problem Based Learning In Medical EducationProblem Based Learning In Medical Education
Problem Based Learning In Medical Education
Soha Rashed
 
Report writing
Report writingReport writing
Report writing
rahulmathur
 
Research article structure
Research article structureResearch article structure
Research article structure
Ahmed Meri
 
Communication Skills Syllabus
Communication Skills SyllabusCommunication Skills Syllabus
Communication Skills Syllabus
Dr. Cupid Lucid
 
Public speaking & oral presentation
Public speaking & oral presentationPublic speaking & oral presentation
Public speaking & oral presentation
Sanjay Kumar
 
Large Group Teaching
Large Group TeachingLarge Group Teaching

What's hot (8)

MEU Workshop, writing objectives, Developing objectives from competencies, li...
MEU Workshop, writing objectives, Developing objectives from competencies, li...MEU Workshop, writing objectives, Developing objectives from competencies, li...
MEU Workshop, writing objectives, Developing objectives from competencies, li...
 
Scientific Presentations
Scientific PresentationsScientific Presentations
Scientific Presentations
 
Problem Based Learning In Medical Education
Problem Based Learning In Medical EducationProblem Based Learning In Medical Education
Problem Based Learning In Medical Education
 
Report writing
Report writingReport writing
Report writing
 
Research article structure
Research article structureResearch article structure
Research article structure
 
Communication Skills Syllabus
Communication Skills SyllabusCommunication Skills Syllabus
Communication Skills Syllabus
 
Public speaking & oral presentation
Public speaking & oral presentationPublic speaking & oral presentation
Public speaking & oral presentation
 
Large Group Teaching
Large Group TeachingLarge Group Teaching
Large Group Teaching
 

Similar to Science-Writing-Effective-MCQs-Workshop-Nov2020.pptx

App pgr workshop3
App pgr workshop3App pgr workshop3
App pgr workshop3
sarahattersley
 
Feed-forward approaches for enhancing assessment and feedback
Feed-forward approaches for enhancing assessment and feedbackFeed-forward approaches for enhancing assessment and feedback
Feed-forward approaches for enhancing assessment and feedback
jisc-elearning
 
Changing the assessment narrative
Changing the assessment narrativeChanging the assessment narrative
Changing the assessment narrative
Tansy Jessop
 
Sreedevi P.S
Sreedevi P.SSreedevi P.S
Sreedevi P.S
Sree Devi
 
Keynote Sally Jordan - Computer-based assessment friend or foe? - OWD14
Keynote Sally Jordan - Computer-based assessment friend or foe? - OWD14Keynote Sally Jordan - Computer-based assessment friend or foe? - OWD14
Keynote Sally Jordan - Computer-based assessment friend or foe? - OWD14
SURF Events
 
Why a programme view? Why TESTA?
Why a programme view? Why TESTA?Why a programme view? Why TESTA?
Why a programme view? Why TESTA?
Tansy Jessop
 
TESTA, SIAST Universities of Regina & Saskathewan Webinar (November 2013)
 TESTA, SIAST Universities of Regina & Saskathewan Webinar (November 2013) TESTA, SIAST Universities of Regina & Saskathewan Webinar (November 2013)
TESTA, SIAST Universities of Regina & Saskathewan Webinar (November 2013)
TESTA winch
 
Testing & examiner guide 2018 teacher's hand out oued semar a lgiers
Testing & examiner guide 2018  teacher's hand out  oued semar a lgiersTesting & examiner guide 2018  teacher's hand out  oued semar a lgiers
Testing & examiner guide 2018 teacher's hand out oued semar a lgiers
Mr Bounab Samir
 
Testing & bem guide 2018
Testing & bem guide 2018Testing & bem guide 2018
Testing & bem guide 2018
Mr Bounab Samir
 
Testing teacher's hand testing & examiner guide 2018
Testing teacher's hand testing & examiner guide 2018Testing teacher's hand testing & examiner guide 2018
Testing teacher's hand testing & examiner guide 2018
Mr Bounab Samir
 
MMU TESTA Keynote
MMU TESTA KeynoteMMU TESTA Keynote
MMU TESTA Keynote
Tansy Jessop
 
Testa interactive masterclass
Testa interactive masterclassTesta interactive masterclass
Testa interactive masterclass
Tansy Jessop
 
An evidence based model
An evidence based modelAn evidence based model
An evidence based model
Jisc
 
10 Innovative Formative Assessment Examples for Teachers
10 Innovative Formative Assessment Examples for Teachers10 Innovative Formative Assessment Examples for Teachers
10 Innovative Formative Assessment Examples for Teachers
Jessica Salatambos
 
Level 5 ppp assessment for learning final
Level 5 ppp assessment for learning finalLevel 5 ppp assessment for learning final
Level 5 ppp assessment for learning final
Lee Hazeldine
 
CIRTL: The gentle art of questioning
CIRTL:  The gentle art of questioningCIRTL:  The gentle art of questioning
CIRTL: The gentle art of questioning
Stephanie Chasteen
 
PGT202E - BASIC EDUCATIONAL MEASUREMENT & EVALUATION
PGT202E - BASIC EDUCATIONAL MEASUREMENT & EVALUATIONPGT202E - BASIC EDUCATIONAL MEASUREMENT & EVALUATION
PGT202E - BASIC EDUCATIONAL MEASUREMENT & EVALUATION
Waheeda Annur Sani
 
Materi mc
Materi mcMateri mc
Question, question
Question, questionQuestion, question
Question, question
kathrynlangford
 
Assessing Listening with Dr. Larry Vandergrift
Assessing Listening with Dr. Larry VandergriftAssessing Listening with Dr. Larry Vandergrift
Assessing Listening with Dr. Larry Vandergrift
Language Acquisition Resource Center
 

Similar to Science-Writing-Effective-MCQs-Workshop-Nov2020.pptx (20)

App pgr workshop3
App pgr workshop3App pgr workshop3
App pgr workshop3
 
Feed-forward approaches for enhancing assessment and feedback
Feed-forward approaches for enhancing assessment and feedbackFeed-forward approaches for enhancing assessment and feedback
Feed-forward approaches for enhancing assessment and feedback
 
Changing the assessment narrative
Changing the assessment narrativeChanging the assessment narrative
Changing the assessment narrative
 
Sreedevi P.S
Sreedevi P.SSreedevi P.S
Sreedevi P.S
 
Keynote Sally Jordan - Computer-based assessment friend or foe? - OWD14
Keynote Sally Jordan - Computer-based assessment friend or foe? - OWD14Keynote Sally Jordan - Computer-based assessment friend or foe? - OWD14
Keynote Sally Jordan - Computer-based assessment friend or foe? - OWD14
 
Why a programme view? Why TESTA?
Why a programme view? Why TESTA?Why a programme view? Why TESTA?
Why a programme view? Why TESTA?
 
TESTA, SIAST Universities of Regina & Saskathewan Webinar (November 2013)
 TESTA, SIAST Universities of Regina & Saskathewan Webinar (November 2013) TESTA, SIAST Universities of Regina & Saskathewan Webinar (November 2013)
TESTA, SIAST Universities of Regina & Saskathewan Webinar (November 2013)
 
Testing & examiner guide 2018 teacher's hand out oued semar a lgiers
Testing & examiner guide 2018  teacher's hand out  oued semar a lgiersTesting & examiner guide 2018  teacher's hand out  oued semar a lgiers
Testing & examiner guide 2018 teacher's hand out oued semar a lgiers
 
Testing & bem guide 2018
Testing & bem guide 2018Testing & bem guide 2018
Testing & bem guide 2018
 
Testing teacher's hand testing & examiner guide 2018
Testing teacher's hand testing & examiner guide 2018Testing teacher's hand testing & examiner guide 2018
Testing teacher's hand testing & examiner guide 2018
 
MMU TESTA Keynote
MMU TESTA KeynoteMMU TESTA Keynote
MMU TESTA Keynote
 
Testa interactive masterclass
Testa interactive masterclassTesta interactive masterclass
Testa interactive masterclass
 
An evidence based model
An evidence based modelAn evidence based model
An evidence based model
 
10 Innovative Formative Assessment Examples for Teachers
10 Innovative Formative Assessment Examples for Teachers10 Innovative Formative Assessment Examples for Teachers
10 Innovative Formative Assessment Examples for Teachers
 
Level 5 ppp assessment for learning final
Level 5 ppp assessment for learning finalLevel 5 ppp assessment for learning final
Level 5 ppp assessment for learning final
 
CIRTL: The gentle art of questioning
CIRTL:  The gentle art of questioningCIRTL:  The gentle art of questioning
CIRTL: The gentle art of questioning
 
PGT202E - BASIC EDUCATIONAL MEASUREMENT & EVALUATION
PGT202E - BASIC EDUCATIONAL MEASUREMENT & EVALUATIONPGT202E - BASIC EDUCATIONAL MEASUREMENT & EVALUATION
PGT202E - BASIC EDUCATIONAL MEASUREMENT & EVALUATION
 
Materi mc
Materi mcMateri mc
Materi mc
 
Question, question
Question, questionQuestion, question
Question, question
 
Assessing Listening with Dr. Larry Vandergrift
Assessing Listening with Dr. Larry VandergriftAssessing Listening with Dr. Larry Vandergrift
Assessing Listening with Dr. Larry Vandergrift
 

More from AbdulRashidAdams

COLLECTION AND SMEARING TECHNIQUES FOR PAP SMEARS
COLLECTION AND SMEARING TECHNIQUES FOR PAP SMEARSCOLLECTION AND SMEARING TECHNIQUES FOR PAP SMEARS
COLLECTION AND SMEARING TECHNIQUES FOR PAP SMEARS
AbdulRashidAdams
 
Shock-hemodynamic disorders general pathology
Shock-hemodynamic disorders general pathologyShock-hemodynamic disorders general pathology
Shock-hemodynamic disorders general pathology
AbdulRashidAdams
 
Chapter 2 Acute-inflammation general pathology
Chapter 2 Acute-inflammation general pathologyChapter 2 Acute-inflammation general pathology
Chapter 2 Acute-inflammation general pathology
AbdulRashidAdams
 
Chapter neoplasia-basics general pathology
Chapter neoplasia-basics general pathologyChapter neoplasia-basics general pathology
Chapter neoplasia-basics general pathology
AbdulRashidAdams
 
cartilage and bone histology. Microscopic anatomy
cartilage and bone histology. Microscopic anatomycartilage and bone histology. Microscopic anatomy
cartilage and bone histology. Microscopic anatomy
AbdulRashidAdams
 
Healing of bone fracture- general pathology
Healing of bone fracture- general pathologyHealing of bone fracture- general pathology
Healing of bone fracture- general pathology
AbdulRashidAdams
 
CELL INJURY II.ppt
CELL INJURY II.pptCELL INJURY II.ppt
CELL INJURY II.ppt
AbdulRashidAdams
 
histopathological_techniques.ppt
histopathological_techniques.ppthistopathological_techniques.ppt
histopathological_techniques.ppt
AbdulRashidAdams
 
mcq bone.docx
mcq bone.docxmcq bone.docx
mcq bone.docx
AbdulRashidAdams
 
mcq bone.docx
mcq bone.docxmcq bone.docx
mcq bone.docx
AbdulRashidAdams
 
cartilage and bone pdf.pdf
cartilage and bone pdf.pdfcartilage and bone pdf.pdf
cartilage and bone pdf.pdf
AbdulRashidAdams
 
cartilage and bone pdf.pdf
cartilage and bone pdf.pdfcartilage and bone pdf.pdf
cartilage and bone pdf.pdf
AbdulRashidAdams
 
cartilage and bone pdf.pdf
cartilage and bone pdf.pdfcartilage and bone pdf.pdf
cartilage and bone pdf.pdf
AbdulRashidAdams
 
routine staining of tissues.pptx
routine staining of tissues.pptxroutine staining of tissues.pptx
routine staining of tissues.pptx
AbdulRashidAdams
 
data analysis in research.pptx
data analysis in research.pptxdata analysis in research.pptx
data analysis in research.pptx
AbdulRashidAdams
 
results writing.pptx
results writing.pptxresults writing.pptx
results writing.pptx
AbdulRashidAdams
 
Preparation for the QIHC ExamFinal.pdf
Preparation for the QIHC ExamFinal.pdfPreparation for the QIHC ExamFinal.pdf
Preparation for the QIHC ExamFinal.pdf
AbdulRashidAdams
 
writing the results section.pptx
writing the results section.pptxwriting the results section.pptx
writing the results section.pptx
AbdulRashidAdams
 
Lecture 8.ppt
Lecture 8.pptLecture 8.ppt
Lecture 8.ppt
AbdulRashidAdams
 
8, Histology of bone.pptx
8, Histology of bone.pptx8, Histology of bone.pptx
8, Histology of bone.pptx
AbdulRashidAdams
 

More from AbdulRashidAdams (20)

COLLECTION AND SMEARING TECHNIQUES FOR PAP SMEARS
COLLECTION AND SMEARING TECHNIQUES FOR PAP SMEARSCOLLECTION AND SMEARING TECHNIQUES FOR PAP SMEARS
COLLECTION AND SMEARING TECHNIQUES FOR PAP SMEARS
 
Shock-hemodynamic disorders general pathology
Shock-hemodynamic disorders general pathologyShock-hemodynamic disorders general pathology
Shock-hemodynamic disorders general pathology
 
Chapter 2 Acute-inflammation general pathology
Chapter 2 Acute-inflammation general pathologyChapter 2 Acute-inflammation general pathology
Chapter 2 Acute-inflammation general pathology
 
Chapter neoplasia-basics general pathology
Chapter neoplasia-basics general pathologyChapter neoplasia-basics general pathology
Chapter neoplasia-basics general pathology
 
cartilage and bone histology. Microscopic anatomy
cartilage and bone histology. Microscopic anatomycartilage and bone histology. Microscopic anatomy
cartilage and bone histology. Microscopic anatomy
 
Healing of bone fracture- general pathology
Healing of bone fracture- general pathologyHealing of bone fracture- general pathology
Healing of bone fracture- general pathology
 
CELL INJURY II.ppt
CELL INJURY II.pptCELL INJURY II.ppt
CELL INJURY II.ppt
 
histopathological_techniques.ppt
histopathological_techniques.ppthistopathological_techniques.ppt
histopathological_techniques.ppt
 
mcq bone.docx
mcq bone.docxmcq bone.docx
mcq bone.docx
 
mcq bone.docx
mcq bone.docxmcq bone.docx
mcq bone.docx
 
cartilage and bone pdf.pdf
cartilage and bone pdf.pdfcartilage and bone pdf.pdf
cartilage and bone pdf.pdf
 
cartilage and bone pdf.pdf
cartilage and bone pdf.pdfcartilage and bone pdf.pdf
cartilage and bone pdf.pdf
 
cartilage and bone pdf.pdf
cartilage and bone pdf.pdfcartilage and bone pdf.pdf
cartilage and bone pdf.pdf
 
routine staining of tissues.pptx
routine staining of tissues.pptxroutine staining of tissues.pptx
routine staining of tissues.pptx
 
data analysis in research.pptx
data analysis in research.pptxdata analysis in research.pptx
data analysis in research.pptx
 
results writing.pptx
results writing.pptxresults writing.pptx
results writing.pptx
 
Preparation for the QIHC ExamFinal.pdf
Preparation for the QIHC ExamFinal.pdfPreparation for the QIHC ExamFinal.pdf
Preparation for the QIHC ExamFinal.pdf
 
writing the results section.pptx
writing the results section.pptxwriting the results section.pptx
writing the results section.pptx
 
Lecture 8.ppt
Lecture 8.pptLecture 8.ppt
Lecture 8.ppt
 
8, Histology of bone.pptx
8, Histology of bone.pptx8, Histology of bone.pptx
8, Histology of bone.pptx
 

Recently uploaded

Genetics Teaching Plan: Dr.Kshirsagar R.V.
Genetics Teaching Plan: Dr.Kshirsagar R.V.Genetics Teaching Plan: Dr.Kshirsagar R.V.
Genetics Teaching Plan: Dr.Kshirsagar R.V.
DrRavindrakshirsagar1
 
NAEYC Code of Ethical Conduct Resource Book
NAEYC Code of Ethical Conduct Resource BookNAEYC Code of Ethical Conduct Resource Book
NAEYC Code of Ethical Conduct Resource Book
lakitawilson
 
How to Manage Access Rights & User Types in Odoo 17
How to Manage Access Rights & User Types in Odoo 17How to Manage Access Rights & User Types in Odoo 17
How to Manage Access Rights & User Types in Odoo 17
Celine George
 
What is Rescue Session in Odoo 17 POS - Odoo 17 Slides
What is Rescue Session in Odoo 17 POS - Odoo 17 SlidesWhat is Rescue Session in Odoo 17 POS - Odoo 17 Slides
What is Rescue Session in Odoo 17 POS - Odoo 17 Slides
Celine George
 
Individual Performance Commitment Review Form-Developmental Plan.docx
Individual Performance Commitment Review Form-Developmental Plan.docxIndividual Performance Commitment Review Form-Developmental Plan.docx
Individual Performance Commitment Review Form-Developmental Plan.docx
monicaaringo1
 
The basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptxThe basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptx
heathfieldcps1
 
The Jewish Trinity : Sabbath,Shekinah and Sanctuary 4.pdf
The Jewish Trinity : Sabbath,Shekinah and Sanctuary 4.pdfThe Jewish Trinity : Sabbath,Shekinah and Sanctuary 4.pdf
The Jewish Trinity : Sabbath,Shekinah and Sanctuary 4.pdf
JackieSparrow3
 
DANH SÁCH THÍ SINH XÉT TUYỂN SỚM ĐỦ ĐIỀU KIỆN TRÚNG TUYỂN ĐẠI HỌC CHÍNH QUY N...
DANH SÁCH THÍ SINH XÉT TUYỂN SỚM ĐỦ ĐIỀU KIỆN TRÚNG TUYỂN ĐẠI HỌC CHÍNH QUY N...DANH SÁCH THÍ SINH XÉT TUYỂN SỚM ĐỦ ĐIỀU KIỆN TRÚNG TUYỂN ĐẠI HỌC CHÍNH QUY N...
DANH SÁCH THÍ SINH XÉT TUYỂN SỚM ĐỦ ĐIỀU KIỆN TRÚNG TUYỂN ĐẠI HỌC CHÍNH QUY N...
thanhluan21
 
SEQUNCES Lecture_Notes_Unit4_chapter11_sequence
SEQUNCES  Lecture_Notes_Unit4_chapter11_sequenceSEQUNCES  Lecture_Notes_Unit4_chapter11_sequence
SEQUNCES Lecture_Notes_Unit4_chapter11_sequence
Murugan Solaiyappan
 
Year-to-Date Filter in Odoo 17 Dashboard
Year-to-Date Filter in Odoo 17 DashboardYear-to-Date Filter in Odoo 17 Dashboard
Year-to-Date Filter in Odoo 17 Dashboard
Celine George
 
"DANH SÁCH THÍ SINH XÉT TUYỂN SỚM ĐỦ ĐIỀU KIỆN TRÚNG TUYỂN ĐẠI HỌC CHÍNH QUY ...
"DANH SÁCH THÍ SINH XÉT TUYỂN SỚM ĐỦ ĐIỀU KIỆN TRÚNG TUYỂN ĐẠI HỌC CHÍNH QUY ..."DANH SÁCH THÍ SINH XÉT TUYỂN SỚM ĐỦ ĐIỀU KIỆN TRÚNG TUYỂN ĐẠI HỌC CHÍNH QUY ...
"DANH SÁCH THÍ SINH XÉT TUYỂN SỚM ĐỦ ĐIỀU KIỆN TRÚNG TUYỂN ĐẠI HỌC CHÍNH QUY ...
thanhluan21
 
Bedok NEWater Photostory - COM322 Assessment (Story 2)
Bedok NEWater Photostory - COM322 Assessment (Story 2)Bedok NEWater Photostory - COM322 Assessment (Story 2)
Bedok NEWater Photostory - COM322 Assessment (Story 2)
Liyana Rozaini
 
How To Update One2many Field From OnChange of Field in Odoo 17
How To Update One2many Field From OnChange of Field in Odoo 17How To Update One2many Field From OnChange of Field in Odoo 17
How To Update One2many Field From OnChange of Field in Odoo 17
Celine George
 
New Features in Odoo 17 Sign - Odoo 17 Slides
New Features in Odoo 17 Sign - Odoo 17 SlidesNew Features in Odoo 17 Sign - Odoo 17 Slides
New Features in Odoo 17 Sign - Odoo 17 Slides
Celine George
 
H. A. Roberts: VITAL FORCE - Dr. Niranjan Bapat
H. A. Roberts: VITAL FORCE - Dr. Niranjan BapatH. A. Roberts: VITAL FORCE - Dr. Niranjan Bapat
H. A. Roberts: VITAL FORCE - Dr. Niranjan Bapat
Niranjan Bapat
 
How to Create a New Article in Knowledge App in Odoo 17
How to Create a New Article in Knowledge App in Odoo 17How to Create a New Article in Knowledge App in Odoo 17
How to Create a New Article in Knowledge App in Odoo 17
Celine George
 
How to Handle the Separate Discount Account on Invoice in Odoo 17
How to Handle the Separate Discount Account on Invoice in Odoo 17How to Handle the Separate Discount Account on Invoice in Odoo 17
How to Handle the Separate Discount Account on Invoice in Odoo 17
Celine George
 
How to Manage Large Scrollbar in Odoo 17 POS
How to Manage Large Scrollbar in Odoo 17 POSHow to Manage Large Scrollbar in Odoo 17 POS
How to Manage Large Scrollbar in Odoo 17 POS
Celine George
 
Imagination in Computer Science Research
Imagination in Computer Science ResearchImagination in Computer Science Research
Imagination in Computer Science Research
Abhik Roychoudhury
 
Bài tập bộ trợ anh 7 I learn smart world kì 1 năm học 2022 2023 unit 1.doc
Bài tập bộ trợ anh 7 I learn smart world kì 1 năm học 2022 2023 unit 1.docBài tập bộ trợ anh 7 I learn smart world kì 1 năm học 2022 2023 unit 1.doc
Bài tập bộ trợ anh 7 I learn smart world kì 1 năm học 2022 2023 unit 1.doc
PhngThLmHnh
 

Recently uploaded (20)

Genetics Teaching Plan: Dr.Kshirsagar R.V.
Genetics Teaching Plan: Dr.Kshirsagar R.V.Genetics Teaching Plan: Dr.Kshirsagar R.V.
Genetics Teaching Plan: Dr.Kshirsagar R.V.
 
NAEYC Code of Ethical Conduct Resource Book
NAEYC Code of Ethical Conduct Resource BookNAEYC Code of Ethical Conduct Resource Book
NAEYC Code of Ethical Conduct Resource Book
 
How to Manage Access Rights & User Types in Odoo 17
How to Manage Access Rights & User Types in Odoo 17How to Manage Access Rights & User Types in Odoo 17
How to Manage Access Rights & User Types in Odoo 17
 
What is Rescue Session in Odoo 17 POS - Odoo 17 Slides
What is Rescue Session in Odoo 17 POS - Odoo 17 SlidesWhat is Rescue Session in Odoo 17 POS - Odoo 17 Slides
What is Rescue Session in Odoo 17 POS - Odoo 17 Slides
 
Individual Performance Commitment Review Form-Developmental Plan.docx
Individual Performance Commitment Review Form-Developmental Plan.docxIndividual Performance Commitment Review Form-Developmental Plan.docx
Individual Performance Commitment Review Form-Developmental Plan.docx
 
The basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptxThe basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptx
 
The Jewish Trinity : Sabbath,Shekinah and Sanctuary 4.pdf
The Jewish Trinity : Sabbath,Shekinah and Sanctuary 4.pdfThe Jewish Trinity : Sabbath,Shekinah and Sanctuary 4.pdf
The Jewish Trinity : Sabbath,Shekinah and Sanctuary 4.pdf
 
DANH SÁCH THÍ SINH XÉT TUYỂN SỚM ĐỦ ĐIỀU KIỆN TRÚNG TUYỂN ĐẠI HỌC CHÍNH QUY N...
DANH SÁCH THÍ SINH XÉT TUYỂN SỚM ĐỦ ĐIỀU KIỆN TRÚNG TUYỂN ĐẠI HỌC CHÍNH QUY N...DANH SÁCH THÍ SINH XÉT TUYỂN SỚM ĐỦ ĐIỀU KIỆN TRÚNG TUYỂN ĐẠI HỌC CHÍNH QUY N...
DANH SÁCH THÍ SINH XÉT TUYỂN SỚM ĐỦ ĐIỀU KIỆN TRÚNG TUYỂN ĐẠI HỌC CHÍNH QUY N...
 
SEQUNCES Lecture_Notes_Unit4_chapter11_sequence
SEQUNCES  Lecture_Notes_Unit4_chapter11_sequenceSEQUNCES  Lecture_Notes_Unit4_chapter11_sequence
SEQUNCES Lecture_Notes_Unit4_chapter11_sequence
 
Year-to-Date Filter in Odoo 17 Dashboard
Year-to-Date Filter in Odoo 17 DashboardYear-to-Date Filter in Odoo 17 Dashboard
Year-to-Date Filter in Odoo 17 Dashboard
 
"DANH SÁCH THÍ SINH XÉT TUYỂN SỚM ĐỦ ĐIỀU KIỆN TRÚNG TUYỂN ĐẠI HỌC CHÍNH QUY ...
"DANH SÁCH THÍ SINH XÉT TUYỂN SỚM ĐỦ ĐIỀU KIỆN TRÚNG TUYỂN ĐẠI HỌC CHÍNH QUY ..."DANH SÁCH THÍ SINH XÉT TUYỂN SỚM ĐỦ ĐIỀU KIỆN TRÚNG TUYỂN ĐẠI HỌC CHÍNH QUY ...
"DANH SÁCH THÍ SINH XÉT TUYỂN SỚM ĐỦ ĐIỀU KIỆN TRÚNG TUYỂN ĐẠI HỌC CHÍNH QUY ...
 
Bedok NEWater Photostory - COM322 Assessment (Story 2)
Bedok NEWater Photostory - COM322 Assessment (Story 2)Bedok NEWater Photostory - COM322 Assessment (Story 2)
Bedok NEWater Photostory - COM322 Assessment (Story 2)
 
How To Update One2many Field From OnChange of Field in Odoo 17
How To Update One2many Field From OnChange of Field in Odoo 17How To Update One2many Field From OnChange of Field in Odoo 17
How To Update One2many Field From OnChange of Field in Odoo 17
 
New Features in Odoo 17 Sign - Odoo 17 Slides
New Features in Odoo 17 Sign - Odoo 17 SlidesNew Features in Odoo 17 Sign - Odoo 17 Slides
New Features in Odoo 17 Sign - Odoo 17 Slides
 
H. A. Roberts: VITAL FORCE - Dr. Niranjan Bapat
H. A. Roberts: VITAL FORCE - Dr. Niranjan BapatH. A. Roberts: VITAL FORCE - Dr. Niranjan Bapat
H. A. Roberts: VITAL FORCE - Dr. Niranjan Bapat
 
How to Create a New Article in Knowledge App in Odoo 17
How to Create a New Article in Knowledge App in Odoo 17How to Create a New Article in Knowledge App in Odoo 17
How to Create a New Article in Knowledge App in Odoo 17
 
How to Handle the Separate Discount Account on Invoice in Odoo 17
How to Handle the Separate Discount Account on Invoice in Odoo 17How to Handle the Separate Discount Account on Invoice in Odoo 17
How to Handle the Separate Discount Account on Invoice in Odoo 17
 
How to Manage Large Scrollbar in Odoo 17 POS
How to Manage Large Scrollbar in Odoo 17 POSHow to Manage Large Scrollbar in Odoo 17 POS
How to Manage Large Scrollbar in Odoo 17 POS
 
Imagination in Computer Science Research
Imagination in Computer Science ResearchImagination in Computer Science Research
Imagination in Computer Science Research
 
Bài tập bộ trợ anh 7 I learn smart world kì 1 năm học 2022 2023 unit 1.doc
Bài tập bộ trợ anh 7 I learn smart world kì 1 năm học 2022 2023 unit 1.docBài tập bộ trợ anh 7 I learn smart world kì 1 năm học 2022 2023 unit 1.doc
Bài tập bộ trợ anh 7 I learn smart world kì 1 năm học 2022 2023 unit 1.doc
 

Science-Writing-Effective-MCQs-Workshop-Nov2020.pptx

  • 1. Get chatting Before the workshop starts at 14:00, briefly introduce any previous experiences using Moodle quizzes •Contexts •Positives •Negatives Join TEL Community of Practice on Teams This session is being recorded Mute Chat
  • 2. tel@bath.ac.uk Writing Effective Multiple Choice Questions Bertie Squire & Josh Lim Technology Enhanced Learning (TEL) team Instructional Designers Team teachinghub.bath.ac.uk
  • 3. Today’s Session At the end of this workshop you will learn:  The benefits of using MCQs both for both assessment & learning  How to write effective MCQs and pitfalls to avoid  How to construct, analyze and alter MCQs questions  The experiences, ideas and tips of others! Recording session & will make slides (with links) available Please get in touch afterwards tel@bath.ac.uk for any follow-up queries
  • 4. Good quiz practices = link to learning Empower students as self-regulated learners Activity linked to learning objectives Aims for learners • Impact through quality feedback = students act on feedback • Bitesize, low stakes  build motivation & self-esteem • Combine with other learning & teaching strategies D. J. Nicol & D. Macfarlane‐Dick (2006) For authors Planning • Learning objectives + cognitive skills • Feedback  next steps, dialogue (e.g. tutorials, signposting, peer support, forums, problems classes) Writing • See following guidance... • Continuously Managing • Build question bank over time • Use data to inform • Review questions
  • 5. Multiple Choice Questions Advantages Disadvantages • Different cognitive levels – fact recall/apply/evaluate • Finding “good” distractors hard • Dominance of “recall” questions? • “Higher order” vs. surface learning? • Wide coverage (less “sampling”) • Fast feedback for students • Easy to mark/score - can apply frequently • Test analytics • Different Uses (diagnostic, revision, Assessment for/as Learning) • Many accessible features “built-in” • Difficult to “show working” • Not well suited to maths • Accuracy/suitability of assessment • Engagement • Students guessing • Poor questions -> teachers work harder than students • Fairness/accuracy
  • 6. "Non-googleable MCQs" Learning objectives + cognitive skill = MCQ test 1. Shift focus from recall to application – Understand (explain) – Apply (recall in context) – Analyse (explain in context) – Evaluate (best fit) 2. Use novel context 3. Use common mistakes/misconceptions as distractors 4. Multiple select (1 or 2 correct)* 5. Embrace open book exams… Bloom’s Taxonomy by Nicoguaro is licensed under CC BY-SA
  • 7. Constructing Effective Questions Stem should be meaningful on its own Don’t include irrelevant material Use negative phrasing carefully 1 2 3 Writing Good Multiple Choice Test Questions – C. Brame (2013)
  • 8. Constructing Effective Distractors Plausible Clear and concise Mutually exclusive Homogenous in nature Free from having clues to correct answer All the distractors should be: The distractors aren’t functional if they are not plausible, as students who don’t understand the learning outcomes can still be rewarded. Use common student errors – these make the best distractors. Students who know the correct answer should not have to worry about unclear and wordy answers and grasping what the answer is actually trying to say Writing Good Multiple Choice Test Questions – C. Brame (2013) A ‘trick’ question can sometimes lead to multiple answers being correct. Don’t have this ambiguity – make the answers mutually exclusive This means the answers should all be in the same topic area. If one answer is focused on a different subject, it can lead students to the correct answer. Savvy test takers can spot clues to correct answers such as differences in grammar, length, formatting, and language choices.
  • 9. MCQ pitfalls to avoid 1) Select the correct option MCQs are not… A) Unlikely to be never ineffective B) somewhat easy C) Difficult to write, particularly to make them valid D) All of the above 2) Select the errors demonstrated in question 1 from the list (Answers: 2,4,6,7,8,9) Cognitive load 1. Superfluous information (minimise reading) 2. Unfocused question stems (avoid lots of text or open ended) and blanks 3. Unimportant content/minutia – instead test main points 4. Negative stem or answer options (if necessary, be clear to emphasise the negative element “Which of this is NOT…”) 5. Confusingly ordered options (e.g. by size) Wording Clues 6. Absolute terms (easy to dismiss) or vague terms (open to interpretation) 7. Grammatical clues (options deduced from stem) Systematic 8. Unequal option length (correct answers often take more info) 9. “All/none of the above” option (e.g. trivial if 2+ match) 10. Logical clues (avoid mutually exclusive pair options, use multi select question instead) 11. Convergent option – correct answer has similarities with others Webb et al., 2015
  • 10. Limiting student collusion Some strategies (with collusion disincentive in brackets), listed from more extrinsic to intrinsic student motivation: A. Police/enforce/invigilate (the “stick” approach) B. Restrict duration (limit opportunities) C. Don’t have marks associated with assessment – separate formative/summative assessment (remove incentive) D. Question randomisation: mix up the order in the test or set question variants (remove ease of colluding) E. Use confidence based marking to score students and encourage students to reflect (dissuade blind copying) F. Follow up assessment e.g. create consequences: pick students to explain answer in tutorial, random to provide written worked solution justification to accompany course material (create incentives to be honest & seek help) G. Use variable/context randomisation – this could be used to promote peer collaboration/discussion (avoid simple copying) H. Educate and promote ethos, purpose of assessment– make it clear to students (from all backgrounds) what collusion is, aims of assessment and disadvantages to learning (educate learners) I. Broaden assessment aims to allow collaboration and group/peer instruction (remove incentive) J. High intrinsic value of the activity and feedback (make motivation to collude counterproductive) Disclaimers! • This list isn’t exhaustive and the exact and extent of the solution(s) will be context specific (the above is not a hierarchy), but may be useful to consider! • Many of these issues are not unique to e-Quizzes, but digitising does give some solutions!
  • 12. Improve an MCQ question Open WORKSHOP ACTIVITY Powerpoint Write an improved question • Improve question stem? • Improve distractors? • Shift question emphasis? One person per group > Feedback on what you did Groups: 3-4 10 mins Feel free to add text box comments to your own slides!
  • 13. MCQ Experiences Dicussion • Think about past experiences both taking and delivering MCQs  What went well?  Do you refine and revisit questions?  Did you vary question types?  Were then un-googleable?  Any tips which you found helped?  Did it help students’ understanding?
  • 14. Key points • Start with learning objectives & the focus for the quiz • Write effective questions! • Review and revisit your question bank
  • 15. References and further reading • Quick read: Writing Good Multiple Choice Test Questions – C. Brame (2013) • Good summary with links to literature: The Importance of Writing Effective Distractors – C Malamed (2020) • Nicol, D. J & Macfarlane‐Dick, D., 2006. Formative assessment and self‐regulated learning: a model & 7 principles of good feedback practice, Studies in Higher Education 31(2) p199 • Carneson J, Delpierre G, Masters K., 2016. Handbook: Designing and Managing Multiple Choice Questions (2nd Ed) [online only] • Gierl, M et al., 2017. Developing, Analyzing, and Using Distractors for Multiple-Choice Tests in Education: A Comprehensive Review. Review of Educational Research, 87: 6, pp. 1082 –1116. • Rodriguez, M. (2005). Three Options Are Optimal for Multiple‐Choice Items: A Meta‐Analysis of 80 Years of Research, Educational Measurement: Issues and Practice, 24:2. • Haladyna, T.M. et al., 2002. A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment. Applied Measurement in Education 15(3) p309 • Webb, E et al., 2015. Does Educator Training or Experience Affect the Quality of Multiple-Choice Questions? Academic Radiology 22(10) pp. 1317 • McCoubrie, P. 2004. Improving the fairness of multiple-choice questions: a literature review. Medical Teacher 26(8) p709 Moodle Quiz Settings and Setup FAQ on the Teaching Hub
  • 16. How to get engagement in formative assessment and feedback? Link to general engagement strategies: employability & real world context, develop skills, student voice/ownership, peer assisted, student generated feedback, stakeholder engagement, social learning, diversify/integrate with other teaching activities, educate students on benefits, combine with summative, students reflect on learning, reflect & adapt teaching strategy …. Strategies specific to online quizzes: • Vary quiz behaviour • low stakes & “fast fail” approach: multiple attempts (with penalties), instant feedback • Use social aspects as a lever with technology • Variable/context randomisation to promote peer collaboration • Social learning – forums, MS Teams etc to support discussion • Student generated feedback – students as assessors (e.g. Moodle workshop) • Vary feedback delivery • instant timely feedback, selective feedback (reveal feedback only, hide score etc), blended feedback (in tutorial) • Stress High intrinsic value of the activity and feedback • Assess aspects that would be take to long to assess traditionally (e.g. wide coverage, calculations with real data set etc). • Timely feedback can be more effective • Create quality bespoke feedback (this becomes scalable)
  • 17. Before you start • Guidelines for setting up quizzes have been created for the current assessment period • Read the guidance (even if familiar with Moodle quizzes) 1. Dos and Don'ts of online assessments PDF 2. Quiz settings Setup and FAQ 3. Creating a Quiz Assessment in Moodle as an alternative assessment (PDF) Some settings will be different from the ones you normally use for formative quizzes e.g. visibility of feedback
  • 18. Moodle quiz settings Step by step instructions with screenshots: Creating a Quiz Assessment in Moodle as an alternative assessment (PDF) Quiz settings (rubric, availability, timing etc) Extra time Edit question set View, amend, export grades Quiz settings Setup and FAQ
  • 19. Setting up the Moodle quiz Key things to remember: • Read the guidance and follow carefully • Use default settings unless otherwise specified • Name your quiz clearly • Give clear instructions to students (University Rubric, Academic Integrity Declaration) • Consider using grade transfer • Adjust availability or duration for students with DAPs • Test: Preview your quiz, check availability