MCQ and assessment

1. Assessment- Making Thinking Visible
Cormac McGrath
Director Unit for Medical Education (CLK, LIME)
11/24/15Centre for Medical Education, Karolinska Institutet

2. Agenda: Assessment and MCQs
 Student learning
 Interpolated memory tests
 Formative and summative assessment
 Feedback
 Outcome based education
 Constructive alignment
 Criterion versus norm based assessment
 Sketching a way forward
 Constructing MCQs
 Validating MCQs
 Think aloud protocols
 Digitalising exams
Centre for Medical Education, Karolinska Institutet

3. From idea to action: a series of four
workshops
 MCQ construction
 Building a database
 Testing your MCQs
 Face validity
 Testing on students
 Talk-aloud protocols
 Scaling up
 digitalising the project
11/24/15Centre for Medical Education, Karolinska Institutet

4. Discussion
 Positive experiences of assessing students’ work
 What concerns/ queries do you have about assessment?
 Pair and Share
2-3 min
11/24/15Centre for Medical Education, Karolinska Institutet

5. Do student know what they don’t know?
Dunning-Kreuger effect
 Students may:
 fail to recognize their own
lack of skill
 fail to recognize genuine
skill in others
 fail to recognize the extent
of their inadequacy
 recognize and
acknowledge their own
lack of skill, after they are
exposed to training for
that skill
11/24/15Centre for Medical Education, Karolinska Institutet

6. Student learning: What do we know?
Backwash effect
November 24, 2015
6
ILOs
Teaching
methods
Examination
Examination Study methods Student
learning

7. What supports learning?
 Interpolated memory
tests
students retained focus
less likely to mind
wander
scored better on
cumulative tests
 Szpunar K,K, et al.,
2012 (Interpolated
memory tests reduce
mind wandering and
improve learning of
online learning)
11/24/15Centre for Medical Education, Karolinska Institutet

8. What supports learning?
 Interconnectedness
 Meaningfulness
11/24/15Centre for Medical Education, Karolinska Institutet

9. Student learning: Feedback
Synthesis of 800 meta analysis, involving 80 million students
 Biggest effect on student learning:
• Learners’ knowledge of goals/criteria
• Results/feedback to the learner
• Feedback About the Task
• Feedback About the Processing of the Task
• Feedback About Self-Regulation
• The teacher's pedagogical skills
• Classroom environment
• Analyze teaching occasions together with colleagues
Hattie, Synligt lärande, rapport SKL , 2011

10. How do can we support student
learning?
 Formative assessment
- Continuously runs during the teaching/ learning process
- Diagnostic: gives feedback to students and teachers on
* strengths and weaknesses
* difficulties
* misunderstandings
- Gives an opportunity to modify/ improve
11/24/15Centre for Medical Education, Karolinska Institutet

11. Discussion
 To which degree do to you provide opportunities for interpolated
“memory” tests in your teaching today?
11/24/15Centre for Medical Education, Karolinska Institutet

12. Cormac McGrathCormac McGrath, UME 13
Outcome based education
 Outcomes at different levels dictate student
learning
 Outcomes at programme level
 Specific course goals
 Teaching ocassion outcomes

13. Cormac McGrathCormac McGrath, UME 14
Course design and alingment
Learning outcomes
Assessment
Learning activities
Evaluation
Biggs, 2003

14. 11/24/15Centre for Medical Education, Karolinska Institutet
Progression in the choice of verbs
SOLO taxonomy
Identify
Make
simple
procedures
Number
Describe
List
Combine
Compare
Contrast
Explain
relationships
Analyse
Relate
Apply
Misses
the point
Theorise
Generalise
Make a
hypothesis
Reflect
(Biggs & Tang 2007)

15. Discussion
 Review your intended learning outcomes using the SOLO
taxonomy.
11/24/15Centre for Medical Education, Karolinska Institutet

16. Criterion V norm-referenced assessment
 Criterion-referenced
assessment:
 How well students have learnt
what we intended them to
learn.
 Criterion-referenced tests
and assessments are
designed to measure student
performance against a fixed
set of predetermined criteria
or learning standards—i.e.,
concise, written descriptions
of what students are expected
to know and be able to do at
a specific stage of their
education.
 Norm-referenced
assessment:
 Comparing students
performances with each
other, by ranking.
 Compare to a cohort
 Relative to a group
11/24/15Centre for Medical Education, Karolinska Institutet

17. Discussion
 Which resonates most strongly with you; norm referenced or
criterion referenced assessment?
 2-3 min
 To which extent do the assessment criteria express distinct
qualitative differences?
 Can you give examples?
 How do you share this with the students?
 How well do you discuss examination and
assessment/assessment criteria?
11/24/15Centre for Medical Education, Karolinska Institutet

18. 11/24/15Centre for Medical Education, Karolinska Institutet
Change of focus: Traditional MCQ
assessment
 Emphasis on product and declarative knowledge
 Is fair (?)
 Is economical
 Encourages repetition of large amounts of literature, which can
provide opportunity for overview of the subject
Criticism:
 Encourages quick answers rather than slow investigation
 Correctness rather than risk taking
 Memory rather than conceptual development
 Reproduction rather than creative application
 Little space/time for feedback
 Difficult to measure deep understanding of concepts and
context (Lindström, 1997)

19. Powerful examinations?
The design is supposed to create the possibility to assess:
 specific cognitive competences such as problem-solving,
including formulating questions and critical thinking,
 information competences, such as searching for relevant
information, making informed judgements, efficient use of
information, analysing data,
 communication competences, such as presenting data
communicatively, both oral and written;
 meta-cognitive competences such as self-reflection and self-
evaluation.

20. Bloom and SOLO
Taxonomy Level Representative Verbs SOLO
Evaluation
•Critique
•Summarize
S4
Synthesis
•Organize
•Design
S3
Analysis
•Compare
•Categorize
S2
Application
•Organize
•Solve
S2
Comprehension
•Distinguish
•Match
S1
Knowledge
•Identify
•Label
S1
11/24/15Centre for Medical Education, Karolinska Institutet

21.  Can we achieve and assess higher order thinking with MCQs
11/24/15Centre for Medical Education, Karolinska Institutet

22. From idea to action: a series of four
workshops
 MCQ construction
 Building a database
 Testing your MCQs
 Face validity
 Testing on students
 Talk-aloud protocols
 Scaling up
 digitalising the project
11/24/15Centre for Medical Education, Karolinska Institutet

23. Summary
 Students’ learning is guided by examinations
 Backwash effect
 Formative assessment facilitates learning
 Recurrent testing promotes learning
 As course leader you can influence the design of your course
and the ratio between teaching and examination.
11/24/15Centre for Medical Education, Karolinska Institutet

24. Part two:MCQ
11/24/15Centre for Medical Education, Karolinska Institutet

25. What do we want to assess? How do we use
MCQs to achieve that?
 Different levels of learning/understanding
 Identify (what is this muscle called)
 Describe/list (which muscle are involved in elbow flexion)
 Relate/apply (muscle x is injured how does it affect movement y)
 Theorise/generalise/Evolve

26. The different types of questions
Anatomy
”Problem solv
Isolated structure
Single joint
Multiple Joints
1 2 3 4
Identify/list
Describe/apply
Describe/apply/evolve

27. The pros and cons of MCQ

28. What parts are there to a question?

29. The stem

31. Some things to think about when crafting
questions

33. Cormac McGrathPedagogik för universitetslärare
35
Progression of verb
SOLO taxonomy
Identifies
Simple
procedures
Numbers
Describes
Lists
Combines
Compares
Contrasts
Explains
connections
Analyses
Applies
Misses
the point
Creates theories
Generalises
Hypothesises
Reflects
(Biggs 2003)

34. 11/24/15Centre for Medical Education, Karolinska Institutet
References
 Biggs, J. &C. Tang (2007). Teaching for Quality Learning at
University. SRHE & Open University Press, Buckingham.
 Carroll, J. &C.-M. Zetterling (2009). Guiding students away from
plagiarism. KTH Learning Lab, Stockholm.
 McConnell, DA., Steer, DN., & Owens, KD. (2003) Assessment
and active learning strategies for introductory geology courses.
Journal of Geoscience Education, v. 51 n:o 2, p. 205-216