Assessment fyfa cmg

Assessment- Making Thinking Visible
Cormac McGrath
Director Unit for Medical Education (CLK, LIME)
11/24/15Centre for Medical Education, Karolinska Institutet

Agenda: Assessment and MCQs
 Student learning
 Interpolated memory tests
 Formative and summative assessment
 Feedback
 Outcome based education
 Constructive alignment
 Criterion versus norm based assessment
 Sketching a way forward
 Constructing MCQs
 Validating MCQs
 Think aloud protocols
 Digitalising exams
Centre for Medical Education, Karolinska Institutet

From idea to action: a series of four
workshops
 MCQ construction
 Building a database
 Testing your MCQs
 Face validity
 Testing on students
 Talk-aloud protocols
 Scaling up
 digitalising the project

Discussion
 Positive experiences of assessing students’ work
 What concerns/ queries do you have about assessment?
 Pair and Share
2-3 min

Do student know what they don’t know?
Dunning-Kreuger effect
 Students may:
 fail to recognize their own
lack of skill
 fail to recognize genuine
skill in others
 fail to recognize the extent
of their inadequacy
 recognize and
acknowledge their own
lack of skill, after they are
exposed to training for
that skill

Student learning: What do we know?
Backwash effect
November 24, 2015
6
ILOs
Teaching
methods
Examination
Examination Study methods Student
learning

What supports learning?
 Interpolated memory
tests
students retained focus
less likely to mind
wander
scored better on
cumulative tests
 Szpunar K,K, et al.,
2012 (Interpolated
memory tests reduce
mind wandering and
improve learning of
online learning)

What supports learning?
 Interconnectedness
 Meaningfulness

Student learning: Feedback
Synthesis of 800 meta analysis, involving 80 million students
 Biggest effect on student learning:
• Learners’ knowledge of goals/criteria
• Results/feedback to the learner
• Feedback About the Task
• Feedback About the Processing of the Task
• Feedback About Self-Regulation
• The teacher's pedagogical skills
• Classroom environment
• Analyze teaching occasions together with colleagues
Hattie, Synligt lärande, rapport SKL , 2011

How do can we support student
learning?
 Formative assessment
- Continuously runs during the teaching/ learning process
- Diagnostic: gives feedback to students and teachers on
* strengths and weaknesses
* difficulties
* misunderstandings
- Gives an opportunity to modify/ improve

Discussion
 To which degree do to you provide opportunities for interpolated
“memory” tests in your teaching today?

Cormac McGrathCormac McGrath, UME 13
Outcome based education
 Outcomes at different levels dictate student
learning
 Outcomes at programme level
 Specific course goals
 Teaching ocassion outcomes

Cormac McGrathCormac McGrath, UME 14
Course design and alingment
Learning outcomes
Assessment
Learning activities
Evaluation
Biggs, 2003

Progression in the choice of verbs
SOLO taxonomy
Identify
Make
simple
procedures
Number
Describe
List
Combine
Compare
Contrast
Explain
relationships
Analyse
Relate
Apply
Misses
the point
Theorise
Generalise
Make a
hypothesis
Reflect
(Biggs & Tang 2007)

Discussion
 Review your intended learning outcomes using the SOLO
taxonomy.

Criterion V norm-referenced assessment
 Criterion-referenced
assessment:
 How well students have learnt
what we intended them to
learn.
 Criterion-referenced tests
and assessments are
designed to measure student
performance against a fixed
set of predetermined criteria
or learning standards—i.e.,
concise, written descriptions
of what students are expected
to know and be able to do at
a specific stage of their
education.
 Norm-referenced
assessment:
 Comparing students
performances with each
other, by ranking.
 Compare to a cohort
 Relative to a group

Discussion
 Which resonates most strongly with you; norm referenced or
criterion referenced assessment?
 2-3 min
 To which extent do the assessment criteria express distinct
qualitative differences?
 Can you give examples?
 How do you share this with the students?
 How well do you discuss examination and
assessment/assessment criteria?

Change of focus: Traditional MCQ
assessment
 Emphasis on product and declarative knowledge
 Is fair (?)
 Is economical
 Encourages repetition of large amounts of literature, which can
provide opportunity for overview of the subject
Criticism:
 Encourages quick answers rather than slow investigation
 Correctness rather than risk taking
 Memory rather than conceptual development
 Reproduction rather than creative application
 Little space/time for feedback
 Difficult to measure deep understanding of concepts and
context (Lindström, 1997)

Powerful examinations?
The design is supposed to create the possibility to assess:
 specific cognitive competences such as problem-solving,
including formulating questions and critical thinking,
 information competences, such as searching for relevant
information, making informed judgements, efficient use of
information, analysing data,
 communication competences, such as presenting data
communicatively, both oral and written;
 meta-cognitive competences such as self-reflection and self-
evaluation.

Bloom and SOLO
Taxonomy Level Representative Verbs SOLO
Evaluation
•Critique
•Summarize
S4
Synthesis
•Organize
•Design
S3
Analysis
•Compare
•Categorize
S2
Application
•Organize
•Solve
S2
Comprehension
•Distinguish
•Match
S1
Knowledge
•Identify
•Label
S1

 Can we achieve and assess higher order thinking with MCQs

Summary
 Students’ learning is guided by examinations
 Backwash effect
 Formative assessment facilitates learning
 Recurrent testing promotes learning
 As course leader you can influence the design of your course
and the ratio between teaching and examination.

Part two:MCQ

What do we want to assess? How do we use
MCQs to achieve that?
 Different levels of learning/understanding
 Identify (what is this muscle called)
 Describe/list (which muscle are involved in elbow flexion)
 Relate/apply (muscle x is injured how does it affect movement y)
 Theorise/generalise/Evolve

The different types of questions
Anatomy
”Problem solv
Isolated structure
Single joint
Multiple Joints
1 2 3 4
Identify/list
Describe/apply
Describe/apply/evolve

What parts are there to a question?

Some things to think about when crafting
questions

Cormac McGrathPedagogik för universitetslärare
35
Progression of verb
SOLO taxonomy
Identifies
Simple
procedures
Numbers
Describes
Lists
Combines
Compares
Contrasts
Explains
connections
Analyses
Applies
Misses
the point
Creates theories
Generalises
Hypothesises
Reflects
(Biggs 2003)

References
 Biggs, J. &C. Tang (2007). Teaching for Quality Learning at
University. SRHE & Open University Press, Buckingham.
 Carroll, J. &C.-M. Zetterling (2009). Guiding students away from
plagiarism. KTH Learning Lab, Stockholm.
 McConnell, DA., Steer, DN., & Owens, KD. (2003) Assessment
and active learning strategies for introductory geology courses.
Journal of Geoscience Education, v. 51 n:o 2, p. 205-216

Assessment fyfa cmg

More Related Content

What's hot

Viewers also liked

Similar to Assessment fyfa cmg

More from Cormac McGrath

Recently uploaded

Assessment fyfa cmg

Editor's Notes