Assessment of doctoral education: courses.

1. Assessment- Making Thinking Visible
Cormac McGrath
Director Unit for Medical Education (CLK, LIME)
04/22/15Centre for Medical Education, Karolinska Institutet

2. Summary
 Students’ learning is guided by examinations
 Formative assessment can facilitate learning
 Examination at PhD level is oftentimes too basic
 As course leader you can influence the design of your course
and the ratio of teaching and examination.
04/22/15Centre for Medical Education, Karolinska Institutet

3. Do you remember
 All Boys Should Come Home Please
04/22/15Unit for Medical Education, Karolinska Institutet

4. 04/22/15Centre for Medical Education, Karolinska Institutet
Aim
 Participants will analyse their courses in terms of constructive
alignment, specifically emphasising assessment forms and
share experiences and ideas for development.

5. 04/22/15Centre for Medical Education, Karolinska Institutet
Learning outcomes:
After the session you will have an understanding of:
• The function of assessment
• The effect of assessment on student learning
• Formative and summative assessment
• ”Backwash effect”
• SOLO taxonomy
• Criteria and norm-referenced assessment
And you will be able to:
Distinguish between formative and summative assessment
Set up a grading scheme for a number of objectives
Distinguish between criteria referenced and norm-referenced assessment

6. 04/22/15Centre for Medical Education, Karolinska Institutet
Discuss:
(2 mins)
1. Positive experiences of
assessing students’ work
2. What concerns/ queries do
you have about assessment?
Share your discussion with the
group

7. 04/22/15Centre for Medical Education, Karolinska Institutet
Two kinds of assessment
Formative assessment
- Continuously runs during the teaching/ learning process
- Diagnostic: gives feedback to students and teachers on
* strengths and weaknesses
* difficulties
* misunderstandings
- Gives an opportunity to modify/ improve
Summative assessment
- Final. At the end of a course
- Descriptive. How well did the students learn the material/
knowledge/ skill
- For ranking and selection. Usually no possibility to modify/
improve

8. 04/22/15Centre for Medical Education, Karolinska Institutet 22 april 2015 8
When do we assess?
Formative
assessment Summative
assessment
Course/Education

9. 04/22/15Centre for Medical Education, Karolinska Institutet
Use 2 minutes to write down
- An example of
formative/summative assessment
that you thought was good.
Good: one or more of the
following:
Educational, Valid, Reliable, Cost
effective, acceptable to faculty &
feasible

10. 04/22/15Centre for Medical Education, Karolinska Institutet
Why do we assess our
students’ work?
Write down three of the most
important reasons for assessing
students’ work.
Turn to the person next to you.
Compare and pick three of your
reasons that you think are most
important.
Turn to another pair and pick the
three most important reasons
(snowballing technique)

11. The function of assessment
To facilitate learning:
-Find out whether students achieved the aims and goals
-Motivate students to learn (stick or carrot?)
-Make visible, assist and improve learning by giving useful
feedback.
To facilitate teaching:
-Give feedback, to give an opportunity to improve teaching
For institutional/ professional requirements:
-Grading, ranking or select students
-Maintain standards

12. 04/22/15Centre for Medical Education, Karolinska Institutet
Assessment is the main factor that
influences student learning
It is the students’ understanding of
the requirements for assessment
that makes the ”hidden curriculum”
and that has an impact on how
students learn.

13. April 22, 2015
13
Backwash effect
ILOs
Teaching
methods
Examination
Examination Study methods Student
learning

14. Implications for Teaching & Learning
 What is the ratio between teaching and examination?
 When can examination act as a catalyst for learning and
teaching?
04/22/15Centre for Medical Education, Karolinska Institutet

15. 04/22/15Centre for Medical Education, Karolinska Institutet
Structure of the Observed Learning
Objectives (SOLO)
A way of describing how learners’ performance (level) grow
in complexity when mastering academic tasks

16. Name the 6 wives of Henry VIII
 Critique your examination
 Look at your exam questions
 What type of knowledge are you asking for
 Promoting Higher order thinking
 How do the questions stand up in terms of SOLO?
04/22/15Centre for Medical Education, Karolinska Institutet

17. 04/22/15Centre for Medical Education, Karolinska Institutet
Progression in the choice of verbs
SOLO taxonomy
Identify
Make
simple
procedures
Number
Describe
List
Combine
Compare
Contrast
Explain
relationships
Analyse
Relate
Apply
Misses
the point
Theorise
Generalise
Make a
hypothesis
Reflect
(Biggs & Tang 2007)

18. Powerful examinations?
The design is supposed to create the possibility to assess:
 specific cognitive competences such as problem-solving,
including formulating questions and critical thinking,
 information competences, such as searching for relevant
information, making informed judgements, efficient use of
information, analysing data,
 communication competences, such as presenting data
communicatively, both oral and written;
 meta-cognitive competences such as self-reflection and self-
evaluation.

19. 04/22/15Centre for Medical Education, Karolinska Institutet
Alignment
 Criterion-referenced assessment:
How well students have learnt what we intended them to learn.
 Norm-referenced assessment
Comparing students performances with each other, by ranking.

20. 04/22/15Centre for Medical Education, Karolinska Institutet
Constructive alignment (Biggs, 1999)
Curriculum that we teach
Teaching methods that we use
Assessment procedures, and methods of reporting results
Climate that we create in interactions with students
Institutional climate, rules and procedures we have to follow.
Deep learning

21. Peer Learning 22 april 2015Cormac McGrath
21
Authentic learning and examination
 Authentic learning and examination gives the students the
opportunity to explore, discuss and create/construct meaningful
concepts in a contexts that reflect knowledge that is valued.
 Integrated
 Representational as opposed to
performance of understanding

22. Traditional contra Authentic
Traditional Authentic
 Selecting a Response ----------------------------------Performing a Task
 Contrived -----------------------------------------------Real-life
 Recall/Recognition ------------------------------------Construction/Application
 Teacher-structured -----------------------------------Student-structured
 Indirect Evidence ------------------------------------ Direct Evidence
15-04-22Cormac McGrath

23. 04/22/15Centre for Medical Education, Karolinska Institutet
Traditional assessment
 Emphasis on product
 Is fair (?)
 Is economical
 Encourages repetition of large amounts of literature, which can
provide opportunity for overview of the subject
Criticism:
 Encourages quick answers rather than slow investigation
 Correctness rather than risk taking
 Memory rather than conceptual development
 Reproduction rather than creative application
 Little space for feedback
 Difficult to measure deep understanding of concepts and
context (Lindström, 1997)

24. Towards progressive
Performances of Understanding
Scenario/case
Individual Practical Exam
Master Programme in Medical Biology,
5th
semester spring term -04
Jan and Birgitta have, during some time, tried to conceive a sibling to Lisa, 4
years, without succeeding. The couple had to try quite some time before Birgitta
became pregnant with Lisa but this time it has taken more than a year without
success. Birgitta’s situation has, since she wet back to work three years ago,
changed radically. She has been working really hard and been promoted to an
intermediate executive position. The work has meant a lot of meetings,
representations, and increased responsibility at the same time as it has been
very stimulating. This has lead to a substantial increase in weight, irregularities
in her menstrual cycle, and drastic drop in time spent with the family. After more
than one year of trying to conceive Birgitta decides to seek help. Physical testing
confirms her weight gain, mainly as abdominal fat, and drop in cardio vascular
fitness. Blood test also shows high blood fat values. She receives the
recommendation to change diet and starting to take care of herself.

25. Performance of understanding
 Part I. Day I

 Do a general problem definition based on the included scenario
 Formulate as many sub problems/questions as you consider motivated.
Motivate shortly way you believe your sub problems are relevant and how
you relate them to general problem.
 Decide up on and point out one sub problem that you will penetrate in
detailed during day II.
 Do a work plan for how you intend to handle your sub problem. It should be
clear on what you need to find out, how you intend to find it out and it
should also indicate how you intend to present it.
 Hand in your answers.
 Day two…

26. Powerful examinations?
The design is supposed to create the possibility to assess:
 specific cognitive competences such as problem-solving,
including formulating questions and critical thinking,
 information competences, such as searching for relevant
information, making informed judgements, efficient use of
information, analysing data,
 communication competences, such as presenting data
communicatively, both oral and written;
 meta-cognitive competences such as self-reflection and self-
evaluation.

27. Just a thought!
 How do we embed generic skills in the context of performances
of understanding allows students to see why such skills are
important
 How do we contextualize examination questions?
 If we re-conceptualise examinations as performances, would
this allow an opportunity for re-drafting course design?
 Do we want to?

28. Performance of understanding
 Presenting a problem that requires
 Problem identification
 Decision making
 Inferences inductive-deductive
 Divergent thinking skills
 Evaluating thinking skills
 Is Genuine and authentic
 Requires reasoning

29. Discussion I
 Look at a colleagues examination,
 Where is the focus, content, skills, performance?
 Give examples, elaborate and justify
 5 min preparation
 15 ventilation

30. Closing Thoughts on Alignment
 Is your teaching aligned?
 Goals-Teaching and learning activities-
Examination
 How do we achieve inter-assessor reliability?
 How do we get the students to understand the assessment
criteria?

31. The muddiest point
 Take a post it note and write down one or a few things that still
seem unclear to you, post on the reverse side of the door as
you leave for the reading log task.
04/22/15Centre for Medical Education, Karolinska Institutet

32. Examination and assessment and
meaningful learning
 To which extent does your examination support meaningful
learning?
 To which extent does your examination allow your students the
opportunity to demonstrate that they have achieved the ILOs?
Centre for Medical Education, Karolinska Institutet

33. Discussion
 To which extent do the assessment criteria express distinct
qualitative differences?
 Can you give examples?
 How do you share this with the students?
 How well do you discuss examination and
assessment/assessment criteria?
 Individual preparation 5 min
 Group discussions 10 min

34. Summary
 Students learning is guided by examinations
 Formative assessment can facilitate learning
 Examination at PhD level is oftentimes too basic
 As course leader you can influence the design of your course
and the ratio of teaching and examination.
04/22/15Centre for Medical Education, Karolinska Institutet

35. 04/22/15Centre for Medical Education, Karolinska Institutet
References
 Biggs, J. &C. Tang (2007). Teaching for Quality Learning at
University. SRHE & Open University Press, Buckingham.
 Carroll, J. &C.-M. Zetterling (2009). Guiding students away from
plagiarism. KTH Learning Lab, Stockholm.
 McConnell, DA., Steer, DN., & Owens, KD. (2003) Assessment
and active learning strategies for introductory geology courses.
Journal of Geoscience Education, v. 51 n:o 2, p. 205-216

36. 04/22/15Centre for Medical Education, Karolinska Institutet
Links
 Concept maps:
http://users.edte.utwente.nl/lanzing/cm_home.htm
 Venn diagrams:
http://www.enchantedlearning.com/graphicorganizers/venn/