This document summarizes a project between 3 Dutch medical schools to develop computer-based assessments of clinical reasoning. The project aims to reduce workload for teachers while improving assessment quality. Literature identified Extended Matching Questions and Comprehensive Integrative Puzzles as suitable formats. Teachers were trained on question development. A prototype system was piloted with positive feedback. Future work includes developing more questions and implementing assessments in curricula.

1. Computer Based Assessment
of Clinical Reasoning
A cooperative project between 3 Dutch
medical schools
Mathijs Doets, MSc
Erasmus University Medical Center
(Rotterdam, The Netherlands)

2. Presentation outline
• Project overview and goals
• Clinical reasoning in medical education
• Project activities and results
• Pilots
• Future directions

3. Project participants
AMC Amsterdam
2.300 medical students
250 doctors graduate yearly
UMC Utrecht
2.100 medical students
275 doctors graduate yearly
Erasmus MC Rotterdam
2.400 medical students
280 doctors graduate yearly
Organisation for ICT in higher education

4. Project overview
• March 2011 – March 2013
• Different phases:
– Literature study
– Determine formats for assessment
– Training of teachers
– Development of questions
– Technical infrastructure
– Pilots
– Implementation in curriculum

5. What is clinical reasoning?
Clinical reasoning is a core competence of a doctor and a
key element of medical education
Definition
The process through which the physician defines a patient's
most probable diagnosis and determines further policy
To answer a clinical reasoning question, students should
have underlying knowledge and should be able to apply this
knowledge to solve a patient's problem

6. Assessment of clinical reasoning
Assessment of clinical reasoning is very often paper based
or oral
Disadvantages
•Very labour intensive
•Oral exams often subjective
if not very well structured

7. Aim of the project
Reduce workload for teachers (and staff) in high quality
assessments of clinical reasoning,
by developing a more objective and more efficient form
of assessment of clinical reasoning
Solution
Joint development of questions
•collaborating per discipline
•analysis and validation of test items
Making the assessments computer based
•creating a shared question database
•developing good digital formats

8. Literature study
Research question: Which questions types are preferred
from an educational and practical point of view and are
suitable for CBA of clinical reasoning in large groups of
undergraduate medical students?
9 suitable question types were identified for assessment
of clinical reasoning
•Extended Matching Question (EMQ)
•Comprehensive Integrative Puzzle (CIP)
•Script Concordance Test (SCT)
•Modified Essay Question (MEQ)
•Short Answer Question (SAQ)
•Long Menu Question (LMQ)
•Multiple Choice Question (MCQ)
•(Multiple) True/false Question (MTFQ)
•Virtual Patients (VP)

9. Summary of findings
• All question types have a different focus and assess different aspects of
clinical reasoning
• Regardless of the chosen question type, patient vignettes should be used
as a standard stimulus format to assess clinical reasoning.
• MEQ and SAQ can not be rated automatically and therefore do not reduce
workload
• LMQ are not suitable for more than one word answers
• Validity of MTFQ and VP is questionable
• Differentiation in scoring generates higher discrimination values ​and
increases reliability (SCT and CIP)
• The need to have a panel to identify scoring instructions for SCT may be a
challenge
• Teachers tend to test knowledge only by MCQ

10. Conclusions literature study
• Combine CIP en EMQ to assess clinical reasoning,
because this combination:
– covers most aspects of clinical reasoning
– produces valid and reliable test results
– are suitable for use in CBA (automatic scoring)
• Regardless of the chosen question type, patient
vignettes should be used as a standard stimulus format
You are a physician and are seeing a 52 year old man
• Article to be published in Dutch Journal forand cough productive
He has had increasing dyspnea Medical
Education of purulent sputum for 2 days (...)
Which of the following is the most likely diagnosis?

11. Extended Matching Question (EMQ)
• Starting with a theme
or clinical problem
• A list with 8-15 options
• Question
• 2 or 3 patient vignettes
• Scoring: points for each
correct answer; multiple
answers may be correct
EMQ example: Case and Swanson, 1996

12. Comprehensive Integrative Puzzle (CIP)
• Combines data from: medical history, physical examination, diagnostic
tests into a logical, coherent patient case, given a diagnosis (1st column).
Treatment
• Matrix
from 4x4 to 7x7 rows/columns
• Alternatives are presented for each column
• Answer options may be used once, more than once, or not at all
(reduces guessing)
• Scoring: one point per cell of points for complete row: to be determined in
pilot

13. Training of teachers
• Online instruction on writing good patient vignettes and
constructing CIP and EMQ questions
• Workshop for teachers

14. Technical infrastructure:
Store and share questions
• Search for a system which:
– Stores CIP and EMQ
– Allows searching by discipline and topic
– Supports reviewing questions between medical schools
• Cooperation with IMS from april 2012
• Implementation of CIP summer 2012
• Pilot with users september-december 2012
• Export to assesment systems: QMP, Testvision, Blackboard

15. Pilot assessment
55 (clinical) students from all 3 medical schools did a
(voluntary) assessment in a prototype assessment system,
consisting of 4 CIP and 11 EMQ from different topics
•Students were enthousiastic
•Assessment was moderately reliable (alpha 0,625)
•Differences between difficulty and discriminatory power of
items
•Complete analysis available
in the coming months

16. Project future activities
• Development of more questions
• Entering and classifying questions in IMS
• Reviewing questions through IMS
• Implementation of assessments in curriculum
– Clinical part (internships)
– Different choices for each medical school (by
discipline or integrated assessments)
• Lasting cooperation beyond the project

17. Questions?

18. References
• Ber, R. (2003). The CIP (Comprehensive Integrative Puzzle) assessment method. Medical Teacher, 25,
171-176.
• Beullens, J., Struyf, E., & Van Damme, B. (2005) Do extended matching multiple-choice questions measure
clinical reasoning? Medical Education, 39, 410-417.
• Beullens, J., van Damme, B., Jaspaert, H., Janssen, P.J. (2002) Are extended-matching multiple-choice
items appropriate for a final test in medical education? Medical Teacher, 24 (4) 390-395.
• Bhakta, B., Tennant, A., Horton, M., Lawton, G., Andrich, D. (2005) Using item response theory to explore
the psychometric properties of extended matching questions examination in undergraduate medical
education. BMC Medical Education, 5: 1-9.
• Case, S.M., Swanson, D.B., Ripkey, D.R. (1994) Multiple-Choice Question Strategies. Comparison of Items
in Five-option and Extended-matching Formats for Assessment of Diagnostic Skills. Academic Medicine, 69
(10), 1-3.
• Coderre, S.P., Harasym, P., Mandin, H., Fick, G. (2004) The impact of two multiple-choice question formats
on the problem-solving strategies used by novices and experts. BMC Medical Education, 4 (23), 1-9.
• Groothoff, J.W., Frenkel, J., Tytgat, G.A., Vreede, W.B., Bosman, D.K., ten Cate, O.Th. J. (2008) Growth of
analytical thinking skills over time as measured with the MATCH test. Medical Education, 42 (10),
1037-1043.
• Samuels A (2006) Extended Matching Questions and the Royal Australian and New Zealand College of
Psychiatrists written examination: an overview. Australasian Psychiatry, 14, (1), 63-66.
• Schuwirth, L.W.T., van der Vleuten, C.P.M. (2003) ABC of learning and teaching in medicine. Written
assessment, BMJ (326), 643-645.
• Schuwirth, L.W.T., van der Vleuten, C.P.M. (2004) Different written assessment methods: what can be said
about their strengths and weaknesses? Medical Education (38), 974-979.