BbWorld 2012 Marcus Merrin & James M. RiceExamination Challenges in a Caribbean Medical School and the Study of "Decision Fatigue" in Examinations
 

BbWorld 2012 Marcus Merrin & James M. RiceExamination Challenges in a Caribbean Medical School and the Study of "Decision Fatigue" in Examinations

on

  • 677 views

Client Presentation

Client Presentation

Statistics

Views

Total Views
677
Views on SlideShare
677
Embed Views
0

Actions

Likes
0
Downloads
5
Comments
0

0 Embeds 0

No embeds

Accessibility

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

BbWorld 2012 Marcus Merrin & James M. RiceExamination Challenges in a Caribbean Medical School and the Study of "Decision Fatigue" in Examinations BbWorld 2012 Marcus Merrin & James M. RiceExamination Challenges in a Caribbean Medical School and the Study of "Decision Fatigue" in Examinations Presentation Transcript

  • Examination Challenges in aCaribbean Medical School and theStudy of "Decision Fatigue" inExaminations Marcus Merrin Director of Educational Technology. James Rice Associate Dean of Research. American University of Antigua College of Medicine
  • Background: Who we are and what we do.• Medical School, accredited in New York and California. Also CAAM accredited• Provide medical education for candidates expecting to practice in the US• 2000 + Medical Students, 1000 in Antigua, the rest on clinical rotations in the US 2
  • Our Students• Many from non-traditional backgrounds for medical school.• Students can be admitted without MCAT scores• About 60% privately funded by self / parents• 40% in receipt of student loans 3
  • Our Challenges• University policy mandates regular in-house exams in all subjects. NBME external exams are administered at the end of every two semesters.• Some of our students are not well equipped to deal with the rigours of medical education upon entry.• Many students are under immense financial and social pressure to succeed.• Motivation to cheat on exams is high for a significant segment of our student population.• While Antigua enjoys good internet connectivity with mainland North America, brief outages are common and bandwidth is expensive. 4
  • What we try to achieve• Maximize our students’ chances of performing well on examinations• Provide a fair exam for all. Approximately 250 – 300 students are taking exams at the same time.• Appropriate accommodations for students with disabilities• Prevent students getting illicit assistance• Preserve integrity and secrecy of exam materials 5
  • What we try to achieve• Provide exams that have formative value• Expose and accustom students to the style of exams used by NBME• Deny exams to students who do not have an 80% attendance record in lectures / small group sessions or labs 6
  • Our Solutions to our Challenges• All departments prepare and upload exams using the Respondus 4.0 exam authoring software (campus-wide license)• Some faculty reuse questions with well-established performance data (à la NBME)• Some faculty prefer to write new questions for each exam.• Exams are set to be available only through the RespondusLockDown Browser• Exams are password protected 7
  • Our Solutions to our Challenges (setup)• Exams always set for one-at-a-time presentation• Adaptive release is used to exclude students whose attendance is inadequate.• Separate versions of the exams are created for students requiring testing accommodations. Adaptive release is used to ensure these versions are only visible to qualified students• Because of connectivity issues, we are forced to permit students to resume an exam in progress – this presents additional security challenges. 8
  • Our Solutions to our Challenges (examadministration)• The exam center provides hard-wired exam rooms with university supplied laptops with RespondusLockDown Browser installed• The exam center provides randomized seating assignments for each exam• Students are examined in rooms of approximately 100 students with 3 proctors per room.• Cameras are available in rooms to provide evidence that a particular student was present (not always used, but a useful threat) 9
  • Our Solutions to our Challenges (administration)• Before the students are admitted, the proctors ensure all previous logins have been reset• At the start of the exam, The exam link is made available and the password projected on LCD overhead screens.• GSM blockers have been installed to prevent students messaging the exam password to external accomplices• As soon as all students have started the exam, the password is changed. 10
  • Our Solutions to our Challenges (administration)• Students needing bathroom breaks must be chaperoned• Bathrooms are searched before exam sessions• At the conclusion of the exam, Students are not permitted to leave the room.• The exam link is closed, and feedback options are activated for the exam.• After a 15 minute review period, the review options are closed, and the students are permitted to leave the room. 11
  • 12
  • We’re going to talk about research & statistics? 13
  • Examination Challenges in a Caribbean Medical Schooland the Study of "Decision Fatigue" in Examinations• So, the experimental questions we looked at were: • Do our medical students report stress, and ―decision fatigue‖-type problems? • Are there any data to suggest ―decision fatigue‖ in our medical students? 14
  • Examination Challenges in a Caribbean Medical Schooland the Study of "Decision Fatigue" in Examinations• Do our medical students report stress, and ―decision fatigue‖ problems? • Our students? –265 AUA medical students in their first semester of medical school • Students also completed a survey after one exam examining their test preparation, amount of sleep, and test-taking strategies 15
  • Examination Challenges in a Caribbean Medical Schooland the Study of "Decision Fatigue" in Examinations Let’s look at some sleep survey dataHow many hours did you sleep last night (before the exam)?• 42.5% of students got 5 hours or less sleep• 18% got 4 hours of sleep OR LESSWhat is the usual number of hours you sleep per night?• 22% of students reported 5 hours or less sleep• 4% of students reported 4 hours of sleep or less 16
  • Examination Challenges in a Caribbean Medical Schooland the Study of "Decision Fatigue" in Examinations Let’s look at some nutrition survey data:• My breakfast today (morning of the exam) consisted of the following items (check all that apply):• 25% of the students reported consuming a beverage without caffeine• 46% of the students reported consuming a beverage with caffeine• 63% of the students consumed a carbohydrate food item• 30% of the students consumed a protein food item)• 12% of the students consumed a vegetable/fruit item 17
  • Examination Challenges in a Caribbean Medical Schooland the Study of "Decision Fatigue" in Examinations• Are there any data to suggest ―decision fatigue‖ in our medical students? • Our students? –265 AUA medical students in their first semester of medical school • The data we collected were from three internal exams in Human Structure and Function I, a course that attempts to integrate both Anatomy and Physiology • All Exams were given on Blackboard 18
  • Examination Challenges in a Caribbean Medical Schooland the Study of "Decision Fatigue" in Examinations Are there any data to suggest ―decision fatigue‖ in our medical students?• But first, a brief digression: Let’s talk about ―Decision Fatigue‖• A term coined by Dr. Roy Baumeister, a Social Psychologist at Florida State University who is interested in decision-making and the amount of . energy expended in making decisions and other types of mental activity 19
  • Examination Challenges in a Caribbean Medical Schooland the Study of "Decision Fatigue" in Examinations Some methodological concerns/issues:• One assumption is that students essentially complete exams in an ordinal manner. So they answer Item #1 first, Item #2 second, etc. We know this is not entirely true, and students do go back and answer difficult items ;• GPA may mitigate against decision fatigue—that is, if you are ―smart‖ enough, you may be able to work through any fatigue and still get the item correct• Overall item difficulty may also be a covariate. In-other-words, the ordinal location of an item may not matter; the item may be missed because it is a difficult item; and• It was important to confirm that Blackboard indeed randomizes item presentation on the exams• 20
  • Examination Challenges in a Caribbean Medical Schooland the Study of "Decision Fatigue" in Examinations Are there any data to suggest ―decision fatigue‖ in our medical students?• So, the experimental question was:• After controlling for GPA and Overall Item Difficulty, does the Order of item presentation affect whether the item was missed or not? 21
  • Examination Challenges in a Caribbean Medical Schooland the Study of "Decision Fatigue" in Examinations Are there any data to suggest ―decision fatigue‖ in our medical students?The data from four quizzes were reviewed for this analysis:• Mini-exam #2 70 mult choice questions• Mini-Exam #3 80 mult choice questions• Mini-Exam #4 (Part 1) 60 mult choice questions• Mini-Exam #4 (Part 2) 60 mult choice questions 22
  • Examination Challenges in a CaribbeanMedical School and the Study of "Decision Fatigue" in Examinations What kind of questions are being asked of our medical students? Why don’t you try a couple and find out? Welcome to Human Structure & Function I Mini-Exam #2!! 23
  • Examination Challenges in a CaribbeanMedical School and the Study of "Decision Fatigue" in Examinations A 46- year old male patient had surgery to repair the thoracic diaphragm skeletal muscle. The patient did not survive the operation. If the patient had recovered from the surgery, the muscle fibers removed during surgery would have been replaced by proliferation of which of the following cells?• a) Fibroblast• b) Monocyte• c) Satellite• d) Multiplication of mature muscle fibers 24
  • Examination Challenges in a CaribbeanMedical School and the Study of "Decision Fatigue" in Examinations A 46- year old male patient had surgery to repair the thoracic diaphragm skeletal muscle. The patient did not survive the operation. If the patient had recovered from the surgery, the muscle fibers removed during surgery would have been replaced by proliferation of which of the following cells?• a) Fibroblast• b) Monocyte• c) Satellite• d) Multiplication of mature muscle fibersItem Difficulty was 13.06% 25
  • Examination Challenges in a CaribbeanMedical School and the Study of "Decision Fatigue" in ExaminationsA 500 Daltons protein was injected into the venous systemic system of a 50 year old male patient. This protein will not be detected or present in which of the following tissues or organs?• a) Brain• b) Kidney• c) Spleen• d) Adrenal gland• e) Liver 26
  • Examination Challenges in a CaribbeanMedical School and the Study of "Decision Fatigue" in ExaminationsA 500 Daltons protein was injected into the venous systemic system of a 50 year old male patient. This protein will not be detected or present in which of the following tissues or organs?• a) Brain• b) Kidney• c) Spleen• d) Adrenal gland• e) LiverItem Difficulty is 62.69% 27
  • Examination Challenges in a CaribbeanMedical School and the Study of "Decision Fatigue" in Examinations Analysis of the data• Logistic regression• What is it? It is a statistical test used when the dependent (criterion) variable is binary (yes/no , correct/incorrect), and there are one or more continuous predictor variables or covariates. It is NOT the same as multiple regression and does not produce a standard solution, but a probabilistic solution. 28
  • Examination Challenges in a CaribbeanMedical School and the Study of "Decision Fatigue" in Examinations Analysis of the dataSo, what did we find?• Mini-exam #2—18,760 lines of data-- both covariates were significant—―Order‖ was not a significant predictor• Mini-exam #3—20,960 lines of data—both covariates AND ―Order‖ were statistically significant—p < .01• Mini-Exam #4—31,800 lines of data—both covariates were significant—―Order‖ was not a significant predictor 29
  • Examination Challenges in a CaribbeanMedical School and the Study of "Decision Fatigue" in Examinations So, what are the implications from this study?• Medical students are stressed. Many students sleep too little, have marginal nutrition, and are stressed and distracted during exams;• There is weak evidence with longer exams that ―decision fatigue‖, defined as getting questions incorrect because of the order the item is presented, may in fact exist. This effect was found (in one of three exams analyzed) even after student GPA and overall item difficulty were factored out of the model.• Since the purpose of examinations is to give medical students the opportunity to demonstrate mastery over their Basic Science coursework, AUA Administration may wish to consider more innovative strategies to provide more rest breaks during exams in order to minimize the possibility of decision fatigue so that a more accurate assessment of student performance can be obtained. 30
  • And we hope youdon’t think this was!! Thank you!! 31
  • Contributing ColleaguesDr Roslyn Haley Cecelia Downs, M.A.• Chairperson, Educational • Faculty, Educational Enhancement Department, AUA Enhancement Department, AUA College of Medicine College of Medicine 32
  • We value your feedback!Please fill out a session evaluation. 33