Ben and Frank were just at Dartmouth Medical School taking about the science behind Gunner's approach. Check out the presentation and let us know if you like free Pizza (i.e. if you'd like us to visit your school to talk about USMLE prep, match, etc.)! :)

1. Memory Science in
Medical Education
Frank Lau, MD
Co-Founder & Chief Medical Officer
GunnerTraining.com

2. The Challenge for Medical Students
“I feel like every time I
review something or learn something new, I
forget something else.”
– C.C., 2nd year medical student,
University of Michigan Medical School

3. The Challenge Persists for Residents
 Resuscitation, July 2009
 70 Johns Hopkins pediatrics residents
 66% failed to start timely CPR (basic life support)
 95% of the residents had received this level of training
 83% made errors during defibrillation (shocking the patient)
 80% of the residents had been trained in defibrillation
 More years of training did not improve physician performance
Medical knowledge is routinely forgotten,
even by the best and the brightest.

4. The Challenge Persists for Attending
Physicians
 Journal of Trauma, June 1996
 60 practicing trauma physicians took the standard Advanced
Trauma Life Support course, followed by an examination
 National course for managing traumas (car accidents, gunshots, stab
wounds, etc)
 Immediate testing: 86% average score
 6 months later, 50% failed
 13 years after this study, no novel approaches to learning and
retaining medical knowledge
Basic, critical knowledge is routinely forgotten,
even by experienced specialists.

5. Who Should Care?
 Medical Students
 Physicians-in-training (residents)
 Educators
 Foreign medical graduates
 Practicing physicians
 Patients
 Healthcare policy makers
 Patient safety advocates

6. The 2 Phases of Knowledge Acquisition
Mastering the
material
Maintaining that
knowledge

7. Mastering the Material
 We think we’re good at this
 We did well on the MCAT, we pass our exams
 But these are graded on a curve!
 Patient outcomes don’t work on a curve
 In reality, we do poorly at judging how well we’ve
mastered specific topics

8. Mastery: The Problem
32%
of the
reviewed
material is
immediately
forgotten.

9. Mastery: The Problem

10. Maintaining Your Knowledge
 We rarely focus on knowledge maintenance
 Johns Hopkins study, the ATLS study
 Affects patient outcomes
 For Step 1, the review process involves re-learning a lot of
material that you once knew but have since forgotten

11. Intro to Gunner Training
 The first Knowledge Management platform
 Created to help students master and maintain their Step 1
knowledge base
 Personalized: users choose which subjects to study  can
correlate with school's curriculum
 Dynamic: custom review schedule changes according to
student's forward learning
 Mobile:
 iPhone/mobile browser compatible
 No software installation required

12. Intro to Gunner Training
 Launched April 2009:
 Hundreds of users
 Thousands of data points
 We use this data and user feedback to continually validate
and improve Gunner Training

13. The GT Way
1. Learn the material from flashcards
• High-yield, interactive
2. Immediate testing on that material
• Open-ended study questions
3. Spaced Learning algorithms create a personalized review
schedule for that material
• Incorporates Step 1 style multiple choice questions
• Short intervals help you master that material
• Longer intervals maintain your knowledge
4. Exam simulation
• Eases anxiety
• Increases test-taking stamina

14. The GT Way
1. Learn the material from flashcards
• High-yield, interactive
2. Immediate testing on that material
• Open-ended study questions Apply 19
3. Spaced Learning algorithms create a principles of
personalized review schedule for that
material efficient,
• Incorporates Step 1 style multiple
choice questions effective
• Short intervals help you master that
material learning
• Longer intervals maintain your
knowledge
4. Exam simulation
• Eases anxiety
• Increases test-taking stamina

15. GT: Flashcard Learning Principles
1. Contiguity Effects. Ideas that are associated should be
presented contiguously in space and time
 One flashcard per topic, rather than one-fact-per-flashcard
2. Dual Code and Multimedia Effects. Visual + multimedia
form richer representations vs. a single medium
 Hundreds of relevant images: gram stains, CT scans, X-rays,
clinical photos
3. Coherence Effect. Materials should explicitly link related
ideas

16. GT: Flashcard Learning Principles
4. Segmentation Principle. A complex lesson should be broken
down into manageable subparts
 Two levels of bulletpoints
5. Manageable Cognitive Load. The information presented to
the learner should not overload working memory
 Second bulletpoints are initially hidden
6. Cognitive Flexibility. This improves w/ multiple viewpoints
linking facts and deep conceptual principles
 Material presented in several locations
 Ex: TB is covered under Microbiology, Pulmonology, Immunology
7. Anchored Learning. Materials are anchored in real world
problems that you care about
 Clinical correlation

17. Contiguity, Dual code/Multimedia, &
Coherence Effects

18. Segmentation & Manageable Cognitive Load

19. Cognitive Flexibility

20. Anchored Learning

21. GT: Study Question Principles
8. Imperfect Metacognition. We rarely have an accurate
knowledge of our cognition  need assistance with
calibrating comprehension, learning, and memory
 Immediate testing solves this
9. Generation Effect. Learning is enhanced when learners
produce answers (vs. recognizing answers)
 This is why GT employs open-ended study questions
10. Testing Effect. Testing enhances learning, especially when
the tests are aligned with important content

22. GT: Study Question Principles
11. Feedback Effects. Students benefit from immediate
feedback on their performance
12. Negative Suggestion Effects. Learning wrong information
can be reduced when feedback is immediate
13. Deep questions. More benefit from answering questions
that elicit explanations (e.g., why, why not, how, what-if)

23. Imperfect Metacognition, Generation Effect
& Testing Effect

24. Feedback Effects & Negative Suggestion
Effects

25. GT: Review Schedule Principles
14. Goldilocks Principle. Assignments should be at the right
difficulty level
15. Self-regulated Learning. Most students need to self-
regulate their learning
16. Exam Expectations. Students benefit more from
repeated testing
17. Spaced Learning.

26. Goldilocks, Self-regulated Learning, and
Exam Expectations

27. GT: MCQ Principles
18. Desirable difficulties. Challenges make learning and
retrieval effortful  improve retention
 Multi-part questions, on par with Step 1 question difficulty
20. Cognitive Disequilibrium. Deep reasoning and learning is
stimulated by problems that create cognitive
disequilibrium, such as obstacles to goals, contradictions,
conflict, and anomalies
 “2-jump” & “3-jump” questions

28. MCQs in Review Schedule

29. MCQs in Review Schedule

30. Spaced Learning
in Gunner Training
Personalized Review Schedule Generation

31. Spaced Learning
 In 1913, Ebbinghaus discovered that the best time to
review something was just before it was forgotten
 Minimizes the amount of time spent learning
 Minimizes the number of times the same material must be
relearned

32. Spaced Learning: Example
 Let’s say it takes 2 minutes to memorize the 15 bones of
the wrist
 Reviewing this material if it’s not forgotten takes 30
seconds
 Re-learning this material if it’s forgotten takes another 2
minutes
 Without review, this information is forgotten in 4 days
 There is a test at the end of Day 6

33. Spaced Learning: Example
 Option 1: Review it every day
 Day 0: 2 minutes learning
 Day 1: 30 second review
 Day 2: 30 second review
 Day 3: 30 second review
 Day 4: 30 second review
 Day 5: 30 second review
 Day 6: 30 second review  Take the test
 Total time: 5 minutes
 Pros: In this approach, the knowledge is never forgotten
 Cons: Large burden of review  few people will adhere to this
review schedule

34. Spaced Learning: Example
 Option 2: Cram for the exam
 Day 0: 2 minutes learning
 Day 1: no time spent
 Day 2: no time spent
 Day 3: no time spent
 Day 4: no time spent (knowledge forgotten)
 Day 5: no time spent
 Day 6: 2 minutes re-learning  Take the test
 Total time: 4 minutes
 Pros: It’s “easy” – learn it once and re-learn it just before the
test
 Cons: The knowledge is forgotten
 Cramming produces high stress
 Doesn’t work in clinical settings (no predetermined “test” date)
 Less effective as the volume of information goes up

35. Spaced Learning: Example
 Option 3: Spaced Learning
 Day 0: 2 minutes learning
 Day 1: no time spent
 Day 2: no time spent
 Day 3: no time spent
 Day 4: 30 second review
 Day 5: no time spent
 Day 6: Take the test
 Total time: 2 minutes, 30 seconds
 Compared to daily review (5 mins): 50% less time spent
 Compared to cramming (4 mins): 38.5% less time spent
 Pros: In this approach, information is never forgotten and
time spent studying is minimized

36. Spaced Learning Works in Medicine
 3 randomized, controlled trials at Harvard Medical School
demonstrate that Spaced Learning improves retention of
medical knowledge
 RCTs are the gold standard experimental methodology
 Published in:
 American Journal of Surgery, January 2009
 Journal of General Internal Medicine, January 2008
 Medical Education, January 2007

37. Basic Research on Spaced Learning
 The molecular mechanism behind Spaced Learning is an
active area of research
 Cell, October 2009. Cold Spring Harbor & Mount Sinai School
of Medicine
 The Phosphatase SHP2 Regulates the Spacing Effect for Long-Term Memory Induction
 Journal of Neuroscience, August 2009. McGill University
 PKC Differentially Translocates during Spaced and Massed Training in Alypsia
 Neuron, December 2006. Baylor College of Medicine
 Drosophila Mushroom Body Neurons Form a Branch-Specific, Long-Term Cellular
Memory Trace after Spaced Olfactory Conditioning
There is rock-solid science behind Spaced Learning.

38. Gunner Training Works!

39. Gunner Training’s Effectiveness
*
19.3%
(p<0.000001)**
* All data where user completed 4 reviews within the maximum time (115 days)
** Heteroscedastic, one-tailed t-test. Same results with paired, 2-tailed t-test.

40. Gunner Training’s Effectiveness
Avg time to 4th
review: 26.4 days
Max time to 4th
review: 115 days
 Two major successes:
 1) Knowledge isn’t lost during >100 days since initial review
 2) 19.3% gain in retained knowledge

41. Summary
 GT is the first Knowledge Management specifically
platform for medical students
 The application of 19 Learning Principles yields a highly
efficient and effective program
 Users who follow our algorithms gain an average of 19.3%
in ability to recall knowledge after 4 reviews
 Users don’t forget what they have already learned

42. Priced for Students

43. User Comments
 “I absolutely loved this site. I  “I'll definitely let the 2nd years
didn't get far into the program in my school know about
because of time constraints, but your program. I've already
I really wish I knew about it recommended the program to
before starting 2nd year. It the Dean of Student Affairs at
would've been amazing to use my school.”
it while studying for the – J.W., 2nd year
course. I used to study with
medical student
RecallPlus and Anki during the
course but found it so time
consuming to make the cards,
so you can say Gunner Training
is just perfect.”
– H.A., 2nd year
medical student

44. Anonymous Comments
 It's surprising how much I've learned  “On my exam, there were questions
in such a short time span with what that I actually thought, ‘I know that
feels like not a lot of effort. I've maybe flashcard exactly.’ This is, however,
spent 30 minutes to an hour every after I had continuously gotten the nit
other day (sometimes every 2nd or picky factoid wrong on several
3rd day) on it and it's gone quickly. occasions at Gunner Training. ”
I've been tracking my progress with Caboose, on
the analytics it provides. After 2
weeks of studying and a few Student Doctor Network
repetitions my recall has gone up
significantly (the amount of 5's has
increased by almost 20% - this
meaning that when I do my scheduled
review questions, the number of
answers that I get right without having
to think about the question is
increasing). ”
AggieSean, on
Student Doctor Network

45. http://www.gunnertraining.com