This document analyzes different methods for preparing for the USMLE Step 1 exam. The author conducted surveys of medical students at UT Health Science Center (UTHSC) about their study methods and Step 1 scores. Regression analysis found that using High Yield review materials had a positive impact on scores, while group tutoring sessions and checking out books from the student academic support services had negative impacts. Surveys of students on an online forum found some differences in common study resources compared to UTHSC students. The author concludes that specific study resources may have little effect and recommends schools provide alternative prep support and consider curricular reforms. Future research with multiple schools and better controls is needed.
1. +
Analysis of Step 1 Prep
Methods
Jenny Loome
Summer Educational Research Fellowship
2. +
What is Step 1?
Step 1 of United States Medical Licensing Examination
End of M2 year
High stakes
Intended only for licensing purposes
Used for selecting residents
3. +
Research Questions
What are best practices for Step 1 prep?
Focusing on extracurricular study
How are worldwide students preparing for Step 1?
How are UT students preparing for Step 1?
How effective are common preparations methods?
4. +
Background
Anxiety causes indiscriminate use of many anti-anxiety
resources (Encandela et al. 2014)
Can’t take student perceptions as fact
Students use huge range of test-prep resources
Lack of recent third-party studies
Old evidence of ineffective or detrimental commercial resources
(Thadani et al. 2000, Werner et al. 2003)
Old evidence of effective free/curriculum-based resources (Thadani
et al. 2000)
5. +
UTHSC Survey
Students completed surveys after Step 1 2012-2013
Test prep resources
Practice tests
Step 1 scores
6. +
Survey Analysis
Repeated regressions until p≤0.05 for all variables
Eliminated one variable at a time
Multivariate linear regressions on the variables of p<0.200.
Controlled for NBME score and year
Univariate analyses v. Step 1 score and ∆score.
U Mann Whitney for binary variables
Linear regressions for continuous
variables
Chose variables
Present both years and relevant to test prep
7. +
Results from UT
Significant Variables
Model
Unstandardized
Coefficients
Standardized
Coefficients
t Sig.B Std. Error Beta
1 (Constant)
-21333.807 4308.576 -4.951 .000
Year
10.657 2.140 .278 4.981 .000
NBME score
.598 .044 .770
13.68
8
.000
Group tutor sessions
-18.315 6.539 -.155 -2.801 .006
Check out SASS
books
-4.305 2.132 -.108 -2.019 .046
High Yield
7.513 3.611 .112 2.081 .040
a. Dependent Variable: Step 1 score
8. +
Results from UT
Insignificant variables:
First Aid
DIT
USMLERx
UWorld Q Bank
Kaplan Q Bank
½-length practice exams
Full-length practice exams
Any practice exams
Goljan Rapid Review
Flashcards
Microbiology Made Ridiculously
Simple
Step-up to Step 1
Anatomy review
Making a schedule with and
educational specialist
Meeting with specialist about test
anxiety
Number of resources
9. +
Online Source Analysis
On Dr-Ed listserv, recorded reports of institutional
test-prep methods
Anecdotal evidence
On the Student Doctor Network (Official 2012/13
Step 1 Experiences and Scores), recorded voluntary
reports of study methods
Used t-tests to compare study resource frequencies on
SDN and UTHSC
10. +
DR-ED Results
Two schools provided free access to online question banks
Six schools use mandatory custom NBMEs in their curricula.
Four schools provide their students access to CBSEs for free.
Nine schools use mandatory CBSEs as evaluations in their
curricula.
CBSEs and NBMEs are often used to screen for students at
risk of failing Step 1 and provide them interventions.
11. +
SDN Results
First Aid
DIT
USMLE Rx
Kaplan Question Bank
Practice exams
Goljan
Flashcards
Step-up to USMLE Step 1
USMLE World Question Bank
Microbiology Made Ridiculously
Simple
Number of resources
Different from UTHSC Similar to UTHSC
12. +
SDN Results
Comparing Frequencies at UTHSC and SDN
Source N Mean Std. Deviation
Std. Error
Mean
P value
1st_Aid SDN
85 .918 .2765 .0300
.018
UTHSC
132 .992 .0870 .0076
DIT SDN
85 .176 .3835 .0416
.000
UTHSC
132 .432 .4972 .0433
USMLERx SDN
85 .188 .3932 .0427
.003
UTHSC
132 .045 .2091 .0182
World_QB SDN
85 .988 .1085 .0118
.377
UTHSC
132 .970 .1721 .0150
Kap_QB SDN
85 .494 .5029 .0546
.000
UTHSC
132 .121 .3276 .0285
13. +
SDN Results
Comparing Frequencies at UTHSC and SDN
Source N Mean Std. Deviation
Std. Error
Mean
P value
Total practice
exams
SDN
85 .73 .447 .048
.001
UTHSC
132 .91 .289 .025
Goljan RR SDN
85 .506 .5029 .0546
.023
UTHSC
132 .348 .4783 .0416
Flashcards SDN
85 .153 .3621 .0393
.043
UTHSC
132 .265 .4431 .0386
MMRS SDN
85 .094 .2937 .0319
.437
UTHSC
132 .129 .3362 .0293
Step-up SDN
85 .000 .0000 .0000
.014
UTHSC
132 .045 .2091 .0182
Number of
Resources
SDN
85 4.25 1.214 .132
.986
UTHSC
132 4.25 1.162 .101
14. +
Findings
Only High Yield had a significant positive impact at UT
Using more resources did not result in higher scores
Possible explanations:
Specific resources do little to affect scores
Extracurricular study during designated time has little effect
Med schools report very little Step 1 prep
Students report full-time Step 1 prep
15. +
Improvements
Schools provide alternative Step 1 prep
Peer-led review (Alcamo et al. 2010)
Encourage use of class-related instead of commercial resources
(Thadani et al. 2010)
Provide anti-anxiety planning (Encandela et al. 2014)
Curricular reform
Studies have varying results (Cuddy et al. 2013, Williams et al.
2011)
16. +
Future Research
Survey of student test prep methods
Multiple schools
Designed for this study
Consistent across years
Include MCAT scores
Better control groups
17. +
Works cited
Alcamo AM, Davids AR, Way DP, Lynn DJ, Vandre DD. The Impact of a Peer-
Designed and -Led USMLE Step 1 Review Course: Improvement in Preparation and
Scores. Academic Medicine. 2010;85(10):S45-S48.
Cuddy MM, Swanson DB, Drake RL, Pawlina W. Changes in anatomy instruction and
USMLE performance: Empirical evidence on the absence of a relationship.
Anatomical Sciences Education. 2013;6(1):3-10.
Encandela J, Gibson C, Angoff N, Leydon G, Green M. Characteristics of test anxiety
among medical students and congruence of strategies to address it. Medical
Education Online. 2014;19.
Thadani RA, Swanson DB, Galbraith RM. A Preliminary Analysis of Different
Approaches to Preparing for the USMLE Step 1. Academic Medicine.
2000;75(10):S40-S42.
Werner LS, Bull BS. The effect of three commercial coaching courses on Step One
USMLE performance. Medical Education. 2003;37(6):527-531.
Williams PCMD, Epps ACP, McCammon SM. The Strategic Impact of a Changing
Curriculum and Learning Environment on Medical Students' Academic Performance.
Journal of the National Medical Association. 2011;103(9/10):802-810.