1. Does an Intensive Two-Week Post
Residency-Match M4 Elective
Improve Competency Prior to
Starting OBGYN Residency?
Gynecology and Obstetrics Fundamentals of Residency: Internship Training Trial
Francis S. Nuthalapaty, MD
For
Collaborative
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund
and the
Office of Philanthropy and Partnership
2. Investigators
Greenville Health System/University of South
Carolina School of Medicine
Carolinas Medical Center
•
Francis S. Nuthalapaty, MD (PI)
•
H. Lee Higdon, PhD
•
David A. Forstein, DO
•
Robert V. Higgins, MD
Medical College of Georgia
•
Palmetto Health/University of South Carolina
School of Medicine
•
Chadburn Ray, MD
•
Medical University of South Carolina
Ashlyn Savage, MD
Sarah Smith, MD
University of Tennessee at Knoxville
Florida State University College of Medicine
•
•
Julie Z. DeCesare, MD
St. Vincent’s Hospital
• Spencer G. Kuper, MD
University of Tennessee Memphis
•
Claudette Shepherd, MD
Nikki B. Zite, MD
Mayo Clinic College of Medicine
•
Brian C. Brost, MD
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
3. Disclosure
I have no relevant conflicts of
interest to disclose
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
4. Purpose of the M4 Year
Walling and Merando, Acad Med 2010
1. To facilitate completion of the
predoctoral phase of medical education
2. To transition to the residency phase of
medical education
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
5. Study Objective
To determine the impact of
participation in an intensive twoweek post residency-match M4
elective, as compared to usual
student activities, on basic clinical,
procedural, and knowledge
competencies assessed at the start
of OBGYN internship.
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
6. Study Design
Phase 1
Oct 2011 – Sept 2012
Collaborator
recruitment,
training and
IRB approval
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
7. Collaborative Centers
8 ACGME Accredited Residency Programs:
1. Greenville Health System/University of South Carolina Program | Greenville, SC
2. Carolinas Medical Center Program | Charlotte, NC
3. Florida State University College of Medicine Program | Pensacola, FL
4. Medical College of Georgia Program | Augusta, GA
5. Medical University of South Carolina Program | Charleston, SC
6. Palmetto Health/University of South Carolina School of Medicine | Columbia, SC
7. University of Tennessee Program | Memphis, TN
8. University of Tennessee Medical Center at Knoxville Program | Knoxville, TN
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
8. Study Design
Phase 1
Mar 2012 – Sept 2012
Collaborator
recruitment,
training and
IRB approval
Phase 2
Oct 2012 – Jan 2013
Participant
recruitment
during
residency
interviews
Phase 3
Phase 4
March 2013
April 2013 – June 2013
Participant
enrollment
Group
assignment,
study
interventions
and posttesting
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
10. Study Subjects
• Inclusions:
– Participated in the 2013 NRMP
– Applied to ObGyn Residency
– Matched into one of the Collaborative Centers
– Available to relocate to Primary Study Center
from April 15th – 26th, 2013
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
11. Study Subjects
• Exclusions:
– Unavailable to complete end-of-study
assessments
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System OfAice of Philanthropy and Partnership
12. Curriculum Design
• APGO/CREOG Milestone 1 Project Learning
Outcomes
– 35 Medical Knowledge
– 29 Clinical / Procedural Skills
• Ranked in order of importance
– 19 Medical Knowledge
– 14 Clinical / Procedural Skills
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
13. Clinical/Procedural Skills
1.
2.
3.
4.
5.
6.
7.
Comprehensive women’s
health history
Breast and pelvic exam
Two-handed surgical knot
Intrapartum cervical exam
Ruptured membranes
evaluation
Fetal heart rate monitor
interpretation
Pap smear and cervical
cultures
8. Names and uses of basic
surgical instruments
9. Wet mount interpretation
of vaginal discharge
10. Suture types and
properties
11. 2nd degree vaginal
laceration repair
12. Spontaneous vaginal
delivery
13. Endometrial biopsy
14. IUD Insertion
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
14. Curriculum Sources
• Search for existing teaching and learning tools
– Open-access, peer-reviewed (e.g. PubMed,
MedEdPortal)
– Professional societies (e.g. APGO, CREOG, ASCCP,
AIUM, CDC)
– Commercial learning tools (e.g. GE Healthcare)
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
15. Curriculum Delivery
1. Live didactics (12 hours)
– Lecture, case-based discussion, webinar
2. Live skills labs (50 hours)
– Self-directed online videos
3. Self-directed learning (4 hours)
– Computer-based training
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
16. Curriculum Schedule
Day 1
Day 2
Day 3
Orientation
Day 4
Day 5
Didactics (1.5 hrs)
Baseline
Skills
Assessment
Skills Labs (3.0 hrs)
Debrief
Skills Labs (3.0 hrs)
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
17. Curriculum Schedule
Day 6
Day 7
Day 8
Didactics (1.5 hrs)
Skills Labs (3.0 hrs)
Day 9
Day 10
Debrief
Final
Skills
Assessment
Skills Labs (3.0 hrs)
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
18. Study Assessments
• Medical Knowledge
– NBME ObGyn Subject Exam
– Surgical instrument identification
– Suture identification
– Surgical needle / blade identification
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
19. Study Assessments
• OSCE (Standardized Patient)
– Comprehensive women’s health history
– Breast and pelvic exam
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
20. Study Assessments
• OSAT
– Two-handed square knot
– IUD insertion
– Endometrial biopsy
– Intrapartum cervical assessment
– Spontaneous vaginal delivery
– 2nd degree vaginal laceration repair
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
21. Assessment Design
• Analysis of published guidelines, texts
• Development of anchored checklist
– Complete
– Incomplete
– Not performed
• Reviewed by 1-2 faculty for content validity
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
22. Performance Checklist
Assessment Design
Name:
Site:
CMC FSU GHS MCG MUSC USC UT-Mem UT-Knox
Cannot Judge
Not Done
Performance Checklist
Complete
Grading Key:
Complete (2 points) = Examinee performed the task meeting all criteria in the
description.
Incomplete (1 point) = Examinee performed the task, but failed to meet all criteria in the
Competency Assessment
description.
Not Done (0 points) = Examinee failed to perform the task. Vaginal Delivery
Station 3: Normal
Cannot Judge (0 points) = Unable to determine if the examinee performed the task.
Incomplete
Date:
Performance Checklist
General Approach / Interpersonal Communication Skills
Name:
2
MUSC USC UT-Mem UT-Knox
Examinee used either hand sanitizer or soap/water and proper technique to clean
his/her hands upon entering the room.
Introduction / Communication
Incomplete
Complete
Grading Key:
Upon entering the room, examinee greeted the patient, stated his/her name,
Complete (2 points) = Examinee performed the task meeting all criteria in the participate in her care.
ascertained/confirmed the patient’s name and voiced intent to
description.
Incomplete (1 point) = Examinee performed the task, but failed to meet all criteria in the
description.
Not Done (0 points) = Examinee failed to perform the task.
Labor Assessment / Preparation
Cannot Judge (0 points) = Unable to determine if the examinee performed the task.
3
Examination
Performance Checklist
Examinee verbalized intent to perform a pelvic exam, performed the exam and then
Cannot Judge
Date:
Hand Cleansing
CMC FSU GHS MCG
Not Done
1
Site:
verbalized the need to prepare for delivery.
General Approach / Interpersonal Communication Skills
Delivery table
1 4 Hand Cleansing
The examinee positioned the delivery table within arms reach.
Examinee used either hand sanitizer or soap/water and proper technique to clean
his/her hands upon entering the room.
2
Introduction / Communication
Support for this project was made possible in part by the
Universal Precautions
Upon entering the room,
the patient, stated his/her name,
APGO Medical Educationexaminee greetedFund and the to participate in her care. System Office of Philanthropy and Partnership
Endowment and voiced intent Greenville Hospital
ascertained/confirmed the patient’s name
23. Study Interventions
• Baseline Self-Assessment of Competency
• GO FOR IT Group:
– Relocated to Greenville, SC
– April 15th – 26th, 2013
– Participated in curricular activities
• Control Group:
– Usual activities
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
24. End-of-study Assessments
• NBME ObGyn Subject Exam
– June 15th – June 29th, 2013
– Prometric testing center
– Scheduled at student’s convenience
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
25. End-of-study Assessments
• OSCE / OSAT Assessments:
– June 18th – July 2nd, 2013
– Conducted at 5 of the collaborative centers
– Standardized assessment toolbox
– Video recording of all procedural stations
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
26. End-of-study Assessments
• Assessment Toolbox:
– Learning Outcomes
– Case Scenario (if applicable)
– Materials / Supplies / Personnel
– Student Instructions
– Assessment Checklist
– Photos
– Scoring Instructions (if applicable)
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
27. Data Management
• Scored immediately at time of testing by
Proctor:
– Health History OSCE
– Breast and Pelvic Exam OSCE
– Vaginal Delivery OSAT
– 2nd Degree Vaginal Laceration Repair OSAT
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
28. Data Management
• Scored post-hoc at Data Coordination
Center (DCC) via video review:
– IUD Insertion OSAT
– Endometrial Biopsy OSAT
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
29. Data Management
• Scored post-hoc at DCC:
– Square Knot
– Intrapartum Cervical Assessment
– Instruments, Sutures, Needles, Blades
Identification
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
30. Study Outcomes
• Primary:
– Composite score of all OSCE / OSAT end-ofstudy assessments
• Secondary:
– Individual OSCE / OSAT assessment scores
– NBME percentile score
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
31. Sample Size and Power
• Estimate of Primary Outcome:
– GO FOR IT Group:
80%
– Control Group:
64%
– Standard Deviation: 15%
• Alpha: .05
Power: 80%
• Two-tailed t-test
Sample Size = 15 subjects per group
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
32. Data Analysis
• All data entered into Research Electronic
Data Capture (REDCap) at DCC
• Exported to SPSS for analysis
– Descriptive statistics
– Student t-test for continuous data
– Chi-square for categorical data
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
33. Enrollment Results
42
Matched into
OBGYN
9
2
21
10
Available to
Participate in either
Study Group
Willing to
Participate with
School Permission
Able to participate
in Control Group
only
Declined
Participation
11
21
GO FOR IT
Group
Control
Group
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
34. Demographics
GO FOR IT
(N=11)
Control
(N=21)
P value
27
26
.3
Graduation in 2013
91%
100%
.4
USMLE Results
Step 1
Step 2 CK
Step 2 CS (pass)
224
241
100%
221
238
95%
.6
.5
.6
Characteristic
Age (yrs)
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
35. M3/M4 Experience
11.7 wks
9.4 wks
6.5 wks
5.4 wks
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
36. M3/M4 Experience
72.7%
77.7%
81.8%
50.0%
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
37. Self-rated Competence
GO FOR IT
(N=11)
Control
(N=21)
P value
Two-handed knot tying
2.9 ± 0.7
3.6 ± 0.7
.01
Normal vaginal delivery
1.9 ± 0.3
2.5 ± 0.5
.001
Clinical Skill / Procedure
• 1 = I am unable to perform the entire procedure under supervision
• 2 = I am able to perform the procedure under supervision
• 3 = I usually do not require supervision but may need help
occasionally
• 4 = I am competent to perform the procedure unsupervised (I can
deal with complications).
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
38. Primary Outcome
Composite Score on OSCE / OSAT
GO FOR IT
Control
70.8%
40.6%
SD =13.2
vs.
SD =19.5
p=.0003
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
45. Secondary Outcomes
Square Knot
GO FOR IT
Control
100%
85.7%
(11/11)
vs.
(18/21)
p=.5
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
46. Secondary Outcomes
NBME ObGyn Subject Exam
Overall-year percentile
GO FOR IT
Control
57.8
49.6
SD = 29.0
vs.
SD =28.9
p=.5
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
47. Conclusion
Participation in an intensive twoweek post residency-match M4
elective results in significantly
greater clinical, procedural, and
knowledge competency as
compared to the usual activities
students pursue at the conclusion of
the M4 year.
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
48. Discussion
• Greatest impact on procedural skills
• Lesser impact on medical knowledge
• Effect was sustained over 8-10 week period
between training and final testing
• Self-rated competency did not correlate with
outcomes
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
49. Weaknesses
• Quasi-experimental design
• Un-weighted procedural checklists
• Non-blinded procedural rating
• Exposure bias in GO FOR IT group
• Unknown transference to actual clinical
performance and outcomes
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
50. Strengths
• Multicenter
• Large sample size
• Heterogeneous study population
• Reproducible curriculum based on national
recommendations and non-proprietary tools
• Multiple forms of competency assessment
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
51. Recommendations
• OBGYN educators should implement
intensive residency preparation opportunities
near the end of the M4 year to assure
achievement of desired competency prior to
starting residency
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
53. Investigators
Greenville Health System/University of South
Carolina School of Medicine
Carolinas Medical Center
•
Francis S. Nuthalapaty, MD (PI)
•
H. Lee Higdon, PhD
•
David A. Forstein, DO
•
Robert V. Higgins, MD
Medical College of Georgia
•
Palmetto Health/University of South Carolina
School of Medicine
•
Chadburn Ray, MD
•
Medical University of South Carolina
Ashlyn Savage, MD
Sarah Smith, MD
University of Tennessee at Knoxville
Florida State University College of Medicine
•
•
Julie Z. DeCesare, MD
St. Vincent’s Hospital
• Spencer G. Kuper, MD
University of Tennessee Memphis
•
Claudette Shepherd, MD
Nikki B. Zite, MD
Mayo Clinic College of Medicine
•
Brian C. Brost, MD
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
54. Course Length/Content
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
55. Course Timing
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
56. Financial Responsibility
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership
57. Participant Comments
“Thank you Thank you Thank you. You have
saved my butt. I didn't realize how much I had
forgotten and how much I didn't know. You have
made me feel much more confident and
competent going into July.”
“The amount of confidence gained from this
course alone has made it worthwhile. I don't feel
as much as an imposter anymore :)”
Support for this project was made possible in part by the
APGO Medical Education Endowment Fund and the Greenville Hospital System Office of Philanthropy and Partnership