Assessing Procedural CompetenciesAmy S. Oxentenko, MD, FACP, FACG Associate Professor of MedicineFellowship Director, Mayo Clinic-Rochester
Outline: Procedural Competency• The Numbers• The Skills• The Future
Competent or Not?• 3rd year fellow • 3rd year fellow• 285 colonoscopies • 427 colonoscopies• Cecal intubation 95% • Cecal intubation 69%• Adenoma detection 35% • Adenoma detection 12%• Withdrawal 10 min • Withdrawal 6 min• Patient tolerance 97% • Patient tolerance 85% fair to excellent fair to excellent YES NO
What Needs to be Measured? CompetencyMinimum DefinedProcedure level of Number ability
Minimum Cited Numbers• Flex sigs 25-30• EGDs 130• Colons 140 – Based on goal of cecal intubation > 90% – Others found competence at 275 cases1 – Others found competence at 500 cases2 1 Using Sedlack data 2 Using Spier data
Spier BJ, et al. Gastrointest Endosc 2010;71:319-24.
Procedure Logs: Not Just Numbers Anymore! • “A skilled preceptor must be available to teach and supervise the fellows in the performance and interpretation of procedures, which must be documented in each fellows record, including indications, outcomes, diagnoses, and supervisor(s).”IV.A.6.d).(2) on Page 19, GI Program Requirements, “Tracked Changes” document
Procedure Logging• “Assessment of procedural competence should include a formal evaluation process and NOT be based solely on a minimum number of procedures performed.• Each program must define criteria for competence for all required and elective procedures.• The record of evaluation must include the fellow’s logbook or an equivalent method to demonstrate that each fellow has achieved competence in the performance of required procedures.”V.A.1.a).(2) and 1.b).(1).(a) on Page 20-21, GI Requirements, “Tracked Changes” document
Multi-Society Evaluation Form (MSEF)• AASLD, ACG, AGA, ASGE• Part of the GI Core Curriculum• Third Edition, May 2007http://www.asge.org/WorkArea/showcontent.aspx?id=3584
Any Downfalls of the MSEF?• Lacks anchoring characteristics for all points• Not validated for continuous assessment• Grade inflation (our problem, not the form’s)• Compare graduates across programs?• What constitutes competent?
Mayo Colonoscopy SkillsAssessment Tool (MCSAT) • 13-item survey • Staff completed on each colon • Took < 1 minute to complete • Embedded in MERGE database – Allowed for recording of procedure # for fellow, fellow name, etc. Sedlack RE. Gastrointest Endosc 2010;72:1125-33.
Overall Skill in Colonoscopy Sedlack RE. Gastrointest Endosc 2011;74:355-66.
Other DateRetrieved Per Trainee
Barriers of the MCSAT or Similar Systems?• Many procedures performed – Assessment needs to be quick/simple• Differing procedures performed – Similar models needed: EGD, PEGs, capsules, etc• Compliance with completion – Too easy for staff to forget or not take the time• Differing endoscopy database systems – No communication across programs
Sunrise: ProVation October 2012Sunset: MERGEOctober 2012
ProVation Tracking: EGD
ProVation Tracking: Colons
Pros/Cons of ProVation• PROS:• Compliance with completion – Automatic pop-up on all fellow EGDs and colons – Staff cannot sign off until complete• CONS:• Yet another database – How long with it be around? – Not everyone has it – Dependent on others to add features desired• Detail desired – Has to fit into radio buttons, brief, succinct
The Game Has Changed in the Setting of NAS • No longer a numbers game • No longer a competency yes/no game • Now it is all about meeting milestones on the way to becoming competenthttp://www.acgme-nas.org/assets/pdf/NEJMfinal.pdf
A Blueprint for Milestones or Competency? Unacceptable Competent IdealAdenoma > 20% >20%Detection RateColonoscopy 7-15 min > 6 minWithdrawal TimeCecal Intubation > 90% > 95%RateComplication Rate < 1/200 bleed <1/1000 perfPolyp Retrieval > 95% > 10 mmRate > 80% < 10 mmPatient Tolerance > 90% fair to excellent
A Blueprint for Milestones or Competency? Unacceptable Competent IdealAdenoma < 15% > 20% >20%Detection RateColonoscopy > 20 min 7-15 min > 6 minWithdrawal TimeCecal Intubation < 80% > 90% > 95%RateComplication Rate < 1/200 bleed <1/1000 perfPolyp Retrieval > 95% > 10 mmRate > 80% < 10 mmPatient Tolerance > 90% fair to excellent
Milestones and Narratives
Summary• Procedure numbers are not enough – An anchor at which competency assessment should begin• Procedure details are now needed – Indication, findings, complications• Competency tracking is required – Milestones will pave the way, and they need to be carefully developed