Procedural Assessment: Where do we stand? March 2013 Walter J. Coyle MD, FACG, FASGE
Objectives The apprentice model – The way we were Competency based education – Where we are now Outcomes based learning – Procedural focus – Milestone development
Apprenticeship Successfulfor thousands of years Key properties: – Good mentor – Motivated student – Adequate exposure AND hands on time Problems: – Consistency – Objective measures of success – Low ceiling for promotion The Mystery of Mastery. Psychology Today 1986;20:32
Apprenticeship Works well for very sub-specialized areas and few centers of excellence Still model for advanced endoscopy – AEF match – Variation in level of exposure and mastery – Who monitors the mentors? – How do the graduates do? Medicine resents outside monitoring – Better if we did it ourselves
Competency based training ACGME initiative from the 1990s – Applied to all aspects of training – Knowledge, professionalism, procedures Ineffective for procedural training – GI procedures still in apprentice model – Little consensus on assessment and outcomes – Little data to define milestones How should a 2nd year fellow scope? Lurie. Med Educ 2012;46:1365
Competency based training Diverse training methods and assessment techniques Small programs vs large; research fellows Silo mentality: no consistent standard Explains why we have this problem now
Outcomes based learning More process oriented Focus on the process not the problem ACGME wants us to move here Starting point: 1st year fellow Ending point: Staff GI Milestone development: easier for knowledge core vs procedures N Engl J Med 2012;3686:1051-56
Milestone development Final milestone: Colonoscopy – >95% cecal intubation rate – >25% ADR – Low complication – Patient satisfaction Stepwise milestones: None with great data or evidence – 1st year vs 2nd vs 3rd Gastrointest Endosc 2010;71:319-24
Procedural Education: initial focus on process Intense didactic – FYF course, DVDs, local resources Intense hands on training with scope – ? Simulators – Training box/tool – Standardized patient – Example of pilot training?
Procedural Education:subsequent focus on process Ongoing, continuous assessment: – Mentor feedback; patient feedback – Objective outcomes based assessment tool – Universal tool ? Development of outcomes based, data driven milestones that apply throughout fellowship – How???
Procedural Education: A Proposal Universal assessment tool agreed upon Web-based submission of assessments – Collection and development of milestone – Feedback to fellow and program – Fellow compared to peers nationally Progressionthrough milestones will be fellow driven, not fixed year driven
Procedural Education: A Proposal Requirements of system – Ease of use: minutes, APP for phone, link on desktop – Secure – Can provide data back to program and fellow in real time – Dynamic and progressive
Procedural Education: A Proposal Cost:GI programs, GI societies, ACGME Web site location and maintenance – ACGME – CORI database like initiative – GIQuik – Endoscopic report generating systems Provation initiative with Mayo Clinic Time frame
Integrated Assessment Are we ready and committed?? Resource Commitment Staff Commitment Barriers breaking silos Only definite: change is here
Summary Prior models and procedural mentoring are probably inadequate Classic competency based assessment is flawed for procedures Outcomes and milestones are a next step GI directed development of milestones and tools is critical
SummaryNeed to think of the process NOT the problem