The document discusses competency-based medical education across the continuum of training. It begins by outlining the speaker's conflicts of interest in assessment-related organizations. The objectives are then stated as discussing the roles of assessment in a competency-based program, developing approaches to effective assessment, and integrating assessment into Memorial University's education system. Milestones and entrustable professional activities are presented as frameworks to assess competency development longitudinally. Challenges of assessment across training levels and principles of competency-based assessment are also addressed.

1. The Value of
Competency-based
Medical Education
Across the Continuum
Eric S. Holmboe

2. Conflicts of Interest
• Employed by the American Board of
Internal Medicine
• Serve on the boards of the National
Board of Medical Examiners and
Medbiquitous (both non-profit)
• Receive royalties from Mosby-Elsevier
for a textbook on assessment
2

3. Objectives for Today
• Discuss the roles of assessment in a
competency-based medical program
• Develop approaches to help local educators
effectively use performance-based
assessment methods and tools
• Discuss how performance-based assessment
can be integrated into an education and
assessment system for Memorial University.
3

4. Nostalgialitis Imperfecta
 Syndrome characterized by the following signs
and symptoms:
– “When I was an student…<insert superlative>”
– “Medicine was so much better 25 years ago”
• Reality: Not really…
– “Younger physicians today are less
professional, skilled, etc. because of <insert
favorite complaint>”

5. Current Model of Training and
Practice
Retirement
Competent

6. Change in Performance Over
Time
Lower Performance All Outcomes
Choudhry NK, Ann Intern Med, 2005;142:260-73

7. With your immediate
neighbors, discuss what
competency-based education
and training means to you?

8. Competency-Based Medical
Education
 Is an outcomes-based approach to the
design, implementation, assessment and
evaluation of a medical education program
using an organizing framework of
competencies
 the unit of progression is mastery of
specific knowledge, skills and attitudes
Frank, JR, Snell LS, ten Cate O, et. al. Competency-based medical
education: theory to practice. Med Teach. 2010; 32: 638–645

9. So What are the Outcomes
and
Who Determines Them?
The Profession?
The Public?
Policy Makers?

10. Determining Outcomes: Perspectives
 The Profession?
– The “core” of a discipline?
– Competence in the Can MEDS roles? Safe,
effective, patient-centered care?
 The Public?
– Trust that a doctor can do certain things?
 Policy Makers?
– Meeting the needs of the complex and aging
Canadian health population?

11. Traditional versus Competency-based:
Start with System Needs
Frenk J, et al. Health professionals for a new century: transforming
education to strengthen health systems in an interdependent world.
Lancet. 2010 11

12. U.S. Institute of Medicine
Competency Framework
IOM, 2003

13. Implications of CBME
 Curriculum and assessment follows from the
competencies and outcomes, not vice versa
 Requires:
– Definition of milestones of competency
• What does competency look like?
– Robust assessment methods, tools & systems

14. Educational Program
Variable Structure/Process Competency-based
Driving force: Content-knowledge Outcome-knowledge
curriculum acquisition application
Driving force: process Teacher Learner
Path of learning Hierarchical Non-hierarchical
(Teacher→student) (Teacher↔student)
Responsibility: content Teacher Student and Teacher
Goal of educ. encounter Knowledge acquisition Knowledge application
Typical assessment tool Single subject measure Multiple objective measures
Assessment tool Proxy Authentic (mimics real tasks of
profession)
Setting for evaluation Removed (gestalt) Direct observation
Evaluation Norm-referenced Criterion-referenced
Timing of assessment Emphasis on summative Emphasis on formative
Program completion Fixed time Variable time
Carraccio, et al. 2002.

15. Definitions and Frameworks:
The Need for a Shared
Understanding

16. Competency / Competencies
 An observable ability of a health professional
– Reflects a spectrum
– Integrates multiple components such as knowledge,
skills, values, and attitudes
– Multiple competencies can be combined
– Measureable with respect to a defined outcome
© 2009 Royal College and The International CBME Collaborators

17. Competent
 Possessing the required abilities at a
specified stage of medical education
 Is always qualified by a frame of
reference
 …Dyscompetence
 …Supracompetence
© 2009 Royal College and The International CBME Collaborators

18. Frameworks
 Webster’s New Collegiate Dictionary:
– A skeletal or structural frame
– A basic structure (as of ideas)
– Frame of reference

19. CanMEDS Framework

20. KSA Framework
K = Knowledge
S = Skill
 Information gathering skills
– Interview, physical examination, communication
 Ability to use knowledge and information
– Problem solving, clinical judgment
 Management skills
– Diagnosis, treatment, patient education,
counseling, procedural skills
A = Attitudes
 Professionalism, humanism

21. RIME Model
 Developed at USUHS
– Lou Pangaro and Gordon Noel in the 1980s
for use in third year medical student
clerkships in internal medicine
 “Synthetic” Model
– Reporter
– Interpreter
– Manager
– Educator

22. Dreyfus Developmental
Model
• Novice – Don’t know what they don’t know
• Advanced Beginner – Know what they don’t know
• Competent – Able to perform the tasks and roles of the
discipline – restricted breath and depth
• Proficient – Consistent and efficient in performance of the tasks
and roles of the discipline - know what they know and don’t know
• Expert – In depth knowledge concerning the discipline – often
rule based – know what they know
• Master – Expert who relishes the unknown, or the situation that
breaks the rules – who the experts go to for help – don’t know
what they know 1
as presented by Leach, D., modified by Nasca, T.J.
American Board of Internal Medicine Summer Retreat,
August, 1999

23. Competence Learning
 Performance is exhausting
 Sense of what’s important may be lacking
 Volume of various aspects can be
overwhelming
 Competent model of decision making:
– “is a detached, deliberative, and sometimes
agonizing selection among alternatives”
Dreyfus and Dreyfus, 1986

24. Assessment Across the
Continuum:
Challenges and Principles

25. Assessment Strategies in
Undergraduate and
Postgraduate Training

26. Milestones Definition
 A significant point in development
Merriam-Webster
 A scheduled event signifying the completion of a
major deliverable or a set of related deliverables.
mariosalaexandrou.com

27. Milestones and Trajectories
 Milestones should enable the trainee,
program and the regulatory bodies to
know an individuals trajectory of
competency acquisition.
 The focus is developmental

28. Approaches to Developing
Milestones 1
 Discrete
– Defining different behaviors in a domain of
competence at each stage of training
 Continuous
– A similar ability modified at each stage of
training to reflect increasing complexity or
sophistication
– Parameters:
• Setting, players, complexity, supervision
1
From Jason Frank, RCPSC
28

29. Approaches to Developing
Milestones 1
 Narrative
– Detailed descriptions of stages of
development of competency by domain
– Short essays and vignettes that describe a
“competence story”
 EPAs(entrustable professional activity)
1
From Jason Frank, RCPSC
29

30. Patient Care
ACGME Developmental Milestones Approximate Assessment
Competency Informing Time Frame Methods/Tools
ACGME Competencies Trainee to
Achieve
Stage
Clinical skills Historical Data Gathering Standardized
and 1. Acquire accurate and relevant history 6 months patient
reasoning from the patient in an efficiently Direct
customized, prioritized, and Observation
 Manages hypothesis driven fashion Simulation
patients using 2. Seek and obtain appropriate, verified, 9 months
clinical skills of and prioritized data from secondary
interviewing and sources (e.g. family, records,
physical pharmacy)
examination 3. Obtain relevant historical subtleties 18 months
that inform and prioritize both
differential diagnoses and diagnostic
plans, including sensitive,
complicated, and detailed information
that may not often be volunteered by
the patient
Sub-
competency

31. Communicator Milestones Project
Stage 1 Stage 2 Stage 3 Stage 4
Greets the Demonstrates Demonstrates Demonstrates Demonstrates
patient in a rule-based proficiency; proficiency; proficiency &
way; some adapts to efficiency;
Simulation or adaptation; many Across
role play routine contexts; spectrum of
clinical complex practice
clinical
An Introduction to CBME – Frank, Snell, Harris, Holmboe 2012 31

32. Patient Care
The resident is demonstrating satisfactory development of the knowledge, skill and attitudes needed to advance in training.
He/she is demonstrating a learning trajectory that anticipates the achievement of competency for unsupervised practice that
includes the delivery of safe, timely, equitable patient-centered care.
____ Yes ____ No ____ Marginal
32

33. Milestones Benefits
 Provide the learner with a clear path of
progression. There are no surprises.
 Allow for rich formative feedback. Learners
know where they are and where they need to
go.
 Define specific behaviors that can focus
assessment.

34. Milestones Challenge
 Synthesizing milestones into larger global
representations of competency that reflect those
activities that define the profession.
 These activities have been described as
entrustable professional activities or EPAs.

35. Entrustable Professional Activities
 EPAs represent the routine professional-life
activities of physicians based on their
specialty and subspecialty
 The concept of “entrustable” means:
– ‘‘a practitioner has demonstrated the necessary
knowledge, skills and attitudes to be trusted to
independently perform this activity.’’1
1
Ten Cate O, Scheele F. Competency-based postgraduate
training: can we bridge the gap between theory and
clinical practice? Acad Med. 2007; 82(6):542–547.

36. Entrustable Trainee Activities
 ETAs, or entrustable resident or student
activities, can help to define important
benchmarks in a trainee’s development
 ETAs in a training program may mean:
– A trainee has demonstrated the necessary
knowledge, skills and attitudes to be trusted to
perform this activity without constant or direct
supervision.

37. Why “ETAs” to Assess Competence?
 Sampling of events that:
- are critical moments in medical training
- inform developmental progression
- faculty and leaders already implicitly assess
- are manageable for busy training programs
– are logical of assessment for stakeholders
 Supported by generalizability theory
– 8-12 focused assessments can potentially allow
a generalized statement of competency

38. Synthesize to Physician trusted to meet
Analyze to
Educate and the health care needs of
Understand
Evaluate the population
Competency Milestones LANDMARK in EPA in
Training Practice
Medical MK1
Knowledge
MK2 Lead a resident Lead a health
care team care team
Patient Care PC1
PC2
Professionalism Prof1
Prof2 Care for clinic
patients with Practice
Interpersonal ISC1 distance
Skills independently
ISC2 supervision
Systems-based SBP1
Practice
SBP2 Complete an audit Lead Quality
of a panel of Improvement
Practice-based PBLI1 clinic patients initiative
learning
PBLI2
Shared Mental Models and Frameworks

39. Entrustment Trajectories

40. Entrustments in Newfoundland
 With a neighbor(s), discuss an
entrustment you make either with
medical students or post-graduate
trainees
 How do you arrive at this entrustment
judgment?

41. Questions?
Contact Information
eholmboe@abim.org