Three overlapping themes are discussed for effectively assessing competency: 1) Overcoming unintended consequences by reducing emphasis on exams as hurdles and promoting accountability for demonstration of learning. This involves quality improvement activities and using licensing data to facilitate learning plans. 2) Turning quality assurance into quality improvement by further integrating assessment across training with attention to improvement. This involves a formative testing platform and diagnostic assessments to feed data. 3) Ensuring authenticity by using portfolio-supported workplace assessments and increasing real world uncertainties in assessments. Examples include sequential OSCE stations and requiring reflection on alternative actions.

1. Competency-based assessment:
The good, the bad, and the
puzzling
Kevin W. Eva
Medical Education Assessment
Advisory Committee

2. The Medical Education Assessment
Advisory Committee
• Georges Bordage
• Craig Campbell
• Robert Galbraith
• Shiphra Ginsburg
• Eric Holmboe
• Glenn Regehr

3. Our charge

5. 1. Defining the
“complete”
health care
professional

6. “We demand ever more of
the latter even as we
bemoan its dominance,
and yearn for the former
while remaining wary of
its inefficiencies and lack
of uniformity.”
Humanistic expectations vs
Knowledge demands
(Anderson, 2011)

7. 0
50
100
150
200
1957
1971
1978
1980
1984
1987
1989
1991
1993
1995
1997
1999
2001
2003
2005
2007
2009
2011
#ofarticles
Year of Publication
PubMed Search on
'Professionalism, Medical Education'
Publication of CanMEDs 2000

8. 2. Creating
clear and
explicit
milestones

10. % prior to
blueprint
publication
% after
blueprint
publication
p-value
Exam performance 72.5 76.9 ns
Agreement that exam
tested material taught
63.5 81.3 < 0.01
Agreement that
evaluation methods
reflected subject
matter
60.7 81.5 < 0.01
Agreement that exam
was fair
58.0 76.9 < 0.05
(McLaughlin et al., 2005)

11. GeneralTheme
• Broad, competency-based assessment
frameworks, can promote performance
improvement rather than simply measuring
performance
• They create professional culture through
conversation, understanding, and steering

13. 1. Sending the
wrong
message

15. Implicit Messages:
• Expertise is something that can be
achieved
• The goal is to “become independent”
Risk:
• Assessments as hurdles
• Hesitation to disclose difficulties
• Reduced compulsion to offer support
(see MEAAC Report)

16. GoalTheory
Performance Orientation Mastery Orientation
•Desire to perform well •Desire to become proficient
•Satisfaction derived from
grades
•Deeper engagement
•Greater anxiety •Greater perseverance
•Task avoidance •Stronger motivation
(see Teunissen and Bok, 2013)

17. 2. Ignoring the
continuum

18. Violation of 3 fundamental laws
Variability
Big
Losers
Big
Winners
Everyone Else
Simple Probability Bell Curve
Society’s Problem

19. Violation of 3 fundamental laws
Variability
Context specificity
(Norcini, circa 2006)
“The one truth
in medical
education.”

20. Violation of 3 fundamental laws
Variability
Context specificity
Decay
(Custers and ten Cate, 2011)

21. GeneralTheme
• Be wary of the unintended consequences of
adopting a competence-based assessment
framework
(see MEAAC Report)

23. To assess competency effectively requires
assessment strategies that are …
– broadly focused,
– longitudinal,
– integrated,
– continuous, and
– authentic

25. Three (Overlapping)Themes
• Overcoming unintended consequences
• Turning quality assurance into quality
improvement
• Ensuring authenticity

26. 1. Overcoming Unintended
Consequences
Gist:
• Reduce emphasis on exams as point in time
hurdles that prove one’s competence
• Promote notion that trainees are equally
accountable for their demonstration of learning

27. 1. Overcoming Unintended
Consequences
Strategy:
• Build quality improvement activities into
assessment practices
• Use data from licensing process to facilitate
formulation of learning plans
– And further develop system to enforce follow through

28. 1. Overcoming Unintended
Consequences
Examples:
• OSCE/CDM components that require candidates to follow-
up on an error made; ask for help; use clinical decision
supports
• Feedback intra-candidate relative strengths and
weaknesses and require generation of learning plan
• Tailor subsequent assessments to identified weaknesses

29. 2. Turning Quality Assurance into
Quality Improvement
Gist:
• Reduce the tension between high stakes
licensing assessment and genuine
investment in improvement

30. 2. Turning Quality Assurance into
Quality Improvement
Strategy:
• Further integrate assessment practices across
the continuum of learning with deliberate
attention paid to (and reward of) quality
improvement

31. 2. Turning Quality Assurance into
Quality Improvement
Examples:
• Create a formative test tailoring platform for use by
schools/individuals
• Support a national “Diagnostic OSCE” late in UG that can feed
data to subsequent stages of training/practice/assessment
• Testing moments that require demonstration of response to
data (e.g., OSCE station in which candidates bring personal
data)

32. 3. Ensuring Authenticity
Gist:
• Assessment that models the realities of
actual practice increases
credibility, engagement, and ensures that
efforts towards gamesmanship are
pedagogically valuable

33. 3. Ensuring Authenticity
Strategy:
• Portfolio-supported workplace-based
assessment
• Increasing use of real world supports and real
world uncertainties in current practices

34. 3. Ensuring Authenticity
Examples:
• Sequential OSCE stations; SPs who are trained to offer
contradictory information mid-station
• Post-encounter probes that require reflection on why
approach was appropriate and alternative actions
were ruled out
• Internet enabled OSCEs

35. Completing the puzzle
• Broaden the base of assessment
• Build coherent and integrated system
• Emphasize the primacy of learning
• Harness the power of feedback
• Share accountability with the individual

36. Completing the puzzle

37. kevin.eva@ubc.ca
Thanks