1. Ottawa Conference/CCME
C. Touchie, A. Gotzmann, C. Streefkerk,
A. De Champlain,
Medical Council of Canada
Putting the Pieces Together
to Create a Defensible
Blueprint:
A Subject Matter Expert Based
Process
April 2014
Ottawa, Canada
1

2. Credentialing Examinations
• Validity:
– Associated with the interpretation assigned to
test scores NOT with the score or with the test
itself
– Kane’s validity argument:
• Scoring
• Generalization
• Extrapolation  the content of the test is
matched appropriately to the
demands of practice
• Decision
2

3. Practice Analysis
• Practice Analysis Methods:
– Critical Incident Technique
– Task Inventory
– Professional Practice Model
• Professions: dynamic and social
2

4. Purpose of Blueprinting
• … is to assure the public that physicians
licensed to practice medicine have the
required knowledge, skills and attitudes
for safe and effective patient care.
• Only those who meet this standard are
qualified to enter professional practice
4

5. Decision points
• What we are assessing:
– Core foundational competencies expected of all
physicians regardless of discipline for the
purpose of fulfilling one of the requirements of
licensure
• Decision point 1
– Assessments leading to a decision for entry into
residency
• Decision point 2
– Assessments leading to a decision for entry into
independent practice
5

6. Practice Analysis
• Factors to Consider
1. Information required
2. Best sources of information
3. Strategy for obtaining that information
4. Strategy to put it all together
6Modified from M. Raymond, AERA 2014

7. 1. Information Required
7
Focus of practice
descriptors
Level of
specificity
Attributes to be
measured
Current Issues Future of medical
education
Broad and
complex
Assessment needs
Supervising
PGY-1
Transition between
MD and residency
Entrustable
professional
activities
Supervisory needs
Incidence and
Prevalence
Common clinical
problems
Specific
presentations
(age, setting)
Frequency
National Survey MCC objectives Specific KSAs Importance of KSAs at
2 decision points

8. 2. Sources of Information
8
Sources of Information
Current Issues Review of literature, medical educators
Supervising
PGY-1
New PGY-1s, supervisors
Incidence and
Prevalence
Billing data (CIHI, provincial sources), published
literature
National Survey Physicians: recent LMCC recipients, clinical
teachers
Informed public

9. 3. Data Collection Strategy
9
Data Collection Strategy
Current Issues Review of literature, iterative review by experts
Supervising
PGY-1
Consensus technique/Survey
Incidence and
Prevalence
Review of existing data
National Survey Physicians: Survey
Informed public: Survey and focus groups

10. 4. Strategy to pull it all together
Blueprint and Test Specifications
Defined 2 candidates Common Blueprint
Test Specifications for
each decision point
Subject Matter Experts
12 Experts 3 day meeting
4 Reports
Current Issues
PGY-1
supervision
Incidence and
Prevalence
National
Survey

11. Who were the SMEs?
Blueprint
MRA Rep of
Council
Central
Examination
Committee
Objectives
Committee
Test
Committees
RCPSCCFPC
UGME
Deans
PGME
Deans
University
Rep of
Council

12. Results:
Proposed Common Blueprint
Dimensions of Care
Health
Promotion and
Illness
Prevention
Acute Chronic Psychosocial
Aspects
PhysicianActivities
Assessment/
Diagnosis
Management
Communication
Professional
Behaviors
12

13. Dimensions of Care
Health
Promotion
and Illness
Prevention
Acute Chronic Psychosocial
Aspects
Row
Percent
PhysicianActivities
Assessment/
Diagnosis 30±5
Management 20±5
Communication 30±5
Professional
Behaviors 20±5
Column Percent 20±5 30±5 30±5 20±5 100
Assessments leading up to
Decision 1: Entry into Residency
13

14. Dimensions of Care
Health
Promotion
and Illness
Prevention
Acute Chronic Psychosocial
Aspects
Row
Percent
PhysicianActivities
Assessment/
Diagnosis 25±5
Management 35±5
Communication 20±5
Professional
Behaviors 20±5
Column Percent 20±5 25±5 35±5 20±5 100
Assessments leading up to
Decision 2: Entry into Independent Practice
14

15. Topics / Knowing
Processes / Doing
Processes/Doing
Textbook Topics
Academic Disciplines
Organ System
Clinical Problem / Reason for Visit
Medical Specialty
Cognitive Processes or Skills
Competency (ACGME, CanMEDS)
Physician Task or Skill

16. Conclusions
Defensible Blueprint:
• Define assessment purpose
• Choose information sources to inform
appropriately
• Include judgments from stakeholders
Lessons learned:
• Careful planning
– Team
– Information gathered
– Process
16

17. Thank You!
Questions – Comments
ctouchie@mcc.ca
17