The document summarizes the blueprinting project of the Medical Council of Canada (MCC) to define and validate the critical competencies assessed in MCC examinations. It provides an overview of the project timeline and phases, which include a Delphi process using subject matter experts to reach consensus on objectives, a national survey, and revising examination blueprints. The next steps outlined are completing phase 1 and transitioning to operationalizing the new blueprints.

1. Blueprinting Project:
Update and Next Steps
Elaine Rodeck, Ph.D.
Examination Strategist, Blueprint Lead
Medical Council of Canada
Annual General Meeting, September 30, 2012
1

2. Overview
1. Blueprinting Project - Purpose, Timeline
2. Delphi Process
3. National Survey
4. Next Steps
5. Q&A
2

3. Historical Perspective
• 2009
– Medical Council of Canada
(MCC) Objectives Committee
revised objectives using
CanMEDS as the framework
• October 2011
– Report of the Assessment Review Task Force (ARTF)
approved by Council
3
3

4. ARTF Recommendation 2
• The content of the MCC assessment processes shall
be expanded by:
– Defining the knowledge and behaviours in the CanMEDS
roles that demonstrates competence of a physician about to
enter independent practice
– Reviewing the adequacy of the content and skills coverage
on the blueprints for all MCC examinations
– Revising the examination blueprints and reporting systems
– Determining whether any core competencies cannot be
tested by the MCC examinations
4
4

5. MCC Validity Research Agenda
5
5

6. Why do a blueprint exercise?
The exercise is designed:
…to define and validate the critical
competencies that the MCC examinations
should sample
in order:
…to assure Medical Regulatory Authorities
(MRAs) and stakeholders that a physician
has the requisite knowledge, skills and
professional behaviours to enter
supervised/unsupervised clinical practice
6
6

7. Who is participating in the process?
• Blueprinting Core Project Team
• Governance Board
• Steering Group
• Process Participants
– Subject matter experts (SMEs) for the Delphi questionnaire and
follow-up review by SMEs
– National Survey participants – a cross section of health providers,
teachers, learners, and members of the public
– SMEs blueprint panelists
– MCC’s Central Examination Committee
7
7

8. How is this being done?
• Phase 1
– January 2012 to fall 2013
– Information gathering
• Delphi Group/Follow-up with SMEs
• Incidence and Prevalence paper
• National Survey
• Pilot Project on resident supervision
• Medical Education Assessment Advisory Committee paper
– Establish recommendations for the new
MCCQE test specifications
8
8

9. Blueprinting Project…
– Phase I
• Approved blueprints (test specifications) for the revisions to the MCCQE Parts I and II
• Identification of requirements not measurable in formats such as MCCQE Part I or Part II
– Phase II - operational transition
• Gap analysis of the existing item banks
• Content and form development, infrastructure, workflow design
• Pilot testing
• Communication Plan for all stakeholders
• Develop process to consider new types of assessment
– Phase III - the launch
• New examinations and content
• Monitoring for ongoing content renewal.
Consultation with stakeholders throughout the process is key to ensuring outcomes
are the result of a collaborative process!
9
9

10. What have we done?
• Incidence and Prevalence paper
• Delphi process of CanMEDS roles and
MCC objectives
• Follow-up review by experts who have
content knowledge
10
10

11. Blueprinting Project:
Delphi Process
11

12. The Delphi Process…
• Tool for consensus measurement
• Determines the extent of agreement amongst a group of experts*
• Anonymity
– to prevent dominance of the process by one or a few experts
• Iteration
– several rounds are conducted to allow individuals to change opinions
• Controlled feedback
– provide feedback on individual and group responses
• Statistical group response
– synthesis and analysis of the group response
*Jones, J & Hunter D (1995) Consensus methods for medical and health services research.
BMJ 311: 376-80.
12
12

13. The Delphi Process…
The Panelists
• 22 panelists…pan-Canadian medical school
representation including Quebec, medical disciplines,
registrar representation, gender, etc.
The Questionnaire
• Medical expert objectives – Over 200 statements
• Non-medical expert objectives – including…professional,
communicator, scholar, manager, collaborator, advocate
• Results Review – subsequent review by SMEs
13
13

14. The Delphi Process…
Rate objectives for entry
into unsupervised practice.
1.Unnecessary
2.Not Important
3.Slightly Important
4.Important
5.Very Important
6.Essential
7.Cannot Rate
14
14

15. Example - Delphi Process Question
What is your opinion? 1. Unnecessary
2. Not Important
Glucose Abnormal, Serum/Diabetes 3. Slightly Important
Mellitus/Polydipsia 4. Important
Diabetes mellitus 5.
6.
Very Important
Essential
Rationale: Diabetes mellitus is an increasingly 7. Cannot Rate
common multi-system disease associated with
a relative or absolute impairment of insulin 25% 25% 25% 25%
secretion together with varying degrees of
peripheral resistance to the action of insulin.
Key Objectives: Given a patient with diabetes
mellitus, the candidate will diagnose the cause,
severity and complications, and will initiate an
appropriate management plan. Particularly
important are early detection of the disease,
and recognition of medical emergencies such
as acute hypoglycemia, diabetic ketoacidosis,
and hyperosmolar nonketotic coma. 0% 0% 0%
15
15 1 2 3 4 5 6 7

16. Example - Delphi Process Question
What is your opinion? 1. Unnecessary
2. Not Important
Hair and Nail Complaints 3. Slightly Important
Nail complaints 4. Important
5. Very Important
Rationale: Nail disorders are common 6. Essential
conditions. Although in themselves nail 7. Cannot Rate
changes may be innocuous, they may indicate
underlying disease. 25% 25% 25% 25%
Key Objectives: Given a patient with nail
abnormalities, the candidate will diagnose the
cause (local or systemic), severity, and
complications, and will initiate an appropriate
management plan.
0% 0% 0%
16
16 1 2 3 4 5 6 7

17. Example - Delphi Process Question
What is your opinion? 1. Unnecessary
2. Not Important
Abdominal Pain 3. Slightly Important
Abdominal pain, Acute 4. Important
5. Very Important
Rationale: Acute abdominal pain is a common 6. Essential
complaint in adults, leading to frequent 7. Cannot Rate
physician visits both in the Emergency
Department and office setting. Acute 25% 25% 25% 25%
abdominal pain may result from serious intra-
abdominal, intrathoracic, or retroperitoneal
processes.
Key Objectives: Given a patient with acute
abdominal pain, the candidate will diagnose
the cause, severity, and complications, and will
initiate an appropriate management plan. In
particular, the candidate will identify those
patients requiring emergency medical or
surgical treatment. 0% 0% 0%
17
17 1 2 3 4 5 6 7

18. Example - Delphi Process Question
What is your opinion? 1.Unnecessary
2.Not Important
Hair and Nail Complaints 3.Slightly Important
Alopecia 4.Important
5.Very Important
Rationale: Alopecia may be physiological or 6.Essential
due to local scalp disease or underlying 7.Cannot Rate
systemic disease and can result in
psychological distress. 25% 25% 25% 25%
Key Objectives: Given a patient with alopecia,
the candidate will diagnose the cause, severity,
and complications, and will initiate an
appropriate management plan.
0% 0% 0%
18
18 1 2 3 4 5 6 7

19. Example - Delphi Process Question
What is your opinion? 1. Unnecessary
2. Not Important
Population Health 3. Slightly Important
Assessing and measuring health 4. Important
5. Very Important
status at the Population Level 6. Essential
Rationale: Knowing the health status of the 7. Cannot Rate
population allows for better planning and
25% 25% 25% 25%
evaluation of health programs and tailoring
interventions to meet patient/community
needs. Physicians are also active participants
in disease surveillance programs, encouraging
them to address health needs in the population
and not merely health demands.
Key Objectives: Describe the health status of
a defined population. Measure and record the
factors that affect the health status of a
population with respect to the principles of
causation. 0% 0% 0%
19
19 1 2 3 4 5 6 7

20. Blueprinting Project:
National Survey
20

21. National Survey
• Sampling plan in process – fall 2012
– Physicians, Residents, Students, Program
Directors, Recently certified CFPC/RCPSC,
Nurses, Pharmacists, MRAs, public members
• Survey timing – early February 2013
• SME panel review – May 2013
• CEC review – June 2013
21
21

22. Blueprinting Project:
Next Steps
22

23. Next Steps
• 2012 Fall
– Develop national survey questions
– CEC to approve national survey
• 2013
– February – National survey
– March /April – Analyze data, prepare for SME panel meetings
– May – SME panel meetings (two)
– June – CEC to review draft blueprints
– July – Stakeholder invitation to comment
– August/September – Revise blueprints to include stakeholder
comments
– AGM – Present revised blueprints to AGM for consultation
– Late fall – Blueprints to be revised and approved by CEC
– Late 2013/early 2014 – Executive Board to approve blueprints
23
23

24. Blueprinting Project:
Q&A
24

25. Questions, Comments & Discussion
Elaine Rodeck
ERodeck@mcc.ca
Thank you!
25
25