Blueprinting update - 2014 Ottawa Conference

677 views

Published on

Published in: Education, Health & Medicine
  • Be the first to comment

  • Be the first to like this

Blueprinting update - 2014 Ottawa Conference

  1. 1. Ottawa Conference/CCME C. Touchie, A. Gotzmann, C. Streefkerk, A. De Champlain, Medical Council of Canada Putting the Pieces Together to Create a Defensible Blueprint: A Subject Matter Expert Based Process April 2014 Ottawa, Canada 1
  2. 2. Credentialing Examinations • Validity: – Associated with the interpretation assigned to test scores NOT with the score or with the test itself – Kane’s validity argument: • Scoring • Generalization • Extrapolation  the content of the test is matched appropriately to the demands of practice • Decision 2
  3. 3. Practice Analysis • Practice Analysis Methods: – Critical Incident Technique – Task Inventory – Professional Practice Model • Professions: dynamic and social 2
  4. 4. Purpose of Blueprinting • … is to assure the public that physicians licensed to practice medicine have the required knowledge, skills and attitudes for safe and effective patient care. • Only those who meet this standard are qualified to enter professional practice 4
  5. 5. Decision points • What we are assessing: – Core foundational competencies expected of all physicians regardless of discipline for the purpose of fulfilling one of the requirements of licensure • Decision point 1 – Assessments leading to a decision for entry into residency • Decision point 2 – Assessments leading to a decision for entry into independent practice 5
  6. 6. Practice Analysis • Factors to Consider 1. Information required 2. Best sources of information 3. Strategy for obtaining that information 4. Strategy to put it all together 6Modified from M. Raymond, AERA 2014
  7. 7. 1. Information Required 7 Focus of practice descriptors Level of specificity Attributes to be measured Current Issues Future of medical education Broad and complex Assessment needs Supervising PGY-1 Transition between MD and residency Entrustable professional activities Supervisory needs Incidence and Prevalence Common clinical problems Specific presentations (age, setting) Frequency National Survey MCC objectives Specific KSAs Importance of KSAs at 2 decision points
  8. 8. 2. Sources of Information 8 Sources of Information Current Issues Review of literature, medical educators Supervising PGY-1 New PGY-1s, supervisors Incidence and Prevalence Billing data (CIHI, provincial sources), published literature National Survey Physicians: recent LMCC recipients, clinical teachers Informed public
  9. 9. 3. Data Collection Strategy 9 Data Collection Strategy Current Issues Review of literature, iterative review by experts Supervising PGY-1 Consensus technique/Survey Incidence and Prevalence Review of existing data National Survey Physicians: Survey Informed public: Survey and focus groups
  10. 10. 4. Strategy to pull it all together Blueprint and Test Specifications Defined 2 candidates Common Blueprint Test Specifications for each decision point Subject Matter Experts 12 Experts 3 day meeting 4 Reports Current Issues PGY-1 supervision Incidence and Prevalence National Survey
  11. 11. Who were the SMEs? Blueprint MRA Rep of Council Central Examination Committee Objectives Committee Test Committees RCPSCCFPC UGME Deans PGME Deans University Rep of Council
  12. 12. Results: Proposed Common Blueprint Dimensions of Care Health Promotion and Illness Prevention Acute Chronic Psychosocial Aspects PhysicianActivities Assessment/ Diagnosis Management Communication Professional Behaviors 12
  13. 13. Dimensions of Care Health Promotion and Illness Prevention Acute Chronic Psychosocial Aspects Row Percent PhysicianActivities Assessment/ Diagnosis 30±5 Management 20±5 Communication 30±5 Professional Behaviors 20±5 Column Percent 20±5 30±5 30±5 20±5 100 Assessments leading up to Decision 1: Entry into Residency 13
  14. 14. Dimensions of Care Health Promotion and Illness Prevention Acute Chronic Psychosocial Aspects Row Percent PhysicianActivities Assessment/ Diagnosis 25±5 Management 35±5 Communication 20±5 Professional Behaviors 20±5 Column Percent 20±5 25±5 35±5 20±5 100 Assessments leading up to Decision 2: Entry into Independent Practice 14
  15. 15. Topics / Knowing Processes / Doing Processes/Doing Textbook Topics Academic Disciplines Organ System Clinical Problem / Reason for Visit Medical Specialty Cognitive Processes or Skills Competency (ACGME, CanMEDS) Physician Task or Skill
  16. 16. Conclusions Defensible Blueprint: • Define assessment purpose • Choose information sources to inform appropriately • Include judgments from stakeholders Lessons learned: • Careful planning – Team – Information gathered – Process 16
  17. 17. Thank You! Questions – Comments ctouchie@mcc.ca 17

×