Blueprinting 2012

952 views

Published on

  • Be the first to comment

  • Be the first to like this

Blueprinting 2012

  1. 1. Blueprinting Project: Update and Next Steps Elaine Rodeck, Ph.D. Examination Strategist, Blueprint Lead Medical Council of Canada Annual General Meeting, September 30, 20121
  2. 2. Overview 1. Blueprinting Project - Purpose, Timeline 2. Delphi Process 3. National Survey 4. Next Steps 5. Q&A2
  3. 3. Historical Perspective • 2009 – Medical Council of Canada (MCC) Objectives Committee revised objectives using CanMEDS as the framework • October 2011 – Report of the Assessment Review Task Force (ARTF) approved by Council 33
  4. 4. ARTF Recommendation 2 • The content of the MCC assessment processes shall be expanded by: – Defining the knowledge and behaviours in the CanMEDS roles that demonstrates competence of a physician about to enter independent practice – Reviewing the adequacy of the content and skills coverage on the blueprints for all MCC examinations – Revising the examination blueprints and reporting systems – Determining whether any core competencies cannot be tested by the MCC examinations 44
  5. 5. MCC Validity Research Agenda 55
  6. 6. Why do a blueprint exercise? The exercise is designed: …to define and validate the critical competencies that the MCC examinations should sample in order: …to assure Medical Regulatory Authorities (MRAs) and stakeholders that a physician has the requisite knowledge, skills and professional behaviours to enter supervised/unsupervised clinical practice 66
  7. 7. Who is participating in the process? • Blueprinting Core Project Team • Governance Board • Steering Group • Process Participants – Subject matter experts (SMEs) for the Delphi questionnaire and follow-up review by SMEs – National Survey participants – a cross section of health providers, teachers, learners, and members of the public – SMEs blueprint panelists – MCC’s Central Examination Committee 77
  8. 8. How is this being done? • Phase 1 – January 2012 to fall 2013 – Information gathering • Delphi Group/Follow-up with SMEs • Incidence and Prevalence paper • National Survey • Pilot Project on resident supervision • Medical Education Assessment Advisory Committee paper – Establish recommendations for the new MCCQE test specifications 88
  9. 9. Blueprinting Project… – Phase I • Approved blueprints (test specifications) for the revisions to the MCCQE Parts I and II • Identification of requirements not measurable in formats such as MCCQE Part I or Part II – Phase II - operational transition • Gap analysis of the existing item banks • Content and form development, infrastructure, workflow design • Pilot testing • Communication Plan for all stakeholders • Develop process to consider new types of assessment – Phase III - the launch • New examinations and content • Monitoring for ongoing content renewal. Consultation with stakeholders throughout the process is key to ensuring outcomes are the result of a collaborative process! 99
  10. 10. What have we done? • Incidence and Prevalence paper • Delphi process of CanMEDS roles and MCC objectives • Follow-up review by experts who have content knowledge 1010
  11. 11. Blueprinting Project: Delphi Process11
  12. 12. The Delphi Process… • Tool for consensus measurement • Determines the extent of agreement amongst a group of experts* • Anonymity – to prevent dominance of the process by one or a few experts • Iteration – several rounds are conducted to allow individuals to change opinions • Controlled feedback – provide feedback on individual and group responses • Statistical group response – synthesis and analysis of the group response *Jones, J & Hunter D (1995) Consensus methods for medical and health services research. BMJ 311: 376-80. 1212
  13. 13. The Delphi Process… The Panelists • 22 panelists…pan-Canadian medical school representation including Quebec, medical disciplines, registrar representation, gender, etc. The Questionnaire • Medical expert objectives – Over 200 statements • Non-medical expert objectives – including…professional, communicator, scholar, manager, collaborator, advocate • Results Review – subsequent review by SMEs 1313
  14. 14. The Delphi Process… Rate objectives for entry into unsupervised practice. 1.Unnecessary 2.Not Important 3.Slightly Important 4.Important 5.Very Important 6.Essential 7.Cannot Rate 1414
  15. 15. Example - Delphi Process Question What is your opinion? 1. Unnecessary 2. Not Important Glucose Abnormal, Serum/Diabetes 3. Slightly Important Mellitus/Polydipsia 4. Important Diabetes mellitus 5. 6. Very Important Essential Rationale: Diabetes mellitus is an increasingly 7. Cannot Rate common multi-system disease associated with a relative or absolute impairment of insulin 25% 25% 25% 25% secretion together with varying degrees of peripheral resistance to the action of insulin. Key Objectives: Given a patient with diabetes mellitus, the candidate will diagnose the cause, severity and complications, and will initiate an appropriate management plan. Particularly important are early detection of the disease, and recognition of medical emergencies such as acute hypoglycemia, diabetic ketoacidosis, and hyperosmolar nonketotic coma. 0% 0% 0% 1515 1 2 3 4 5 6 7
  16. 16. Example - Delphi Process Question What is your opinion? 1. Unnecessary 2. Not Important Hair and Nail Complaints 3. Slightly Important Nail complaints 4. Important 5. Very Important Rationale: Nail disorders are common 6. Essential conditions. Although in themselves nail 7. Cannot Rate changes may be innocuous, they may indicate underlying disease. 25% 25% 25% 25% Key Objectives: Given a patient with nail abnormalities, the candidate will diagnose the cause (local or systemic), severity, and complications, and will initiate an appropriate management plan. 0% 0% 0% 1616 1 2 3 4 5 6 7
  17. 17. Example - Delphi Process Question What is your opinion? 1. Unnecessary 2. Not Important Abdominal Pain 3. Slightly Important Abdominal pain, Acute 4. Important 5. Very Important Rationale: Acute abdominal pain is a common 6. Essential complaint in adults, leading to frequent 7. Cannot Rate physician visits both in the Emergency Department and office setting. Acute 25% 25% 25% 25% abdominal pain may result from serious intra- abdominal, intrathoracic, or retroperitoneal processes. Key Objectives: Given a patient with acute abdominal pain, the candidate will diagnose the cause, severity, and complications, and will initiate an appropriate management plan. In particular, the candidate will identify those patients requiring emergency medical or surgical treatment. 0% 0% 0% 1717 1 2 3 4 5 6 7
  18. 18. Example - Delphi Process Question What is your opinion? 1.Unnecessary 2.Not Important Hair and Nail Complaints 3.Slightly Important Alopecia 4.Important 5.Very Important Rationale: Alopecia may be physiological or 6.Essential due to local scalp disease or underlying 7.Cannot Rate systemic disease and can result in psychological distress. 25% 25% 25% 25% Key Objectives: Given a patient with alopecia, the candidate will diagnose the cause, severity, and complications, and will initiate an appropriate management plan. 0% 0% 0% 1818 1 2 3 4 5 6 7
  19. 19. Example - Delphi Process Question What is your opinion? 1. Unnecessary 2. Not Important Population Health 3. Slightly Important Assessing and measuring health 4. Important 5. Very Important status at the Population Level 6. Essential Rationale: Knowing the health status of the 7. Cannot Rate population allows for better planning and 25% 25% 25% 25% evaluation of health programs and tailoring interventions to meet patient/community needs. Physicians are also active participants in disease surveillance programs, encouraging them to address health needs in the population and not merely health demands. Key Objectives: Describe the health status of a defined population. Measure and record the factors that affect the health status of a population with respect to the principles of causation. 0% 0% 0% 1919 1 2 3 4 5 6 7
  20. 20. Blueprinting Project: National Survey20
  21. 21. National Survey • Sampling plan in process – fall 2012 – Physicians, Residents, Students, Program Directors, Recently certified CFPC/RCPSC, Nurses, Pharmacists, MRAs, public members • Survey timing – early February 2013 • SME panel review – May 2013 • CEC review – June 2013 2121
  22. 22. Blueprinting Project: Next Steps22
  23. 23. Next Steps • 2012 Fall – Develop national survey questions – CEC to approve national survey • 2013 – February – National survey – March /April – Analyze data, prepare for SME panel meetings – May – SME panel meetings (two) – June – CEC to review draft blueprints – July – Stakeholder invitation to comment – August/September – Revise blueprints to include stakeholder comments – AGM – Present revised blueprints to AGM for consultation – Late fall – Blueprints to be revised and approved by CEC – Late 2013/early 2014 – Executive Board to approve blueprints 2323
  24. 24. Blueprinting Project: Q&A24
  25. 25. Questions, Comments & Discussion Elaine Rodeck ERodeck@mcc.ca Thank you! 2525

×