Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

2018 MCC business session at CCME

233 views

Published on

Embracing exam changes through Assessment Evolution. In this session:
Learn about upcoming changes to the MCC's examination program based on the new Blueprint
Find out more about the new resources available to help candidates challenge these exams
Discover how to interpret new scores and get a sneak-peak of the new results reports

Published in: Health & Medicine
  • Be the first to comment

  • Be the first to like this

2018 MCC business session at CCME

  1. 1. |© 2018 MCC/CMC© 2018 MCC/CMC C C M E C C E M Medical Council of Canada Assessment Evolution and Score Reporting EMBRACING EXAM CHANGES THROUGH ASSESSMENT EVOLUTION Dr. Ian Bowmer, Yves Lafortune and Dr. Liane Patsula
  2. 2. |© 2018 MCC/CMC© 2018 MCC/CMC C C M E C C E M Medical Council of Canada Assessment Evolution and Score Reporting EMBRACING EXAM CHANGES THROUGH ASSESSMENT EVOLUTION In preparation for a quiz at the end of the presentation, please open: • The Kahoot! mobile app, or • Go to www.kahoot.it on your browser
  3. 3. |© 2018 MCC/CMC Topics covered 3 • Introduction • Review of upcoming changes to the MCC's examination program based on the new Blueprint • Candidate support tools and new Practice Test Portal • New service delivery model • Results interpretation • Quiz
  4. 4. CCME CCEM UPCOMING EXAM CHANGES
  5. 5. |© 2018 MCC/CMC Overview of changes - Blueprint & test specifications 5 MCCQE Part I test specifications MCCQE Part II test specifications
  6. 6. |© 2018 MCC/CMC Overview of changes to MCCQE Part I - 2018 6 • Exam based on the new Blueprint ◦ Launch: spring 2018 • Offered twice in 2018, spring and fall • Standard setting exercise • Delay in spring 2018 results release
  7. 7. |© 2018 MCC/CMC Overview of changes – MCCQE Part I International 7 • More frequent and international delivery of the MCCQE Part I starting in 2019 • Up to five sessions per year • Proposed 2019 dates under review: • Better and faster access to testing • As of 2019, international medical students and graduates will no longer need to pass the MCCEE before taking the MCCQE Part I ◦ April - May (3 wks) ◦ July (3 wks) ◦ Sept. - Oct. (3 wks) ◦ Nov. (3 wks)
  8. 8. |© 2018 MCC/CMC Overview of changes to MCCQE Part II Embracing exam changes through Assessment Evolution 8 • Continuing to work on enhancing access and capacity to the MCCQE Part II • Change in eligibility criteria to enable current PGY-1 to take the exam • Starting in fall 2018 ◦ Exam based on the new Blueprint ◦ More authentic, complex scenarios ◦ Will remain a two-day exam – Day 1 – 14-minute stations (8 in all) – Day 2 – 6-minute paired stations (4 in all)
  9. 9. |© 2018 MCC/CMC Overview of changes to MCCQE Part II Embracing exam changes through Assessment Evolution 9 Exam to cover areas such as: • Obtaining a focused history • Conducting a physical exam • Organizing a discharge • Discussing the goals of care • Delivering difficult news • Dealing with forgetful or angry patients • Providing advice • Interacting with colleagues • Addressing conflicts
  10. 10. |© 2018 MCC/CMC Overview of changes – NAC examination 10 Exam launch in 2019 with these changes: • A new scoring methodology • Improved scoring instruments • Language fluency integrated into the Communications Skills competency • Refined exam purpose wording • A new exam standard • Communication Skills and Organization competencies will be removed from some stations
  11. 11. |© 2018 MCC/CMC Overview of application process for international students Embracing exam changes through Assessment Evolution 11
  12. 12. |© 2018 MCC/CMC Overview of application process for Canadian students Embracing exam changes through Assessment Evolution 12 Canadian medical students and graduates will be able to take and retake the MCCQE Part I more frequently Benefits: • Opportunity to retake exam before graduation • If candidate has taken a leave of absence or is off cycle for another reason, can pass the exam sooner
  13. 13. |© 2018 MCC/CMC Preparatory material for candidates Embracing exam changes through Assessment Evolution 13 • Communications and cultural competency modules – free • Clinical decision making video – free • Online OSCE orientation (2018) – free • Multiple-Choice Question Test (now available) • Clinical Decision Making Test (Fall 2018) • Preparatory Examination (Winter 2019)
  14. 14. |© 2018 MCC/CMC Communication and Cultural Competence orientation 14 Publicly available, these modules cover topics such as: • Communication skills • Consent and confidentiality • Cross-cultural communication • Indigenous health • Mental health • Complexities of care of the elderly
  15. 15. CCME CCEM CANDIDATE TOOLS & SERVICE DELIVERY MODEL
  16. 16. |© 2018 MCC/CMC Launch of Practice Tests – starting in 2018 Product 1 Short practice tests to help assess readiness in either MCQ or CDM cases • Two MCQ test forms – 100 MCQs per form (Launched: January 2018) • Three CDM test forms – 14 CDM cases (Launch: fall 2018) • Results and information to candidates in line with the new Blueprint • $100 per test form Product 2 Opportunity to get as close as possible to the real examination experience • Preparatory examination (PE) represents a full MCCQE Part I test form • Test form will have: ◦ 210 MCQs and 38 CDM cases (Launch: Winter 2019) • Feedback at item and new blueprint level, includes references and rationale • Timing options: “self-paced” or “examination mode” • $500 per test form
  17. 17. |© 2018 MCC/CMC Practice Test content 17 Field tested (piloted) all Practice Test content with Canadian Medical Students in their final year (reference group) • Delivered the Pilot Project, over the past 2 years, at no cost • Launched a new and enhanced practice test delivery platform • Analysis of data currently underway to ensure practice test content aligns with new MCCQE Part I examination The MCC appreciates the level of effort all Universities provided to ensure the effective delivery of the pilot project.
  18. 18. |© 2018 MCC/CMC Practice Test Portal – log in page 18
  19. 19. |© 2018 MCC/CMC Organization Portal 19 • Organization representatives can purchase access codes, create test administrations and view results. • Organizations can purchase as many access codes as they want.
  20. 20. |© 2018 MCC/CMC Organization Portal – test results 20 • Organizations have the flexibility to view the test results for any particular test taker, or specific test administration.
  21. 21. |© 2018 MCC/CMC Test Taker Portal 21 • Test takers have the option to purchase a new test, or to redeem an access code that was assigned to them.
  22. 22. |© 2018 MCC/CMC Test Taker Portal 22 • Candidates may also select the language in which they would like to take the test.
  23. 23. |© 2018 MCC/CMC Test Delivery Engine (Multiple-Choice Questions) 23 The MCQ Practice Tests are currently available for purchase.
  24. 24. |© 2018 MCC/CMC Test Delivery Engine (Clinical Decision Making Questions) 24 The CDM Practice Tests will be made available fall 2018.
  25. 25. |© 2018 MCC/CMC Test Taker Portal – test results 25 Test takers are presented their results as per the new Blueprint.
  26. 26. |© 2018 MCC/CMC Changes in the MCCQE Part I delivery model – 2019 26 2018 – MCC delivery continues • New exam based on the new blueprint • Status quo for site delivery • Standard setting: June 2018 • Delayed release of results: July 2018 2019 – vendor delivery What’s different? • Registration and scheduling • Number of administrations • Start/end time of examinations • Enhanced security features: • Biometrics • Video surveillance & monitoring • Escorting & breaks supervised
  27. 27. |© 2018 MCC/CMC MCCQE Part I delivery model – 2019 27 In preparation for vendor delivery in 2019: • Partnering with some Universities in order to meet capacity (1 or 2 x year) • Planning and site visits underway at specific locations across Canada to ensure seamless delivery in 2019 • Purpose of site visits: ◦ Introduce Prometric Staff to University Site Administrators ◦ Evaluate space and technical requirements ◦ Discuss policies and procedures with a goal to replicate the candidate experience across all delivery sites ◦ Plan for additional staff to meet the ratio of two staff per 25 candidates ◦ Discuss training requirement of staff as part of the partnership with Prometric ◦ Strategize MCC/Prometric presence at initial delivery ◦ Testing of systems and integration planning
  28. 28. |© 2018 MCC/CMC New Blueprint webinars: MCCQE Part I and MCCQE Part II 28 MCCQE Part I webinar: • Delivered September 7, 2017 at The University of Ottawa • Offered in French and English • mccevolution.ca/webinar-new- mcc-qualifying-examination-part- whats MCCQE Part II webinar: • Delivered April 15, 2018 in Ottawa • Offered in French and English • mccevolution.ca/webinar-the-new- mcc-qualifying-examination-part-ii- whats-in-it-for-me
  29. 29. CCME CCEM RESULTS INTERPRETATION 29
  30. 30. |© 2018 MCC/CMC || Overview 30 1. New exam score ranges 2. New pass score – standard setting 3. Two exam results documents a) Statement of Results (SoR) b) Supplemental Information Report (SIR) 4. Results interpretation
  31. 31. |© 2018 MCC/CMC || New exam score ranges 31 Guided by best practice: • Different for the two exams and from the current scales to: ◦ Discourage score comparisons between the MCCQE Part I and II ◦ Discourage comparisons to current scales Exam Range M SD Range M SD MCCQE Part I 50-950 5001 1001 100-400 2503 303 MCCQE Part II 50-250 1502 202 50-250 1504 204 1 Based on all first-timers in spring 2015 2 Based on all candidates in spring 2015 3 Based on CMG-1st in spring 2018 4 Based on CMG-1st in fall 2018
  32. 32. |© 2018 MCC/CMC || New exam score ranges 32 How to compare scores with new exam score ranges: • Scores can be compared to the mean (M) and standard deviation (SD) of total scores using results from when the new exam score range is established ◦ Spring 2018 for MCCQE Part I ◦ Fall 2018 for MCCQE Part II • An MCCQE Part I score of 310 on the new 100 - 400 scale with a M of 250 and SD of 30 is two SDs above that of the group mean from spring 2018 • An MCCQE Part I score of 700 on the former 50 - 950 scale with a M of 500 and a SD of 100 is two SDs above that of the group mean from spring 2015
  33. 33. |© 2018 MCC/CMC || Standard setting 33 Examination Administration Standard Setting dates Results release MCCQE Part I Apr 16 – May 9, 2018 Jun 18 – 19, 2018 Jul 2018 MCCQE Part II Oct 27 – 28, 2018 Dec 10 – 12, 2018 Jan 2019 * Criterion referenced and not norm-referenced
  34. 34. |© 2018 MCC/CMC || Two exam results documents 34 Provided to candidates in their physiciansapply.ca account: 1. Statement of Results (SoR) • Candidate’s final result and total score, as well as pass score • Candidates choose whether to share with stakeholders 2. Supplemental Information Report (SIR) • Supplemental information about domain scores and comparative information • Candidates choose whether to share with stakeholders
  35. 35. |© 2018 MCC/CMC || Statement of Results
  36. 36. |© 2018 MCC/CMC || MCCQE Part I: Supplemental Information Report (SIR)
  37. 37. |© 2018 MCC/CMC || MCCQE Part II: Supplemental Information Report (SIR)
  38. 38. |© 2018 MCC/CMC || MCCQE Part I: Supplemental Information Report (SIR) Page 1: Standardized and simplified wording across all exams
  39. 39. |© 2018 MCC/CMC || MCCQE Part I: Supplemental Information Report (SIR) Page 2: New subscores based on new Blueprint
  40. 40. |© 2018 MCC/CMC || MCCQE Part I: Supplemental Information Report (SIR) Page 3: Domain definitions and per cent representation of each subscore
  41. 41. |© 2018 MCC/CMC Two exam results documents Embracing exam changes through Assessment Evolution 41 Guiding principle for SIRs • To provide useful and meaningful feedback to candidates supported by psychometric best practice and evidence while concurrently discouraging misuse of information • Facilitate ◦ Self-referencing to identify relative strengths and weaknesses ◦ Comparing own performance to that of first-time takers who passed the exam (i.e., those who met the standard)
  42. 42. |© 2018 MCC/CMC Results interpretation 42 Dimensions of Care Black bar: Expected score variation if taking this exam again; the shorter the bar, the better the measurement precision Coloured bars: How candidate performs relative to self across different domains; this candidate received a higher score in Health Promotion than Psychological Aspects Grey and coloured bars together: How candidate performs relative to first-time takers who took exam in spring 2018 and passed; this candidate’s score in Chronic is higher than the mean of first-time takers
  43. 43. CCME CCEM QUIZ!!! 43
  44. 44. |© 2018 MCC/CMC || Quiz instructions 1. Open the Kahoot! mobile app or go to www.kahoot.it on your browser 2. Enter the game PIN as shown on screen 3. Use the nickname generator and check to see if you see it in the game lobby on the shared screen 4. Answer the questions by selecting the corresponding shape/colour options displayed for each item – Try not to be distracted by colours 5. Bonus points for correctly answering multiple questions consecutively!
  45. 45. |© 2018 MCC/CMC || Results interpretation: resources 45 MCCQE Part I scoring details: mcc.ca/examinations/mccqe-part-i/scoring MCCQE Part II scoring details: mcc.ca/examinations/mccqe-part-ii/scoring MCC score interpretation guidelines for the MCCEE, NAC Examination, MCCQE Part I and MCCQE Part II: mcc.ca/research-and-development/score-interpretation
  46. 46. CCME CCEM REVIEW ….
  47. 47. |© 2018 MCC/CMC CCME CCEM Any questions or comments? Dr. Ian Bowmer, Yves Lafortune and Dr. Liane Patsula

×