Impacts and Outcomes: McMaster University, IMG Pre-Residency

1. Inge Schabort
McMaster University, IMG Pre-Residency IMG Program
Plenary IV: Research on Impacts and
Outcomes

2. Plenary IV
Research on Impacts and Outcomes
Touchstone Symposium:
Perspectives on Successful Transitions for
Internationally Educated Health Professionals.
Dr Inge Schabort MB ChB CCFP FCFP
Associate Professor, IMG Coordinator
McMaster University
Schabort

3. Conflicts
• I am an IMG
• None other
Schabort

4. Objectives
• Chronological tour of Ontario IMG literature
and research
• A few significant IMG outcome research
studies
Schabort

5. Dealing with Discrimination
Schabort

6. Ontario commissioned Thomson and
Cohl Judicial reports 2011
• 33 recommendations
• Most recommendations accomplished since
Schabort

7. http://www.health.gov.on.ca/en/common/min
istry/publications/reports/thomson/v1_thoms
on.pdf
Schabort

8. Schabort

9. Schabort

10. Thomson Working Group Struck
• 33 recommendations worked through since
2011
Schabort

11. Schabort

12. The selection of international
medical graduates to
residency programs
What selection information best
predicts success at college
certification exams?
Inge Schabort, Sharon Cameron &
Lawrence GriersonSchabort

13. Schabort

14. Background
• Due to the physician shortage, more IMG’s than ever before are admitted
to residency programs in Canada.
0
500
1000
1500
2000
2500
1998 2004 2009
IMGsenteredintogradauteprograms
Schabort

15. COLLEGE EXAM PASS RATE
(FMEC IMG report)
FAMILY MEDICINE
YEAR CMG IMG
residency
trained
2007 90.4% 66.0%
2008 74.0%
2009 64.0%
2010 51.0%
ROYAL COLLEGE PRIMARY SPECIALTY
EXAMS first attempt
YEAR CMG IMG
2005-2009 96% 75%
Walsh A, Banner S, Schabort I, Armson
H, Bowmer I, Granata B. International
Medical Graduates - Current Issues.
Members of the FMEC PG consortium;
2011.
Schabort

16. Literature Review
• IMG Predictors for Success
– Age (Kanna 2009)
– Time since medical school graduation(Bates 2001),
(Bates 2000), (Bessant 2006)
– Recent clinical experience (Part and Markert 1993)
– Performance on Standardized exams (Kanna 2009)(
Part and Markert 1993), (Shiroma 2010),(Perez 2009)
– Variable undergraduate education amongst IMGs (Van
Zanten and Boulet 2008)
– English Language proficiency
– Doing a clerkship in Canada before residency
(MacLellan 2012)
Schabort

17. More predictors
• Medical School attended
• Country of training as predictor for college
complaints
Schabort

18. Hypotheses
• Age; Time since graduation/last practice (Kanna, 2009;
Bessant, 2006)
• Language and Cultural Difficulties (Bates, 2000; 2001)
• Differences in medical education (vanZanten & Boulet,
2008)
• Professional Experience (clinical, internship, research)
(MacLellan, 2012)
Schabort

19. Purpose
• As a first step in untangling the various proposed
factors/mechanisms that impact IMG certification success this
study reports the information that is available to the
McMaster University residency program at the time of the
Canadian Residency Matching Service (CaRMS) that predicts
IMG success at the college certification exams.
Schabort

20. Methods
• Linear and logistic regression analyses determined the degree to which
the information available at the time of residency selection predicts the
performance of McMaster University IMG residents on the CFPC (n = 69)
and RCPSC (n = 85) certification examinations between 2005 and 2011.
• Regression models accounted for:
– Demographics (age, nationality, first language)
– Education (country of medical schooling, years since
graduation)
– Experience (internship, residency, professional, research)
• MCC-EE scores were correlated with certification exam scores
Schabort

21. Results
• Demographic Factors
– CFPC IMGs that do not speak English as a first language
performed about 4 points better on the SOOs, ß = 4.4, p = .05.
– CFPC IMGs born outside of Canada did about 3.5 points better
on the SAMPS than Canadians who studied abroad, ß = 3.4, p =
.049.
– Although not significant, age was a negative predictor for CFPC
IMGs, odds ratio = .86, p = .122, and RCPSC IMGs, odds ratio =
.84, p = .13. With each year older, CFPC IMGs and RCPSC IMGs
were 14% and 16% , respectively, less likely to pass their overall
certification exams.
Schabort

22. Results
• Education Factors
– For every unit increase of the Human
Development Index of the country of medical
schooling, CFPC IMGs' scores on the SOOs
increased by 18 points, ß = 17.8, p = .046.
– Years since graduation was not a significant
predictor
Schabort

23. Results
• Experience Factors
–For CFPC IMGs, professional experience had a significant
negative relationship with outcomes on the Simulated Office
Orals (SOOs), ß = -5.1, p = .042, and Short Answer
Management Problems (SAMPS), ß = -4.4, p = .045.
– RCPSC IMGs that participated in a previous internship increased their odds of
passing the RCPSC certification exam by 4 times; odds ratio = 4.1, p = .02.
– The impact of professional experience on CFPC IMGs’ overall CFPC Certification
outcomes wasn't significant. However, the analysis does indicate that those with
professional experience were approximately 60% more likely to be unsuccessful,
odds ratio = .39, p = .18.
Schabort

24. Results
• MCC-EE scores were positively correlated with CFPC IMGs’
performances on the SAMPS (r = .481, p < .001) and overall
certification success (r = .317, p = .018) as well as RCPSC IMGs’
certification success (r = .264, p = .027).
Schabort

25. Narrowing the Hypotheses
• Age
– Age-related decline or associated factors?
• Differences in Medical Education
– Which Schools in which Countries?
– Normalizing Transcripts…an HDI by transcript algorithm?
• The Role of Previous Experience
– The cultural implications of practice experience?
• …And the seemingly less relevant…
– Time Since Medical School Graduation
– Language Difficulties
Schabort

26. Limitations and/or Next Steps
• Single Institution Data Set
– Collaborative data analysis across institutions
• Limited Test data
– MCC inclusion of QE1 and QE2 scores
– REB approval has just been obtained
Schabort

27. Schabort

28. Schabort

29. Schabort

30. • Book: Women in Medicine 2017
Schabort

31. Schabort

32. Surveys of IMGs -Themes identified
Schabort

33. Schabort

34. Schabort

35. Schabort

36. Schabort, Cervin, Bandiera
Schabort

37. Schabort

38. Schabort

39. Schabort

40. Schabort

41. Schabort

42. MMI and professionalism
• Hoffmeister found that “there is evidence that
the MMI offers a reliable and valid assessment
of professionalism in IMG doctors applying for
Canadian family medicine residencies and that
this clinically situated MMI assessed facets of
competency other than those assessed by the
OSCE.”

43. MMI’s
• MMIs have several strengths: they can be
designed to best fit relevant competencies, e.g.
teamwork, communication skills. Studies suggest
that MMIs are a reliable measure of non-
cognitive attributes such as empathy, cultural
sensitivity, advocacy and communication skills
which are all important attributes for a doctor
which can be observed in a specific context.
Inferences may be made that MMIs help to
protect patient safety as candidates are being
observed and rated on doing a work sample.

44. MMI better predictor than OSCE
• Eva et al. (2004b) and Reiter et al. (2007)
found MMIs to be a better predictor of
performance on OSCEs than more established
methods of selection (e.g. GPAs and
traditional interviews). MMIs cannot be
coached for as there is not one formulaic
answer and therefore security breaches in the
selection process would not affect the
performance of the candidate (Reiter et al.,
2005 and Griffin et al., 2008).

45. MMI Costs
• MMIs are initially costly and time consuming
to set up in terms of person hours and
expense (e.g. development of the blueprint of
competencies and scenarios, training the
raters/interviewers and hiring the number of
rooms required for the stations). However the
value in terms of the reliability and validity
over more traditional unstructured interviews
may outweigh these initial costs.

46. Computer based non-cognitive pre-
interview test
• Dore et al explored the development of a non-
cognitive pre-interview screening test that
correlates with the well-validated MMI and
performs multiple sample evaluation of non-
cognitive skills (CMSENS) .This has possible
future implications for IMG evaluation eg. to
be administered remotely using the computer
platform currently used for eg the MCCEE
where the IMG can perform the assessment at
any of the multiple sites outside of Canada.

47. However…
Schabort

48. Questions?
• ischabo@mcmaster.ca
Schabort

49. References• Examindation Results, Medical Council of Canada. Retrieved from www.mcc.ca/en/NAC/examination_results.shtml
• Akl EA, Izuchukwu IS, El-Dika S, Fritsche L,Kunz R, Schunemann HJ. Integrating an Evidence-Based Medicine Rotation into an Internal Medicine Residency
Program. (2001) Acad Med. Sep;79(09):897-904.
• Al-Amaie, Sameeh M,Al-Baghli N,Barriers Facing Physicians Practicing Evidence-Based Medicine in Saudi Arabia. Journal of Continuing Education in the
Health Professions 24(3) 163-170.
• Allan GM, Manca D, Szafran O, Korownyk C. (2007) EBM a challenge for international medical graduates. Fam Med. Mar;39(3):160.
• Allan GM, Korownyk C, Tan A, Hindle H, Kung L, Manca D. (2008) Developing an Integrated evidence-based medicine curriculum for family medicine
residency at the University of Alberta. Acad med. Jun;83(6):581-7.
• Altmaier, E. et al., (1989). Cross-Institutional Stability of Behavioral Criteria Desirable for Success in Radiology Residency. Investigative Radiology, 24(3),
pp.249-51
• Andrew, R. (2010). How do IMGs compare with Canadian Medical Graduates in a Family Practice Residency Program? Can Fam Physician, 56e:318-22.
• Bates J, A. R. (2000). Program for Licensure for International Medical Graduates in British Columbia - 7 years experience. CMAJ, 162(6): 801-3.
• Bates J, A. R. (2001). Untangling the roots of some IMGs' poor academic performance. Acad Med, 76(1):43-6.
• Bell, J., Kanellitsas, I., & Shaffer, J. (2002). Selection of Obstetrics & Gynecology Residents on the Basis of Medical School Performance. American Journal
of Obstetrics & Gynecology , 186:1091-4.
• Bessant, R., Bessant, D., Chesser, A., & Coakley, G. (2006). Analysis of Predictors of Success in the MCRP (UK) PACES examination in candidates attending a
revision course. Postgraduate Medical Journal , 82:145-149.
• Blonski, J., & Rahm, S. (2003). The Relationship of Residency Performance to Match Status and US versus International Medical Graduates. Family
Medicine, 35(2): 100-4.
• Blouin, D., & Dagnone, J. (2008). Performance Criteria for Emergency Medicine Residents: A Job Analysis. Canadian Journal of Emergency Medicine , 10(6):
539-44.
• Boulet, J., McKinley, D., Rebbecchi, T., & Whelan, G. (2007). Does Composition Medium Affect the Psychometric Properties of Scores on an exercise
Designed to Assess Written Medical Communication Skills? Advances in Health Sciences Education , 12:(2): 157-167.
• Boulet, J., Rebecchi, T., Denton, A., McKinley, D., & Whelan, G. (2004). Assessing the Written Communication Skills of Medical School Graduates. Advances
in Health Sciences Education , 9:(1);47-60.
• Boulet, J., Swanson, D., Cooper, R., Norcinii, J., & McKinley, D. (2006). A comparison of the characteristics and examination performances of US and non-
US citizen international medical graduates who sought Educational Commission for Foreign Medical Graduates certification. Acad Med, 81 (10 Suppl):
S116-9.
• Brotherton, S., & Etzel, S. (2007). Graduate Medical Education. Journal of American Medical Association , 298(9):1081-1096.
• Brown, C., Wakefield, S., & Bullock, A. (2003). The Selection of GP trainees in the West Midlands: Second Audit of Assessment Centre Scores by Ethnicity
and Country of Qualification. Medical Teacher , 25 (6) 649-653.
• Cartwright CA, Korsen N, Urbach LE. (2002) Teaching the Teachers: Helping faculty in a family practice residency improve their informatics skills. Acad Med
May;77(5):385-91.
• Chamberlain, T.C., Catano, V.M. & Cunningham, D.P, (2005). Personality as a predictor of professional behavior in dental school: comparisons with dental
practitioners. Journal of Dental Education. 69(11), pp.1222-37.
• Chamorro-Premuzic, T., Furnham, A. & Ackerman, P.L., (2006). Incremental validity of the typical intellectual engagement scale as predictor of different
academic performance measures. Journal of Personality Assessment. 87(3), pp.261-8.
• Dirschl, D., Dahners, L., Adams, G., Crouch, J., & Wilson, F. (2002). Correlating Selection Criteria with Subsequent Performance as Residents. Clinical
Orthopaedics and Related Research , 399:265-271.
• Dore, K. L., Reiter, H. I., Eva, K. W., Krueger, S., Scriven, E., Siu, E., et al. (2009). Extending the Interview to All Medical School Candidates - Computer-Based
Multiple Sample Evaluation of Noncognitive Skills (CMSENS). Academic Medicine, 84:10 pp S9-S12.
• Eva, K., Reiter, H., Rosenfeld, J., & Norman, G. (2004). The ability of the multi-mini interview to predict preclerkship performance in medical school.
Academic Medicine, 79(10 Suppl), pp. S40-2.
• Eva, K., Reiter, H., Rosenfeld, J., & Norman, G. (2004). The relationship between interviewers' characteristics and ratings assigned during a multiple mini-
interview. Academic Medicine, 79(6), pp 602-9.
Schabort

50. References Continued
• Korenstein D, Dunn A, McGinn, T. Mixing it up: Integrating evidence-based medicine and patient care. Acad Med Jul;77(7):741-2.
• Lee, A., Golnik, K., Oetting, T., Beaver, H., Boldt, H., Olson, R., et al. (2008). Re-Engineering the Resident Application Process in Opthalmology: A
Literature Review and Recommendations for Improvement. Survey of Opthalmology , 53 (2) 164-176.
• MacLellan, A., Brailovsky, C., Rainsberry, P., Bowmer, A., & Desrochers, M. (2010). Examination outcomes for International Medical Graduates
pursuing or completing Family Medicine Residency training in Quebec. Can Fam Physician, 56:912-8.
• McClintock, J., & Gravlee, G. (2010). Predicting Success on the Certification Examinations of the American Board of Anesthesiology. Anesthesiology ,
112:212-9.
• Papadakis, M., G, A., Holmbee, E., & Lipner, R. (2008). Performance during Internal Medicine Residency Training and Subsequent Disciplinary action
by Stat Licensing Boards. Annals of Internal Medicine , 148(11); 869-876.
• Part, H., & Markert, R. (1993). Predicting the first year performances of international medical graduates in an internal medicine residency. Acad Med,
68(11): 856-8.
• Perez JA, Greer ES. (2009) Correlation of United States Medical Licensing Examination and Internal Medicine In-Training Examination performance.
Adv in Health Sc Educ 14;753-758
• Pilotto LS, Duncan GF and Anderson-Wurf J. (2007). Issues for clinicians training international medical graduates: a systematic review. Medical
Journal of Australia; 187(4):225-228.
• Poole, A., Catano, V.M. & Cunningham, D.P. (2007). Predicting performance in Canadian dental schools: the new CDA structured interview, a new
personality assessment, and the DAT. Journal of Dental Education. 71(5), pp.664-76.
• Posthuma, R., Morgeson, F., & Campion, M. (2002). Beyond Employment Interview Validity: A Comprehensive Narrative Review of Recent Research
and Trends Over Time. Personnel Psychology , 55:1-81.
• Quintero, A., Segal, L., King, S., & Black, K. (2009). The Personal Interview: Assessing the Potential for Personality Similarity to bias the Selection of
Orthopaedic Residents. Academic Medicine , 84:1364-1372.
• Rao, N., Meinzer, A., Primavera, L., & Augustine, A. (1991). Psychiatry Residency Selection Criteria for Maerican and Foreign Medical Graduates.
Academic Psychiatry , 15:69-79.
Schabort

51. References Continued
• Griffin, B., Harding, D., Wilson, I., & Yeomans, N. (2008). Does practice make perfect? The effect of
coaching and retesting on selection tests uses for admission to an Australian medical school. Med J.
Aust, Oct 6;189(7):416.
• Hall P, Keely E, Dojeiji S, Byszewski A, and Marks M. (2004). Communication skills, cultural
challenges and individual support: challenges of international medical graduates in a Canadian
healthcare environment. Medical Teacher 26(2):120-125
• Hayden, S., Hayden, M., & Gamst, A. (2005). What Characteristics of Applicants to Emergency
Residency Programs Predict Future Success as an Emergency Medicine Resident? Academic
Emergency Medicine, 12 (3) 206-10.
• Hofmeister, M., Lockyer, J., & Crutcher, R. (2009). The multiple mini-interview for selection of
international medical graduates into family medicine residency education. Medical Education,
Volume 43, Issue 6 pages 573-579.
• Illing, J., Campbell, M., Kergon, C., Thompson, N., Burford, B., Morrow, G., et al. (n.d.). Selection
Methods for Foundation Programme: A Literature Review. Retrieved November 14, 2010, from
www.medschools.ac.uk/aboutus/projects/documents/appendix%20K%20Newcastle%20Literature
%20Review.pdf
• Kanna, B., Gu, Y., Akhuetie, J., & Dimitrov, V. (2009). Predicting Performance using Background
Characteristics of International Medical Graduates in an inner-city University-Affiliated Internal
Medicine residency training program. BioMed Central, 9:42.
•
Schabort

52. 2017 Perspectives Symposium
March 2, 2017
#Perspectives2017