Mini-MAS-a direct observation tool in the era of competency based education

The Campaign for McMaster University
§
THE MINI MILESTONES ASSESSMENT
(MINI-MAS)
A DIRECT OBSERVATIONAL TOOL TO
ASSESS CLINICAL MILESTONES IN
THE ERA OF COMPETENCY-BASED
EDUCATION
MOYEZ B. LADHANI MD, FAAP, FRCPC

Introduction: CBME
the ‘tea-steeping model’, whereby medical educators “…put the
student (tea) in medical school (hot water) for a fixed period of
time and, voila! After a historically determined interval of time, we
assume a competent practitioner, like a good cup of tea, will
result”
Hodges, B. D. (2010). A tea-steeping or i-doc model for medical education?. Academic Medicine, 85(9 Suppl),
S34-44.

Introduction: CBME
Competency-based education is an approach to preparing physicians for
practice that is fundamentally oriented to graduate outcome abilities and
organized around competencies derived from an analysis of societal and
patient needs. It deemphasizes time-based training and promises
greater accountability, flexibility, and learner centeredness
Frank, J. R., Mungroo, R., Ahmad, Y., Wang, M., De Rossi, S., & Horsley, T. (2010). Toward a definition
of competency-based education in medicine: A systematic review of published definitions. Medical
Teacher, 32(8), 631-637.

What the beaver must do
§ Statement of learning outcome
§ Communication with staff and students
§ Educational strategies
§ Learning opportunities
§ Course content
§ Student progression
§ Assessment
§ Educational environment
§ Student selection
§ Harden, R. M. (2007). Outcome-based education--the ostrich, the peacock and the beaver.
Medical Teacher, 29(7), 666-671.
The International Conference on Residency Education | La Conférence internationale sur la formation des résidents
4

Introduction: Assessment
DOES
SHOWS
HOW
Faculty Observation
KNOWS HOW
KNOWS
Standardized Patients
MCQs
Impact on Patient
Clinical vignettes
Miller, G. E. (1990). The assessment of clinical skills/competence/performance. Academic Medicine, 65(9 Suppl), S63-7.

Introduction: WBA
“…the assessment of trainees and physicians across the
continuum of day-to-day competencies and practice in
authentic, clinical environments…It enables the evaluation of
performance in context”
Kogan, J. R., & Holmboe, E. (2013). Realizing the promise and importance of performance-based
assessment. Teaching & Learning in Medicine, 25(Suppl 1), S68-74.

Introduction: WBA
The In-Training Evaluation Report (ITER):
q Does not discriminate (Gray, 1996; Holmboe & Hawkins, 1998 )
q Completed retrospectively (Turnbull et al., 1998).
q Often faculty who have not observed are completing the
form (Epstein, 2007)
q Halo effect (Wilkinson & Wade, 2007)
q Raters fail to use the entire scale (Gray, 1996)

Introduction: WBA
mini-CEX
• The scale used in the mini-CEX is designed for linear
gradations of performance.
• The scores do not give the evaluators a point of reference to
help align a trainee to a category or score
• Faculty assessors resort to norm-referencing.
Crossley, J., & Jolly, B. (2012). Making sense of work‐based assessment: Ask the right questions, in
the right way, about the right things, of the right people. Medical Education, 46(1), 28-37.

Introduction: WBA
mini-CEX
• The raters do not use the full nine-point scale.
• The distribution is right shifted towards the higher end of the
scale.
• The use of the lower end of the scale is infrequent raising
concerns about identifying weaknesses.
• Individual competencies tended to be highly correlated.
Hawkins, R. E., Margolis, M. J., Durning, S. J., & Norcini, J. J. (2010). Constructing a validity argument
for the mini-clinical evaluation exercise: A review of the research. Academic Medicine, 85(9),
1453-1461.

Introduction: Designing a Better Tool
• WBA assessment tools should have anchors measuring the
trainees’ level of progression and development
• Assessors make more reliable judgments of performances
they can see clearly in a particular context or activity.
• The tool should focus on the competence relevant to the
activity, and avoid having multiple competencies to assess at
the same time
Crossley, J., & Jolly, B. (2012). Making sense of work‐based assessment: Ask the right questions, in the right
way, about the right things, of the right people. Medical Education, 46(1), 28-37.
Crossley, J., Davies, H., Humphris, G., & Jolly, B. (2002). Generalisability: A key to unlock professional
assessment. Medical Education, 36(10), 972-978.

Dreyfus and Dreyfus
Figure 2 General Curve of skills Acquisition. (ten Cate et al.,
2010)

Introduction: Faculty Development

Purpose
The purpose of this study is to:
1. Implement a competency-based curriculum into the McMaster
University, pediatric residency program.
2. Develop a tool, the Mini Milestones Assessment (Mini-MAS)
to assess six medical competencies and progression through
milestones using the Dreyfus Developmental Model.
3. Test the psychometric theories to assess the reliability,
validity, acceptability and feasibility of the Mini-MAS tool.

Research Question
Is the Mini-MAS a valid, reliable, acceptable and feasible tool for
the assessment of milestones in history taking, physical exam
skills, clinical reasoning, communication and collaboration for
PGY 1 and PGY 4 pediatric residents at McMaster Children’s
Hospital?

Methods
• Implement CBME curriculum.
Clinical Exposure-Total 22 weeks:
q CTU-4 weeks
q Community Brampton-4 weeks
q Community St. Joseph’s Healthcare-4 weeks
q Float call at McMaster-6 weeks

Methods
• 12 PGY 1 residents at McMaster Children’s Hospital were
required to complete 40 observations
• 10 history taking, 10 physical exam
• 5 clinical reasoning, 5 communication with families,
5 communication with staff and 5 collaboration
• during the 2013-2014 academic year.
• 9 PGY 4 residents were also observed for the same
competencies over the same time period. This group was
required to complete 15-20 encounters.

Methods:
• Following the study period, a survey was completed by the
residents and faculty to assess acceptability and feasibility of
the Mini-MAS tool.
• Kane’s validity framework which is divided into four
components (scoring, generalization, extrapolation and
decision) was used to evaluate the Mini-MAS tool.

Procedures
§ Learners
§ Assessors
§ Other Data

Results

12 PGY 1
474
observations
39 removed
435 (mean
36)
9 PGY 4
1 lost book
1 LOA
7 PGY 4
96 forms
(mean of 16)

Discussion

Scoring
• Learners met requirements 90% and 100% completion rates
• 76% by faculty 24% by residents
• Residents assessors more lenient but not significant
• Trend for observations occurring in later half of year
• Faculty engagement 3.9 PGY 1 and 4.6 PGY 4

Scoring
• Faculty completed the forms in a timely manner 4.8, 5.3, 5.3
• Faculty provided valuable feedback 5.3, 5.0, 5.6
• Faculty felt appropriately trained, though wanted more training
on providing feedback
• Scale was used appropriately 2-5 for PGY 1 and 3-5 fro PGY 4

Scoring
• Learners started with different skill levels
• Scores also help determine where a PGY 1 resident should be
• Individual competencies did not correlate-positive finding
• Faculty and residents found descriptors were long and
sometime vague.

Generalization
• 435 encounters for PGY 1 (mean=36) and 96 encounters for
PGY 4 ( mean =16)
• 45 assessors including the 8 senior residents
• Wide variety of clinical cases
• CTU, ER, SRC, Community Office

Generalization
• The G coefficient overall was 0.8 for the PGY 1 group
• The variance analysis showed the majority of the variance
was from the trainee as would be expected
• For the PGY 4 group, the G coefficient was 0.5

Generalization
• A D-study conducted showed increasing the number of
observations to 10-12 could increase the G-coefficient to
acceptable levels for history taking, communication with
families, communication with health care professionals and
clinical reasoning

Extrapolation
§ Involved observation of what really happens in clinical
practice across a variety of settings.
§ Progression of skills for PGY 1 residents through the year
§ PGY 4 scores increased through the year but not significant
§ The significant difference between PGY 1 and PGY 4
residents overall and in all the competencies

Extrapolation
§ This study did not look at correlation with exit high stakes
exams nor how residents do in practice
§ There were comparisons done with concurrent assessment
tools used in the program, the mini-CEX, MCQ, SAQ and
OSCE exams

Decision
§ Progression of scores through the academic year and that
there is a difference between levels of training makes these
scores defendable.
§ Residents and faculty both reported that the implementation
of the tool improved the frequency of observation,4.0, 4.9, 5.1
and valuable feedback was provided 5.3, 5.0, 5.6

Decision
§ The PGY 4 residents further reported the assessment
process influenced their education 5.1
§ Did not look at the effect remediation and improvement in
scores

Acceptability and Feasibility
§ High completion rate
§ Residents and faculty satisfied with tool 3.8, 4.6, 5.3
§ Faculty satisfaction with tool 7.7/9
§ One lost book

Conclusion
§ We successfully implemented a CBME pilot program in our
residency
§ The Mini-MAS added as a formative assessment mode to a
multi-modal assessment program will benefit the trainee,
informing them on where they stand compared to their level of
training, what competencies they can improve on and how
they can do that.

Conclusion
§ Work-based assessment tool is one that assess trainees
across the continuum of competencies in clinical
environments enabling the evaluation of performance in
context
§ Kogan, J. R., & Holmboe, E. (2013). Realizing the promise and importance of performance-based
assessment. Teaching & Learning in Medicine, 25(Suppl 1), S68-74.

Future Direction
• Having scheduled assessments weekly.
• More assessment by senior residents or fellows to improve
acceptability.
• Simplifying and shortening the anchors.
• Continued faculty training with an emphasis on effective
feedback.

Future Direction
• Expanding the assessment of trainees to all levels of training
and all rotations.
• Consider different competencies to assess for different levels
of training.
• Further studies to assess concurrent validity.

Thank you
§ Dr. Kelly Dore
§ Dr. Meghan
McConnell
§ Dr. Karen McAssey
§ Dr. Jonathan
Sherbino
§ Sharyn Kreuger
§ Pediatric Residents
§ My Family
§ My Online
Classmates

Mini-MAS-a direct observation tool in the era of competency based education

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Mini-MAS-a direct observation tool in the era of competency based education

Similar to Mini-MAS-a direct observation tool in the era of competency based education (20)

Recently uploaded

Recently uploaded (20)

Mini-MAS-a direct observation tool in the era of competency based education