Leah Mallory presented on the development of simulation-based assessment tools for general pediatrics milestones. Four milestones were identified as suitable for simulation-based assessment through a survey of program directors and experts, and separate groups were formed to identify or develop tools for each. The groups have conducted literature reviews and are in different stages of tool development or identification, with some requiring validation studies. The goals are to finalize study designs, obtain necessary approvals, and establish feasibility and validity of the tools across multiple sites in the next year.

1. Progress Report Presentation:
Development of simulation-based
assessment tools for the general
pediatrics milestones
Leah Mallory, MD
The Barbara Bush Children’s Hospital
at Maine Medical Center
IMSH 2016: San Diego, CA/ USA
International Network for Simulation-based Pediatric Innovation, Research and Education

2. • From survey of pediatric program directors and pediatric simulation
experts, 4 general pediatric milestones identified as “suitable to simulation
based assessment” and “gaps in current assessment methodologies
• In these areas, we seek to identify or develop usable simulation-
based assessment tools (*this has evolved into multiple projects
with separate teams)
– PBL1- Identify strengths, deficiencies, and limits in one’s knowledge and expertise
– SBP3- Work in inter-professional teams to enhance patient safety and improve patient
care quality
– ICS2- Demonstrate the insight and understanding into emotion and human response to
emotion that allows one to appropriately develop and manage human interactions
– Prof 6- The capacity to accept that ambiguity is part of clinical medicine and to recognize
the need for and to utilize appropriate resources in dealing with uncertainty
International Network for Simulation-based Pediatric Innovation, Research and Education
Aim(s) - Revised

3. • Fall 2014- survey of pediatric simulation experts and pediatric program
directors through APPD (published in Academic Peds Nov. 2015)
• January 2015- 4 separate groups form to identify or develop assessment
tools in prioritized areas
• Spring- Fall 2015- Groups performs literature search/systematic review of
assessment tools in respective areas
• Current- Each group ends up in a different place
International Network for Simulation-based Pediatric Innovation, Research and Education
Timeline & Current progress

4. • PBL1 (insight)- Utilize methodology of prior studies- multirater
assessment with gap analyses (Calhoun et al, JGME 2011 and Sim in
Healthcare 2009) comparing learner self assessment to observer
assessment. Opportunity to utilize this methodology with peds residents
as we validate and test tools emerging from groups below?
• SBP3 (interprofessional teamwork)- Tools focused on ‘teamwork’ emerge
from literature review, but less robust for interprofessional aspects-
performing modified Delphi including teamwork and interprofessional
eelements from identified tools
• ICS2 (empathy)- 2 well-validated tools, appropriate for our purposes
identified- no modification required? Plan to compare them, assessing
feasibility/usability for our purposes
• Prof 6 (ambiguity)- Systematic review reveals little, using modified Delphi
to develop new tool which will require validation
International Network for Simulation-based Pediatric Innovation, Research and Education
Timeline & Current progress

5. 1. Complete delphi/consensus exercises to develop tools for
(SBP3 and Prof 6)
2. Study design to evaluate tools for each area- how much can
methodology overlap given that each team is in slightly
different place?
3. Funding to carry out validation and/or feasibility assessment:
standardized patients, data collection and analysis
International Network for Simulation-based Pediatric Innovation, Research and Education
Needs / Challenges

6. Today, at INSPIRE @ IMSH 2016:
Use break-out groups to move forward with tool development
or validation phase study design for each group
In upcoming 3-6 months:
-Finalize study design and obtain central and peripheral IRB
-Funding
Later 2016 and 2017:
Establish feasibility and/or validity across multiple sites
International Network for Simulation-based Pediatric Innovation, Research and Education
Goals to accomplish

7. Name: Leah Mallory, MD
Institution: The Barbara Bush Children’s
Hospital at Maine Medical Center
E-mail: mallol@mmc.org
Phone: 207-662-1504
International Network for Simulation-based Pediatric Innovation, Research and Education
Contact Information