ProMIS, an augmented reality surgical simulator, has been shown to be valid based on several studies: 1. ProMIS metrics like instrument path length and smoothness strongly correlate with FLS scores and can differentiate experience levels as well as FLS scoring. 2. ProMIS FLS tasks scored with traditional FLS metrics or ProMIS metrics can discriminate between novices and experts, validating its construct. 3. Automated ProMIS metrics like time, path length and smoothness correlate well with standard FLS performance and can reliably predict readiness for FLS certification.

1. Validated!
Proven wiith FLS
Proven w th FLS
Better than VR
Better than VR
Preferred by users
Preferred by users
ProMIS and FLS
• ProMIS metrics are “excellent predictors of scores
in the standard FLS simulator” and “predict
readiness for FLS Certification”1 2 3
• ProMIS metrics are valid on peg transfer, pre-tied
loop placement and knot-tying tasks4
• “Initial trials of the metrics on the FLS Precision-
Cutting Task show that ProMIS is as accurate as
5
the current method”
ProMIS assesses performance on Augmented Reality combines VR
real models where real haptics with a physical model allowing • FLS tasks are transferable to the ProMIS simulator
are important, eg in Suturing and virtual bleeding and real haptics.
Knot-tying with traditional FLS scoring and intrinsic ProMIS
metrics being good measurement tools. A ProMIS
total path length <4000 mm or total smoothness
25
<6000 reliably predicts a passing FLS score.
ProMIS vs pure VR
• ProMIS out-performed the virtual reality
24
simulators
ProMIS has pure VR Modules, eg Vision technology enables • Scores for ProMIS were significantly higher than
for Instrument Handling. Users tracking of errors and automatic
still use real instruments. calculation of dissected tissue.
for SurgicalSIM for overall realism, thread
behavior, reflection of clinical ability, and overall
educational value.6
• In comparison with LapSim, ProMIS was regarded
by all participants as a better simulator for
laparoscopic skills training on all tested features7
ProMIS surgical simulator • “Only [ProMIS] was able to distinguish between
advanced trainees and beginners (and) was
ProMIS Modules range from Basic Laparoscopic Skills to graded more realistic (70% vs 33%) and more
MIS procedures like LapColectomy. For more information useful (83% vs 62%)” than Xitact8
on ProMIS, please contact us at:
• The ability of performance metrics of [ProMIS] to
Email: promis@haptica.com
discern predicted performance differences
U.S. tel: +1 617 342 7270
between experts and non-experts was better
RoW tel: +353 (0)1 676 7310
than for SimSurgery’s robotic surgery simulator and
1
SAGES 2006 S064 Ritter et al,
SurgicalSIM 9
2
SAGES 2006 P237 McCluney MD, et al,
3
4
SAGES 2007 P279 McCluney et al, ProMIS: the preferred Simulator
SAGES 2006 S065 Vuong et al,
5
SAGES 2007 ETP057 Young et al, • ProMIS can be used effectively with the DaVinci
6
SAGES 2006 P224 Fellinger, et al,
7
robot to obtain performance data with robotic
World J Surg. 2007 Apr;31(4):764-72. Botden et al,
8
SAGES 2007 P270 Hahnloser et al, instrumentation10
9
SAGES 2007 S077 Lin et al,
10
SAGES 2006 Narula et al, • Residents believe that ProMIS is easy to use and
11
24
SAGES 2006 P219 Chang et al, improved their operative skills11
Heinrichs, et al 2007
25
Hungness, et al 2008

2. Validation
1. SAGES 2006 Scientific Session S064 2. SAGES 2006 Education/Outcomes–P237
CONCURRENT VALIDITY OF AUGMENTED REALITY METRICS VALIDATION OF THE PROMIS HYBRID SIMULATOR USING
APPLIED TO THE FUNDAMENTALS OF LAPAROSCOPIC A STANDARD SET OF LAPAROSCOPIC TASKS
SURGERY (FLS) A L McCluney MD, L S Feldman MD, G M Fried, Steinberg-
E. Matt Ritter MD, Tamara W Kindelan MD, Curtis Michael, Bernstein Centre for Minimally Invasive Surgery, McGill
Elisabeth A Pimentel BA, Mark W Bowyer MD, 1NCA Medical University Health Centre, Montreal, QC, Canada
Simulation Center, Department of Surgery, Uniformed Services
Introduction
University, 2Division of General Surgery, National Naval Medical
Center, Bethesda Maryland SAGES Fundamentals of Laparoscopic Surgery (FLS) tasks are
validated measures of technical skills, however FLS scoring
Objective
requires a trained proctor. The ProMIS simulator (Haptica;
Current skills assessment in the Fundamentals of Laparoscopic Dublin, IR) is a ‘hybrid’ system with physical and virtual
Surgery (FLS) program is labor intensive requiring one proctor for reality tasks. It has the flexibility to incorporate any
every 1-2 subjects. The ProMIS Augmented Reality (AR) simulator physical task and score it with ProMIS metrics. Metrics are
(Haptica, Dublin IRE) allows for objective assessment of physical automated and report motion analysis data as instrument
tasks through instrument tracking technology. We hypothesized path length (PL) and instrument smoothness (IS). The
that the ProMIS metrics could differentiate between ability purpose of this study was to test for construct and
groups as well as standard FLS scoring with fewer personnel concurrent validity using FLS tasks in the ProMIS simulator.
requirements.
Methods
Methods
5 laparoscopic novices and 5 experts performed FLS tasks in
We recruited 60 volunteer subjects. Subjects were stratified both the standard FLS simulator box and the ProMIS
based on their laparoscopic surgical experience. Those who had simulator. Assessments were made based on FLS metrics, as
performed more than 100 laparoscopic procedures were well as PL and IS. Student’s t-test was used to compare the
considered experienced (n=8). Those with less than 10 mean (SD) of total scores for novices and experts. Pearson’s
laparoscopic procedure were considered novices (n=44). The rest correlations were calculated for standard FLS scores in
were intermediates (n=8). All subjects performed up to 5 trials relation to ProMIS FLS scores, total PL, and total IS.
of the peg transfer task from FLS in the ProMIS simulator. FLS Significance was defined as p < 0.01 (*).
score, instrument path length, and instrument smoothness
Results
assessment were generated for each trial.
Standard FLS scores correlated strongly with ProMIS FLS
Results
scores (r=0.90), total PL (r=-0.83), and total IS (r=-0.78)
For each of the 5 trials, experienced surgeons outperformed (p&lt; 0.01).
intermediates who in turn out performed novices. Statistically
Conclusions
significant differences were seen between the groups across all
trials for FLS score (p < 0.001), ProMIS path length (p <0.001) and FLS tasks performed in ProMIS, when scored by either
ProMIS smoothness (p < 0.001). When the FLS score was traditional FLS metrics or by intrinsic ProMIS metrics,
compared to the path length and smoothness metrics, a strong discriminate effectively between novices and experts. Based
relationship between the scores was apparent for novices (r = on the observed correlations, ProMIS FLS scores, total PL,
0.78, r = 0.94 , p < 0.001) respectively), intermediates (r = 0.5, p and total IS are excellent predictors of scores in the
= 0.2 , r = 0.98, p < 0.001), and experienced surgeons (r = 0.86, p standard FLS simulator.
= 0.006, r = 0.99, p < 0.001)
Conclusions
The construct that the standard scoring of the FLS peg transfer
task can discriminate between experienced, intermediate, and
novice surgeons is validated. The same construct is valid when
the task is assessed using the metrics of the ProMIS. The high
correlation between these scores establishes the concurrent
validity of the ProMIS metrics. The use of AR for objective
assessment of FLS tasks could reduce the personnel requirements
of assessing these skills while maintaining the objectivity.
www.haptica.com | email: promis@haptica.com | tel (NAm) +617 342 7270 | tel (RoW) +353 (0) 1 676 7310

3. Validation
3. SAGES 2007 Education/Outcomes – P279 4. SAGES 2006 Scientific Sessions S065
AUTOMATED PROMIS SIMULATOR METRICS PREDICT WHAT CAN MOTION DERIVATIVES TELL US ABOUT SKILL
READINESS FOR FLS CERTIFICATION PERFORMANCE?
Anthony L McCluney MD, J Cao, G N Polyhronopoulos MD, D D Laurel N Vuong BS, Steven D Schwaitzberg MD, Caroline G
Stanbridge, L S Feldman MD, G M Fried MD, Steinberg-Bernstein Cao PhD, Tufts University School of Medicine, Cambridge
Centre for Minimally Invasive Surgery, McGill University, Health Alliance, Tufts University School of Engineering
Montreal, QC, Canada
Surgical simulators are a popular topic of discussion on
Introduction training in laparoscopic surgery. They reduce the need to
use human cadavers or animal models for skills
SAGES Fundamentals of Laparoscopic Surgery (FLS) tasks are
development. A subset of the MISTELS methodology has
validated measures of technical skills. Certification requires
been employed in the manual skills assessment for the
travel to a testing site and a fee, thus a reliable method of
Fundamentals Laparoscopic Skills (FLS) program because it
predicting readiness for the exam would be advantageous. The
was shown to be a valid discriminator of surgical
ProMIS simulator (Haptica) provides automated scoring. FLS tasks
experience. Pure performance outcome, such as time to
can be placed in the ProMIS simulator and scored using time
task completion and number of errors, is used for scoring,
(TT), as well as motion analysis metrics: instrument path length
which is dependent on the consistency of the scorer. A new
(PL) and instrument smoothness (IS). This study was designed to
simulator environment has been created which uses motion
evaluate these automated ProMIS metrics and their ability to
tracking for measurement of performance outcome
predict readiness for FLS certification.
measures and motion derivatives such as smoothness and
Methods efficiency.
33 subjects (12 students, 16 residents PGY 1-4, and 5 experts) The purpose of the study was to determine if the motion
performed FLS tasks in the standard simulator and in ProMIS. derivatives can be used to automatically and objectively
Tasks were scored by FLS and ProMIS metrics. For each ProMIS discriminate experience levels. Twenty-one subjects (6
metric, the total score was calculated by summing the scores for medical students, 14 surgical residents, and 1 expert
the 5 FLS tasks. Pearson’s correlations were calculated for surgeon) were recruited to perform the following tasks: peg
ProMIS metrics versus standard FLS scores. Multivariate transfer, pattern cutting, pre-tied loop placement,
regression analysis identified independent predictors of standard extracorporeal and intracorporeal knot-tying in the new
FLS performance. These variables were then used for sensitivity simulator environment. Subjects were evaluated on time to
and specificity calculations in order to establish a ProMIS pass- completion, errors, smoothness and total path length (used
fail score for predicting readiness for FLS certification. to calculate efficiency).
Significance was defined as p<0.05. Results show that experience level is still distinguishable
when using task-dependent parameters to evaluate
Results
performances during peg transfer (p= 0.035), pre-tied loop
TT (r= -0.82), PL (r= -0.56), and IS (r= -0.75) all correlated placement (p= 0.022), extracorporeal (p= 0.0006) and
significantly with standard FLS score. Multivariate regression intracorporeal (p= 0.025) knot tying in this new simulator
analysis identified TT as the strongest predictor of FLS score. A environment.
TT score of 1000 maximizes sensitivity and specificity and was
Evaluation of performance using task-independent
identified as the pass-fail for reliably predicting FLS
parameters significantly distinguished training level in three
performance.
tasks: (1) smoothness of the left instrument was significant
Conclusions as a function of experience level in extracorporeal knot-
tying (p= 0.016), (2) efficiency (total path length divided by
Automated ProMIS metrics correlate well with standard FLS
time to completion) was also significant in the movement of
performance. In this study sample, a TT score less than 1000
the right tool as a function of experience level in peg
reliably predicted a passing FLS certification score.
transfer (p= 0.0011) and (3) pre-tied loop placement (p=
0.013979).
This preliminary analysis shows that automatic and
objectively measured motion derivatives can be associated
with the level of experience. These results indicate a
potential for the application of an automatic and objective
means of skills evaluation.
www.haptica.com | email: promis@haptica.com | tel (NAm) +617 342 7270 | tel (RoW) +353 (0) 1 676 7310

4. Validation
5. SAGES 2007 ETP057 6. SAGES 2006 Education/Outcomes–P224
OBJECTIVE MEASUREMENT OF FLS PRECISION CUTTING TASK COMPLEX LAPAROSCOPIC TASK PERFORMANCE ON TWO
Derek Young, Fiona Slevin, Derek Cassidy, Donncha Ryan, Haptica NEW COMPUTER-BASED SKILLS TRAINING DEVICES
Inc Erika K Fellinger MD, Michael E Ganey MD, Anthony G
Gallagher PhD, Daniel J Scott MD, Ron W Bush BS, Neal E
The Precision-Cutting Task in the SAGES/ACS FLS Program
Seymour MD, Department of Surgery, Baystate Medical
requires the user to dissect a circle of specific size and shape
Center, Springfield, MA
from a marked piece of mesh. Currently, measurement of the
accuracy and area dissected is done by observation and by Introduction
measuring the dissected mesh on a measurement grid. Using
New computer-based skills training devices can simulate and
advanced vision-tracking, the ProMIS surgical simulator takes an
measure performance of complex surgical tasks. The aim of
image of the dissected mesh and automatically generates a
this study is to determine basic face and construct validity
metric, indicating the accuracy of the shape and area dissected.
characteristics of two new devices configured for
Method laparoscopic suturing and knot-tying tasks.
1. Once the user has completed the Precision-Cutting Task, Methods
ProMIS takes an image of the dissected mesh. The image is
At the 2005 SAGES meeting, Learning Center attendees
converted to binary image and then scanned using a blob
evaluated two computer-based skills training platforms:
detection algorithm which produces a list of blobs.
SurgicalSIM (SS), a virtual reality (VR) device (METI,
2. The blob with the largest area is taken as the cut out area and Sarasota, FL; SimSurgery, Oslo, Norway) and ProMIS (PM), a
the number of pixels are counted inside this area. computer-enhanced video trainer (Haptica, Ltd., Dublin,
Ireland). Demographic and training data were collected
3. The actual measurement for area is given in cm2. This is
from 73 subjects. All were asked to perform 2 iterations of
calculated by counting the number of pixels in a known area of
laparoscopic suturing and intracorporeal knot-tying (10-
the image and then using ratios to determine the area of the cut
minute time limit) on each device. A 6-question survey was
out. (Note: this calibration step is achieved by taking the tissue
used to define impressions of task realism, relevance, and
off the tray and running the blob detection and pixel count on
execution using a 5-point Likert scale. Performance data
the uncovered black foam of which the exact area is known)
(SS: time, path length, errors; PM: time, path length,
Results smoothness) were collected on both devices and
comparisons made between user-defined expert and
Initial trials of the metrics on the FLS Precision- Cutting Task
nonexpert (intermediate and novice) groups (ANOVA and
show that ProMIS is as accurate — and frequently more accurate
Mann Whitney U test).
— than the current human observation method.
Results
46 subjects used SS and 56 used PM. Task completion rate
was 80% for SS and 93% for PM. Experts performed better
than nonexperts for all performance measures on SS
(composite score 496±41 vs 699±60, p < 0.005) and PM
(974±111 vs 1466±89, p < 0.005). Post-task survey scores for
PM were significantly higher for perceived realism (overall
realism and thread behavior), reflection of clinical ability,
and overall educational value. Perception of educational
value was not significantly different between the devices
among subjects with prior VR experience.
Conclusions
Using subject-defined expert and nonexpert groups,
construct validity was demonstrated for all performance
measures on both training devices. Surveyed face validity
measures favored the non-VR device, but results also
suggest that subjects with prior VR training experience are
more apt to accept a new VR surgical training platform.
www.haptica.com | email: promis@haptica.com | tel (NAm) +617 342 7270 | tel (RoW) +353 (0) 1 676 7310

5. Validation
7. World J Surg. 2007 Apr;31(4):764-72. 8. SAGES 2007 Education/Outcomes – P270
AUGMENTED VERSUS VIRTUAL REALITY LAPAROSCOPIC COMPARISON AND VALIDATION OF TWO DIFFERENT
SIMULATION: WHAT IS THE DIFFERENCE? : A COMPARISON SURGICAL SKILLS SIMULATORS
OF THE PROMIS AUGMENTED REALITY LAPAROSCOPIC Dieter Hahnloser MD,Rachel Rosenthal MD,Christian
SIMULATOR VERSUS LAPSIM VIRTUAL REALITY Hammel,Daniel Oertli,Markus Müller,Pierre-Alain Clavien,
LAPAROSCOPIC SIMULATOR. Department of Visceral and Transplantation Surgery,
Botden SM, Buzink SN, Schijven MP, Jakimowicz JJ. Catharina University Hospital Zurich, Switzerland
Hospital, Eindhoven, The Netherlands.
Background
Background
Simulators are increasingly incorporated in surgical training
Virtual reality (VR) is an emerging new modality for laparoscopic and validation is important. The simulations need to
skills training; however, most simulators lack realistic haptic resemble the task they are based upon (face validity) and
feedback. Augmented reality (AR) is a new laparoscopic the simulator should be able to differentiate between levels
simulation system offering a combination of physical objects and of experience (construct validity).
VR simulation. Laparoscopic instruments are used within an
Aim
hybrid mannequin on tissue or objects while using video tracking.
This study was designed to assess the difference in realism, To assess two different types of computer-based simulators:
haptic feedback, and didactic value between AR and VR the fully computerised virtual reality (VR) simulator Xitact
laparoscopic simulation. LS500 (VR-simulator) and the hybrid ProMisTM simulator.
Methods: 146 participants (61%) of the 22nd Davos
Methods
International Gastrointestinal Surgery Workshop performed
The ProMIS AR and LapSim VR simulators were used in this study. on a voluntary basis three similar exercises (camera
The participants performed a basic skills task and a suturing task navigation, clip and cut, and dissection) on the two
on both simulators, after which they filled out a questionnaire different simulators. Objective performance parameters
about their demographics and their opinion of both simulators recorded by either simulator and subjective evaluation by
scored on a 5-point Likert scale. The participants were allotted questionnaire were compared between beginner (n=73) and
to 3 groups depending on their experience: experts, advanced participants (n=73).
intermediates and novices. Significant differences were
Results
calculated with the paired t-test.
The camera navigation exercise was completed by 52% of
Results
the participants on the VR- and by 47% on the hybrid
There was general consensus in all groups that the ProMIS AR simulator with no difference in performance parameters
laparoscopic simulator is more realistic than the LapSim VR between beginners and advanced trainees. The hybrid
laparoscopic simulator in both the basic skills task (mean 4.22 simulator was graded more realistic (70% vs. 20%, p=.001)
resp. 2.18, P < 0.000) as well as the suturing task (mean 4.15 and more useful (65% vs. 36%, p=.043) than the VR-
resp. 1.85, P < 0.000). The ProMIS is regarded as having better simulator. Participation was higher at the clip and cut
haptic feedback (mean 3.92 resp. 1.92, P < 0.000) and as being exercise (75% VR- and. 52% hybrid simulator) and advanced
more useful for training surgical residents (mean 4.51 resp. 2.94, trainees performed significantly better (shorter tool-tip-
P < 0.000). travel distance, smoother, quicker and with higher score) on
both simulators compared to beginners. The clip and cut
Conclusions
exercise was graded more realistic on the hybrid (81% vs.
In comparison with the VR simulator, the AR laparoscopic 44%, p=.007) and similar useful on both simulators (77% vs.
simulator was regarded by all participants as a better simulator 72%). The dissection exercise was completed more often on
for laparoscopic skills training on all tested features. the hybrid simulator (47% vs. 23%, p=0.002). Only the hybrid
simulator was able to distinguish between advanced
trainees and beginners, with significantly higher scores for
all performance parameters for the latter. The hybrid
simulator was graded more realistic (70% vs. 33%, p=.016)
and more useful (83% vs. 62%, p=.12). Overall, acceptance
of requirement to train on and to be evaluated by such
simulators is still low (53% and 50%, respectively).
Conclusion
Fully computerized VR- or hybrid simulator performance
parameters can distinguish between beginner and advanced
trainees for perceptual motor skills (proving construct
www.haptica.com | email: promis@haptica.com | tel (NAm) +617 342 7270 | tel (RoW) +353 (0) 1 676 7310

6. Validation
validity), but not for visuo-spatial exercises such as the camera __________________________________________
navigation.
10.
__________________________________________ 11. SAGES 2006 Scientific Sessions S096
A COMPUTERIZED ANALYSIS OF ROBOTIC VERSUS
LAPAROSCOPIC TASK PERFORMANCE
9. SAGES 2007: S077
V K Narula MD, W C Watson MD, S S Davis MD, K Hinshaw BS,
COMPUTER-BASED LAPAROSCOPIC AND ROBOTIC SURGICAL
B J Needleman MD, D J Mikami MD, J W Hazey MD, J H
SIMULATORS: PERFORMANCE CHARACTERISTICS AND
Winston MD, P Muscarella MD, M Rubin, V Patel MD, W S
PERCEPTIONS OF NEW USERS
Melvin MD, The Ohio State University. CMIS. Columbus, OH
David W Lin MD, John R Romanelli MD, Renee E Thompson
MD,Michael E Ganey MD, Ron W Bush BS, Neal E Seymour MD, Introduction
Baystate Medical Center, Department of Surgery Robotic technology has been postulated to improve
The expanding inventory of advanced surgical training devices performance in advanced surgical skills. We utilized a novel
now includes simulators for laparoscopic and robotic surgery. In computerized assessment system to objectively describe the
order to define perceptions of the need and value of such technical enhancement in task performance comparing
devices, we evaluated the initial experience of surgeons using robotic and laparoscopic instrumentation.
both in the course of performance of an advanced laparoscopic
Methods and Procedures
skill.
Advanced laparoscopic surgeons (2- 10 yrs experience)
Methods performed 3 unique task modules using laparoscopic and
At the 2006 SAGES meeting, 62 Learning Center attendees evaluated a Telerobotic Surgical Instrumentation (Intuitive Surgical,
new virtual reality (VR) robotic surgery simulator (RS) [SimSurgery, Oslo, Sunnyvale, CA). Performance was evaluated using a
Norway] as well as either a computer-enhanced laparoscopic [ProMIS computerized assessment system (ProMIS, Dublin, Ireland)
(PM), Haptica, Ltd, Dublin, Ireland] or a VR simulator [SurgicalSIM (SS), and results were recorded as time (sec), total path (mm)
SimSurgery and METI, Inc, Sarasota, FL]. Demographic and training data and precision. Each surgeon had an initial training session
were collected and all were assessed during one iteration of followed by two testing sessions for each module. A Paired
laparoscopic suturing and knot-tying on RS and either PM or SS. An 8- Student’s T-Test was used to analyze the data.
question survey was used to determine users? impressions of task
Results
realism, interface quality, and educational value (5-point Likert scale).
Performance data [time, path length, smoothness (PM), errors (SS/RS)] 10 surgeons completed the study. Objective assessment of
were collected and comparisons made between user-defined groups and the data is presented in the table below. 8/10 surgeons had
different simulation platforms (Mann-Whitney Test, ANOVA). significant technical enhancement utilizing robotic
technology Laparoscopic vs Robotic Time (sec) Total
Results Path(mm) Precision Module 1210 vs 161 # 11649 vs 5571 *
Task completion rate was greater for experts than nonexperts on all 1434 vs 933 * Module 2119 vs 68 * 5573 vs 1949 * 853 vs 406 *
platforms (PM 100% vs 75%; SS 100% vs 36%; RS 93% vs 36%). Experts Module 377 vs 55 * 4488 vs 2390 * 552 vs 358 * # = p < 0.009 *
performed better than nonexperts on all performance measures on PM = p<0.001 Conclusions: The ProMIS computerized assessment
(p<0.05: time 154±16 vs 205±12; path length 820±97 vs 1287±97; system can be modified to objectively obtain task
smoothness 952±111 vs 1582±127). There were no significant differences performance data with robotic instrumentation. All the
between experts and nonexperts for SS and RS performance measures. tasks were performed faster and with more precision using
Perception of value of haptic features was less for subjects with prior robot the robotic technology than standard laparoscopy.
experience (n=10; p<0.05). Otherwise realism, interface quality, and
educational value scores did not differ on the basis of prior simulator or
robot use. Nonexperts found that robotic simulation better reflected
clinical skill than did experts. Overall, subjective quality was scored higher
for PM than for SS or RS.
Conclusions
The ability of performance metrics of the computer-enhanced simulator
to discern predicted performance differences between experts and
nonexperts was better than for VR devices with a single task iteration.
Initial use of VR devices was associated with a lower overall perception
of realism and educational value as compared to use of physical objects
in the non-VR simulator. This may reflect the need for familiarization
with the computer-generated environment before the educational
potential of VR can be realized.
www.haptica.com | email: promis@haptica.com | tel (NAm) +617 342 7270 | tel (RoW) +353 (0) 1 676 7310

7. Validation
12. SAGES 2006 Education/Outcomes–P219 13. SAGES 2007: S080
INTEGRATING SIMULATION LAB TRAINING INTO A SURGICAL THE IMPACT OF KNOWLEDGE OF RESULTS IN SURGICAL
RESIDENCY PROGRAM: IS VOLUNTARY PARTICIPATION SKILLS TRAINING
EFFECTIVE A. O’Connor MD, C. Cao PhD, S. Schwaitzberg MD,
Lily Chang MD, James Petros MD, Donald Hess MD, Caroline Department of Mechanical Engineering, Tufts University.
Rotondi BA, Timothy Babineau MD, Boston Medical Center
Background
Objective
Concerns about the adequacy of advanced laparoscopic
Surgical training programs nationwide are struggling with the training continue to be raised despite a proliferation of
integration of simulation training into their curriculum given the training systems exist. The manner in which the training
constraints of the 80-hour work week. We examine the modules are structured to maximize learning has not been
effectiveness of voluntary training in a simulation lab as part of examined. There are many aspects to the accumulation of
the surgical curriculum. Methods: The ProMIS simulator was laparoscopic skills during training, one of which is
introduced into the general surgery residency at Boston Knowledge of Results (KR), i.e. the information provided to
University Medical Center. All categorical residents (28) were individuals about the outcomes of their motor responses in
required to attend a 2-hour training session and curriculum their environment. We studied the effects of KR on the
review. Non-categorical residents (23) were given the option to learning curve of laparoscopic suturing and knot tying.
complete training. After the introductory session, time spent in
Aims
the lab was encouraged, but voluntary. Use of the simulator was
tracked for all residents. Participation in the simulation We evaluated the learning curves of 9 medical students with
curriculum was defined as 3 or more uses of the simulator. After no previous laparoscopic surgical experience under three
3 months, all residents completed a survey regarding the different conditions, each with different levels of
simulation lab and their simulator usage. Results: 26 (93%) knowledge of results.
categorical residents and 3 (6%) non-categorical residents
completed the introductory simulator training session. Over a 3 Methods
month period, use of the simulator at least once was 31% among Subjects were randomly assigned to one of three groups.
all eligible residents; 80% of PGY1, 40% of PGY2, 60% of PGY3, Each subject attended a training session for 1 hour each
and 0% of PGY4 and PGY5. Four residents (14%) participated in day, 6 days a week for 4 weeks. Group 1 (No feedback)
the simulation curriculum. 71% of simulator usage was during received no knowledge of results (KR) and no performance
working hours while 29% was completed post-call or off duty. feedback. Group 2 (feedback only) received factual KR
Most residents agreed that the simulator was easy to use and following each training session, but no coaching. Group 3
improved their operative skills, but did not think it was a good (feedback and coaching) received KR and coaching. Learning
substitute for actual operative experience. Reported reasons for curves were plotted based on task time, smoothness of
not using the simulator included off-site rotation (44%), no time instruments and instrument’s path length. The task used
(30%), and no interest (11%). was an intracorporeal suture/knot tying in the ProMIS
laparoscopic simulator. Perceived workload for each session
Conclusions
was recorded using a standardized NASA TLX workload
Voluntary use of a surgical simulation lab leads to minimal score.
participation in a training curriculum. Participation should be
mandatory if it is to be an effective part of a residency Results
curriculum. The variability across each session for each student was
calculated for each of the three parameters. There was
statistical significance between the groups for all
parameters (p-values 0.0002, 0.0002 and 0.009). Significant
differences were found between groups 2 and 3 and group 1
(p values 0.0314-0.0410) Groups 2 and 3 learned
significantly faster than those in Group 1, reaching
performance plateaus at earlier sessions. There were no
significant differences between groups 2 and 3 (p-values
0.1211, 0.1758 and 0.1375). Providing individuals with
knowledge of results lowered their perceived workload,
adding instructional feedback lowered this even further.
These results demonstrate that KR is essential for efficient
surgical skill acquisition. Individual coaching, a labor
intensive proposition, reduces workload but has NO added
beneficial effect on the speed of learning. These results
www.haptica.com | email: promis@haptica.com | tel (NAm) +617 342 7270 | tel (RoW) +353 (0) 1 676 7310

8. Validation
provide a useful basis for developing efficient and cost effective 15. SAGES 2007 Education/Outcomes – P284
surgical skills training curriculum. ABSTRACT VIRTUAL REALITY TRAINING DEVELOPS CORE
____________________________________________________
LAPAROSCOPIC SKILLS COMPARABLE TO EXPERIENCED
LAPAROSCOPIC SURGEONS: RESULTS OF A PROSPECTIVE
14. SAGES 2007: Education/Outcomes – P267 RANDOMIZED TRIAL COMPARING TWO VIRTUAL REALITY
THE CONSTRUCT VALIDITY OF COMPUTER-DERIVED TRAINERS
PERFORMANCE METRICS FOR SELECTED SIMULATED
E. Matt Ritter MD, Elisabeth A Pimentel BA, Ryan E Earnest
LAPAROSCOPIC TASKS
BS, Randy S Haluck MD, Mark W Bowyer MD, National Capital
J. A Oostema MD, Matthew Abdel BS, Jon C Gould MD, University Area Medical Simulation Center, Uniformed Services
of Wisconsin School of Medicine and Public Health, Department University, Bethesda, Maryland / Department of Surgery,
of Surgery Pennsylvania State College of Medicine, Hershey,
Pennsylvania
Introduction
A surgical skills assessment tool is said to demonstrate evidence Introduction
of construct validity if users with more experience, and by While simulation is becoming more widely accepted in
inference more skill, perform better or more efficiently. surgical training, comparative trials on the training
Computer derived motion metrics such as smoothness (the effectiveness of these simulators are lacking. We sought to
number of times an instrument tip changes velocity during a compare the effectiveness of two abstract virtual reality
task) and path length may be more sensitive measures of skill for trainers to train laparoscopic skills as assessed by the
a particular task than traditional metrics such as time. Fundamentals of Laparoscopic Surgery (FLS). We then
compared the post training performance of the novice
Methods
subjects with a group of experienced surgeons.
Twenty-four medical students (third year), 19 surgical residents
(PG1-5), and 3 attending surgeons were asked to perform four Methods and Results
different tasks 3 times in a hybrid computer-based physical 20 novice medical students were recruited. Each subject
laparoscopic trainer (ProMIS, Haptica Inc., Dublin). The 4 tasks in performed a pre-test consisting of 3 FLS tasks - Peg Transfer
order of complexity were laparoscopic orientation (Task 1), (PT), Pattern Cut (PC) and Intracorporeal Suture (IS) -
object positioning (Task 2), sharp dissection (Task 3), and intra- placed in the ProMIS augmented reality simulator (Haptica,
corporeal knot tying (Task 4). Metrics recorded were time, path Ireland). They were then randomized to train to
length, and smoothness. Laparoscopic operative experience for predetermined levels of proficiency on 3 tasks of the
each user was quantified using case logs. Correlations were Minimally Invasive Surgical Trainer-Virtual Reality (MIST-VR)
determined using regression analysis and ANOVA. (Mentice, Sweden) or the Rapid Fire/Smart Tutor (RFST)
(Verefi, Elizabethtown, PA). After reaching the proficiency
Results
levels, both groups then took a post test consisting of 3
A statistically significant correlation was observed between trials of the same tasks used for the pre-test. Post test
experience and performance for all three metrics for tasks 2-4 performance by both groups was then compared to a control
(all p< 0.01). Smoothness was the only metric to correlate in the group, composed of 10 experienced surgeons who had
laparoscopic orientation task. Within tasks, time and smoothness completed the same post test.
correlate much more strongly with experience and to a similar
MIST-VR and RFST groups demonstrated statistically
degree. The strongest correlation was observed for the knot
significant improvement from the pre-test to the post test
tying task (r2=0.60 for time and 0.59 smoothness). An r2=1.0
on all 3 FLS tasks (p < 0.0001). There was no significant
would represent a perfect correlation between experience and
difference in post test performance between the MIST-VR
the specified metric.
and RFST groups. When the simulation trained groups were
Conclusions compared to experienced controls there was no significant
difference in performance with respect to PT. The
The computer-derived metrics measured by the hybrid trainer
experienced controls did significantly outperformed the
correlate with laparoscopic experience. Further study is
MIST-VR group in PC (p<0.01) and IS (p<0.05), but
necessary to determine if specific metrics are better indicators
differences between the experienced controls and the RFST
of actual skill.
group did not reach statistical significance.
Conclusion
Simulation based training on either the MISTVR or the RFST
simulator improves the skill level of novices as assessed by
FLS. The post training skill level of these novices compares
favorably with a group of experienced surgeons. Virtual
Reality trainers, such as RFST and MISTVR, train
www.haptica.com | email: promis@haptica.com | tel (NAm) +617 342 7270 | tel (RoW) +353 (0) 1 676 7310

9. Validation
fundamental laparoscopic skills equally and to a level 17. SAGES 2006 Emerging Technologies P036
comparable to a group of experienced practicing surgeons. AUGMENTED REALITY SIMULATOR FOR HAND-ASSISTED
____________________________________________________
LAPAROSCOPIC COLECTOMY
Derek Young, Derek Cassidy, Fiona Slevin, Donncha Ryan,
Haptica Ltd, Dublin, Ireland.
16. SAGES07 Ergonomics/Instrumentation P307 Training in Hand-Assisted Laparoscopic Colectomy (HALC)
A COMPUTERIZED ANALYSIS OF STANDARD VERSUS HIGH
has largely been done using cadavers and porcine models.
DEXTERITY LAPAROSCOPIC INSTRUMENTATION IN TASK
These have drawbacks in terms of realism, logistics and lack
PERFORMANCE
of performance measurement. A Simulator would provide
V K Narula MD,K M Reavis MD,D R Renton MD,D J Mikami MD,B J consistent instruction and practice and provide feedback on
Needleman MD,J W Hazey MD,K E Hinshaw BS,W S Melvin MD, performance. However, given the range of instruments used
THE OHIO STATE UNIVERSITY HOSPITAL, CENTER FOR MINIMALLY in the procedure, and especially the use of a hand, pure
INVASIVE SURGERY virtual reality could not be considered as a solution.
Introduction The ProMIS Augmented Reality simulator platform – by
combining physical and virtual reality - enables interaction
Minimally invasive surgery is becoming the standard of care for
and tracking of real instruments with a physical model. And
the majority of abdominal procedures. Laparoscopic
because of its technological approach (vision-tracking) also
instrumentation is constantly undergoing improvements to give
enables the hand to be tracked. In the new ProMIS HALC
surgeons an advantage. Articulated instrumentation provides a
simulator, 3D models or graphical objects are overlaid on
distinct advantage in the field of robotic surgery. Applying the
the physical model to provide instruction and guidance. For
same principles to standard laparoscopic instrumentation could
example, a 3D animation may be used to demonstrate how
offer increased degrees of freedom to make complex
to complete a step; a graphical guideline ‘A – B’ may be
laparoscopic tasks easier to perform. We utilized a novel
used to indicate a target area for dissection. ProMIS HALC
computerized assessment system to objectively evaluate task
measures surgical skill by gathering data on the movement
performance comparing Standard and High Dexterity (HD)
of commercial laparoscopic instruments while completing a
laparoscopic instrumentation.
standardized task. The main performance metrics are time
Methods taken, total path length and economy of movement.
Additionally metrics specific to a step are calculated to
Advanced laparoscopic surgeons (2-12yrs experience) performed
measure performance associated with a specific instruction
3 unique task modules utilizing Standard and HD laparoscopic
in a specific region of the physical model. Following the
instrumentation (Novare Surgical Systems, Cupertino, CA).
simulated procedure, the user completes a self-assessment
Performance was evaluated using a computerized assessment
which contributes to the metrics for the full procedure. A
system (ProMIS, Dublin, Ireland) and results were recorded as
full analysis is of performance is presented to the user on
time (sec), path (mm), and precision. Each surgeon had an initial
completion of the procedure and self-assessment.
training session followed by two testing sessions for each
module. A Paired Student’s T-Test was used to analyze the data. Results
Results: Nine surgeons completed the study. Objective
Initial trials of the ProMIS HALC simulator indicate that
assessment of the data is presented in the table below. Module 1
practice on the simulator improves performance as
was statistically significant, whereas Module 2 and 3 showed no
measured by the metrics gathered by the simulator. While
difference in task performance with the HD instrumentation.
detailed validation studies remain to be done, initial
Conclusion indicators are that the HALC simulator represents an
“unparalleled opportunity to practice, step by step, a Hand-
HD instrumentation is in its infancy. Results showed no
assisted laparoscopic sigmoid resection” and “a huge step
advantage using HD instrumentation. This could be due to the
forward in surgical training”.
learning curve associated with new instrumentation and
technology. With future developments in HD technology and
training, the user interface will improve and may offer an
advantage over standard laparoscopic instrumentation.
www.haptica.com | email: promis@haptica.com | tel (NAm) +617 342 7270 | tel (RoW) +353 (0) 1 676 7310

10. Validation
18. May 2006 North of England Surgical Society Annual 19. EAES 2006, Poster
Registrar’s Meeting Winner of the George Feggeter Gold Medal DEVELOPING PSYCHOMETRIC ASSESSMENT OF
LAPAROSCOPIC SKILLS ACQUISITION: IS PSYCHOMETRIC LAPAROSCOPIC SKILLS USING THE PROMIS SIMULATOR
MOTION ANALYSIS A VALID ASSESSMENT TOOL? Pellen MGC1;2, Barton JR2, Horgan LF1, Attwood SE1
1;2 2 1 1
Pellen MGC , Barton JR , Horgan LF , Attwood SE Northumbria Northumbria Upper Gastrointestinal Team of Surgeons1;
Upper Gastrointestinal Team of Surgeons1; University of University of Newcastle upon Tyne2, United Kingdom
Newcastle upon Tyne2, Newcastle upon Tyne, United Kingdom
Aims
Aims
Reliable and validated methods of objective skills training
In an evolving climate of competency-based assessment, reliable and assessment are required for trainee surgeons. The
and validated methods of objective skills assessment are ProMIS Simulator (Haptica, Ireland) potentially offers a
required for trainee surgeons. We aimed to assess whether the method of assessing laparoscopic psychomotor performance.
ProMIS Simulator (Haptica, Dublin, Ireland) offers a method of We present initial data from our Centre and Royal College of
assessing laparoscopic psychomotor performance. Surgeons Basic Surgical Skills (BSS) Courses.
Methods Methods and results
Volunteers comprising 17 experienced laparoscopists ( >100 Volunteers comprising 17 experienced laparoscopists (>100
laparoscopic cholecystectomies) and 38 medical students novices laparoscopic cholecystectomies) and 38 medical students
(no laparoscopic experience) performed 3 simulated tasks novices (no laparoscopic experience) were assessed on a
comprising virtual reality camera navigation, object transfer and complex sharp dissection task (glove over balloon). A
sharp dissection task (glove over balloon). A further group of 28 further group of 28 basic surgical trainees (experience
basic surgical trainees (experience limited to 1st assistant) limited to 1st assistant) attending BSS Courses were assessed
attending BSS Courses were assessed on the same tasks before on the same task before and after training in laparoscopic
and after training in laparoscopic skills. Data metrics of time, skills. Data metrics of time, smoothness and path length
smoothness and path length were measured via optical tracking were measured via optical tracking of instrum-ent
of instrument movement. Objective observations of specific movement as well as observations of specific errors.
errors were also recorded.
Data analysis (ANOVA) demonstrated experienced
Results laparoscopists performed target dissection at least 50%
faster, smoother and with more economy of instrument
Non-parametric analysis demonstrated experienced
movement than students (p<0.05). Experienced participants
laparoscopists performed all 3 tasks significantly faster,
performed sharp dissection more accurately (p<0.01)
smoother and with more economy of movement (p<0.05),
although no difference in balloon puncture frequency was
excluding camera navigation path length. Experienced
seen. Similarly significantly better performance over
participants performed sharp dissection more accurately (p<0.01)
trainees was demonstrated. Trainees showed only
although no difference in balloon puncture was seen. Repeat
significantly smoother instrument handling when compared
assessment of BSS Course Trainees showed significant
to students, possibly reflecting greater baseline dexterity in
improvements in simulator metrics (Paired T test, P<0.05),
this selective group. Repeat assessment following course
although smaller yet significant improvements in “untrained”
training showed significant improvements in all metrics by
student performance was also seen.
32-40% (Paired T test, P<0.05). Whilst significant
Conclusions improvements were also demonstrated in repeat assessment
of the untrained student group, these were less marked (15-
Gross analysis of these metrics can distinguish between 18%).
experience levels supporting construct validity of these simulator
tasks. These results suggest potential for objectively measuring Conclusions
baseline skill level and response to training. Further work will
The gross analysis of these metrics can distinguish between
examine the effect of interface familiarisation and defining
experience levels supporting the construct validity of this
target levels of performance in simulated tasks.
simulator task. These results suggest a potential role for
objectively measuring baseline skill level and response to
training in distinct psychomotor challenges. Further work in
progress is examining the effect of interface familiarisation
and repeated task performance on novice learning curves
and defining target levels of performance in a range of
simulated tasks.
www.haptica.com | email: promis@haptica.com | tel (NAm) +617 342 7270 | tel (RoW) +353 (0) 1 676 7310

11. Validation
20. Surg Endosc (May 2006) 20: 900–904 21. Surgical endoscopy ISSN: 0930-2794 (Paper) 1432-
CONSTRUCT VALIDATION OF A NOVEL HYBRID SURGICAL 2218 (Online)
SIMULATOR CONSTRUCT VALIDATION OF THE PROMIS SIMULATOR
D. Broe, P. F. Ridgway, S. Johnson, S. Tierney, K. C. Conlon USING A NOVEL LAPAROSCOPIC SUTURING TASK
Department of Surgery, Professorial Surgical Unit, Level 4, The K. R. Van Sickle1, D. A. McClusky III1, A. G. Gallagher and
Adelaide and Meath Hospital, incorporating the National C. D. Smith1
Children’s Hospital, Tallaght, Dublin 24, Ireland
Background
Background
The use of simulation for minimally invasive surgery (MIS)
Simulated minimal access surgery has improved recently as both skills training has many advantages over current traditional
a learning and assessment tool. The construct validation of a methods. One advantage of simulation is that it enables an
novel simulator, ProMis, is described for use by residents in objective assessment of technical performance. The
training. purpose of this study was to determine whether the ProMIS
augmented reality simulator could objectively distinguish
Methods
between levels of performance skills on a complex
ProMis is a surgical simulator that can design tasks in both virtual laparoscopic suturing task.
and actual reality. A pilot group of surgical residents ranging
Methodology
from novice to expert completed three standardized tasks:
orientation, dissection, and basic suturing. The tasks were tested Ten subjects — five laparoscopic experts and five
for construct validity. Two experienced surgeons examined the laparoscopic novices — were assessed for baseline
recorded tasks in a blinded fashion using an objective structured perceptual, visio-spatial, and psychomotor abilities using
assessment of technical skills format (OSATS: task-specific validated tests. After three trials of a novel laparoscopic
checklist and global rating score) as well as metrics delivered by suturing task were performed on the simulator, measures
the simulator. for time, smoothness of movement, and path distance were
analyzed for each trial. Accuracy and errors were evaluated
Results
separately by two blinded reviewers to an interrater
The findings showed excellent interrater reliability (Cronbach_s reliability of >0.8. Comparisons of mean performance
a of 0.88 for the checklist and 0.93 for the global rating). The measures were made between the two groups using a Mann-
median scores in the experience groups were statistically Whitney U test. Internal consistency of ProMIS measures was
different in both the global rating and the task-specific assessed with coefficient α.
checklists (p < 0.05). The scores for the orientation task alone
Results
did not reach significance (p = 0.1), suggesting that modification
is required before ProMis could be used in isolation as an The psychomotor performance of the experts was superior
assessment tool. at baseline assessment (p < 0.001). On the laparoscopic
suturing task, the experts performed significantly better
Conclusions
than the novices across all three trials (p < 0.001). They
The three simulated tasks in combination are construct valid for performed the tasks between three and four times faster (p
differentiating experience levels among surgeons in training. < 0.0001), had three times shorter instrument path length
This hybrid simulator has potential added benefits of marrying (p < 0.0001), and had four times greater smoothness of
the virtual with actual, and of combining simple box traits and instrument movement (p < 0.009). Experts also showed
advanced virtual reality simulation. greater consistency in their performance, as demonstrated
by SDs across all measures, which were four times smaller
than the novice group. Observed internal consistency of
ProMIS measures was high (α = 0.95, p < 0.00001).
Conclusions
Preliminary results of construct validation efforts of the
ProMIS simulator show that it can distinguish between
experts and novices and has promising psychometric
properties. The attractive feature of ProMIS is that a wide
variety of MIS tasks can be used to train and assess technical
skills.
www.haptica.com | email: promis@haptica.com | tel (NAm) +617 342 7270 | tel (RoW) +353 (0) 1 676 7310

12. Validation
22. EAES 2004 Abstract nr.: O207 23. EAES 2004 abstract nr.: O208
RELATIONSHIP BETWEEN MOTION ANALYSIS, TIME, PSYCHOMOTOR SKILLS ASSESSMENT IN PRACTICING
ACCURACY, AND ERRORS DURING PERFORMANCE OF A SURGEONS PERFORMING ADVANCED LAPAROSCOPIC
LAPAROSCOPIC SUTURING TASK ON AN AUGMENTED REALITY PROCEDURES II: DEMOGRAPHICS AND PERFORMANCE
SIMULATOR PROFILES
Author: D.A.M. McClusky, Emory University School of Medicine, Author: A.G. Gallagher, Emory University, Atlanta, United
Atlanta Georgia, United States of America. Co-author(s): K. Van States of America. Co-author(s): C.D. Smith, Emory
Sickle, Emory University School of Medicine, Atlanta Georgia, University, Atlanta, United States of AmericaR.M. Satava,
United States of America University of Washington, Seattle, United States of America
A.G. Gallagher, Emory University School of Medicine, Atlanta
Background
Georgia, United States of America
This study reports on the objectively assessed psychomotor
Background
performance of minimally invasive surgeons on a box-trainer
Time, efficient movement, accuracy, and safety are reliable and and a virtual reality (VR) task as a function of handedness,
discriminative metrics of proficiency during virtual reality and gender, sight corrected status, and age. Methods: Two
box-trainer based minimally invasive surgical (MIS) training. The hundred and ten surgeons attending the 2001 annual
role these metrics may serve during more advanced skills training meeting of the American College of Surgeons (ACS) in New
are not well understood. Using a novel augmented reality Orleans who reported having completed more than 50
simulator, we sought to gain an understanding of the relationship laparoscopic procedures participated. Subjects completed a
between these metrics during an advanced MIS suturing task. box-trainer laparoscopic cutting task and a similar virtual
Methods: Eleven subjects completed 3 trials of a suturing task reality task twice. Demographic and laparoscopic
designed for a box-trainer and adapted for the ProMIS (Haptica, experience data was also collected. Results: There were no
Dublin, Ireland) simulator. Time, tool path, and smoothness of significant differences between subjects performance on
movement were assessed using computer algorithms. Measures of either tasks in terms of handedness, gender or whether they
accuracy during suture placement and errors in performance were sight corrected or not. A clear and consistent linear
were assessed by two blinded reviewers trained to assess trend emerged in terms of age. Older subjects (ages 60 – 69)
performance with inter-rater reliability > 0.8. A Pearson’s performed significantly worse than younger subjects (ages
correlation coefficient was used to assess the strength of the 30 – 39, 40 – 49) on the box-trainer task for correct incisions
relationship between ProMIS metrics and suturing task (13.1 Vs 19.3, p < 0.008) and incorrect incisions (12.3 Vs 2.5,
performance. Results: Of the ProMIS metrics, time correlated p > 0.05). They also performed worse on the VR task for
with tool path distance and smoothness of movement in three time (132 Vs 71, p < 0.05), error (99 Vs 41, p < 0.05) and
trials (range 0.914 – 0.957, p < 0.0001). When the suturing task economy of movement (22.8 Vs 11.7, p < 0.05). Conclusions:
was analyzed, accuracy and error score demonstrated an equally Increasing age was found to be associated with a decline in
strong relationship (range -0.726 - -0.84, all p < 0.0001). objectively assessed psychomotor performance on two well
Combining all metrics, path distance correlated strongest with validated laparoscopic tasks.
accuracy (2 trials significant, range -0.67 - -0.93), and error
Acknowledgements
score (3 trials significant, range 0.54 – 0.61). Smoothness of
movement significantly correlated with accuracy in 2 trials This study was supported with grants from the ACS, SAGES,
(range -0.63 - -0.88), and time correlated with error score in 2 SLS, TATRC, and Emory University Endosurgery Unit.
trials (range 0.56 – 0.60).
Conclusion
Metrics based on movement efficiency and time, and those based
on task accuracy and error scores strongly correlate when
grouped independently. At this time, a proficiency curriculum
should incorporate both forms of analysis, however further
validation work is needed to replicate these findings and give
further insight into how ProMIS metrics relate to real-world
performance.
www.haptica.com | email: promis@haptica.com | tel (NAm) +617 342 7270 | tel (RoW) +353 (0) 1 676 7310

13. Validation
24. JSLS, Journal of the Society of Laparoendoscopic 25. SAGES 2008 P174
Surgeons, Vol. 11, No. 3. (September 2007), pp. 273- FLS TEST IS TRANSFERABLE TO PROMIS SIMULATOR
302. Eric S Hungness MD, Albert Amini BA, Deb E Rooney MS, Eric
CRITERION-BASED TRAINING WITH SURGICAL SIMULATORS: T Volckman MD, Nathaniel J Soper MD, Feinberg School of
PROFICIENCY OF EXPERIENCED SURGEONS Medicine, Northwestern University, Chicago, IL
Heinrichs, Wm, Lukoff, Brian, Youngblood, Patricia, Dev, Parvati,
Introduction
Shavelson, Richard, Hasson, M Harrith, Satava, M Richard,
The McGill Inanimate System for Training and Evaluation
Mcdougall, M Elspeth, Wetter, Paul Alan
of Laparoscopic Skills (MISTELS) comprises five tasks with
Objective an objective scoring system, and has been incorporated
by SAGES in their Fundamentals of Laparoscopic Surgery
In our effort to establish criterion-based skills training for (FLS) program. MISTELS has high inter-rater and test-
surgeons, we assessed the performance of 17 experienced retest reliability and correlates with operative skill.
laparoscopic surgeons on basic technical surgical skills recorded However, the FLS program is labor intensive, requiring a
electronically in 26 modules selected in five commercially trained proctor. The ProMIS simulator allows for
available, computer-based simulators. assessment of physical tasks (instrument path length and
instrument smoothness) through instrument tracking
Methods/Procedures technology. We hypothesized that the FLS scores
obtained in the ProMIS simulator as well as ProMIS
Performance data were derived from selected surgeons randomly metrics would correlate with standard FLS scoring.
assigned to simulator stations, and practicing repetitively during
three one-half day sessions on five different simulators. We Methods
measured surgeon proficiency defined as efficient, error-free Twenty general surgery residents (13 junior and 7 senior)
performance and developed proficiency score formulas for each had baseline laparoscopic skills assessed using MISTELS in
module. Demographic and opinion data were also collected. the standard FLS and ProMIS simulators (pre-test). Nine
junior and 4 senior residents had a post-test after four
Results weeks of training. Tasks were scored by FLS and ProMIS
metrics. Total path length (TP) and total smoothness (TS)
Surgeons’ performance demonstrated a sharp learning curve with were calculated by adding the path lengths and
the most performance improvement seen in early practice smoothness of each individual task. ANOVA was used to
attempts. Median scores and performance levels at the 10th, compare the mean (SD) of total and individual task scores
25th, 75th, and 90th percentiles are provided for each module. for pre- and post-tests in the FLS and ProMIS simulators.
Construct validity was examined for two modules by comparing Student's t-test was used to compare ProMIS metrics.
experienced surgeons’ performance with that of a convenience Pearson's correlations were calculated for standard FLS
sample of lessexperienced surgeons. scores in relation to ProMIS FLS scores, TP and TS.
Significance was defined as p < 0.01.
Conclusions
Results
A simple mathematical method for scoring performance is All residents showed statistically significant improvement in
applicable to these simulators. Proficiency levels for training post-test total and individual task FLS scores on either the
courses can now be specified objectively by residency directors FLS or ProMIS simulator. 100% and 88% of residents achieved
and by professional organizations for different levels of training passing post-test scores on the FLS and ProMIS simulator,
or post-training assessment of technical performance. But data compared to 30% and 29.2% on the pre-test. There was no
users should be cautious due to the small sample size used in this difference in junior and senior resident posttest scores (87.6
study and the need for further study into the reliability and vs 79.1). ProMIS path length and smoothness were
validity of the use of surgical simulators as assessment tools. significantly reduced across all tasks (range 14- 68%). Total
ProMIS FLS scores (0.729), TP (-0.753) and TS (- 0.769)
Summary comment
significantly correlated with total standard FLS simulator
The simulators used included pure Virtual Reality Simulators and scores. All residents with TP < 4000mm or TS < 6000
two simulators with real haptics (including ProMIS): Lap Mentor achieved a passing total FLS score.
from Simbionix (pure Virtual Reality); LapSim from Surgical-
Conclusions
Science AB (pure Virtual Reality); SurgicalSIM from METI (pure
Virtual Reality); ProMIS from Haptica (real haptics with metrics); All surgical residents achieved a passing FLS score after a
LTS2000 ISM60 from RealSim (real haptics with metrics). 4-week laparoscopic skills curriculum. FLS tasks are
transferable to the ProMIS simulator with traditional FLS
ProMIS out-performed the virtual reality simulators with a mean scoring and intrinsic ProMIS metrics being good
effectiveness rating of 3.56 versus 3.22 and 3.11 for LapSim and measurement tools. A ProMIS total path length <4000 mm
SurgicalSim (LapMentor also scored 3.56). or total smoothness <6000 reliably predicts a passing FLS
score.
www.haptica.com | email: promis@haptica.com | tel (NAm) +617 342 7270 | tel (RoW) +353 (0) 1 676 7310