• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
ProMIS Validation

ProMIS Validation



A summary of abstracts and validation papers published on ProMIS - all demonstrating validity of ProMIS metrics as indicators of surgical proficiency.

A summary of abstracts and validation papers published on ProMIS - all demonstrating validity of ProMIS metrics as indicators of surgical proficiency.
Also on www.haptica.com



Total Views
Views on SlideShare
Embed Views



0 Embeds 0

No embeds



Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

    ProMIS Validation ProMIS Validation Document Transcript

    • Validated! Proven wiith FLS Proven w th FLS Better than VR Better than VR Preferred by users Preferred by users ProMIS and FLS • ProMIS metrics are “excellent predictors of scores in the standard FLS simulator” and “predict readiness for FLS Certification”1 2 3 • ProMIS metrics are valid on peg transfer, pre-tied loop placement and knot-tying tasks4 • “Initial trials of the metrics on the FLS Precision- Cutting Task show that ProMIS is as accurate as 5 the current method” ProMIS assesses performance on Augmented Reality combines VR real models where real haptics with a physical model allowing • FLS tasks are transferable to the ProMIS simulator are important, eg in Suturing and virtual bleeding and real haptics. Knot-tying with traditional FLS scoring and intrinsic ProMIS metrics being good measurement tools. A ProMIS total path length <4000 mm or total smoothness 25 <6000 reliably predicts a passing FLS score. ProMIS vs pure VR • ProMIS out-performed the virtual reality 24 simulators ProMIS has pure VR Modules, eg Vision technology enables • Scores for ProMIS were significantly higher than for Instrument Handling. Users tracking of errors and automatic still use real instruments. calculation of dissected tissue. for SurgicalSIM for overall realism, thread behavior, reflection of clinical ability, and overall educational value.6 • In comparison with LapSim, ProMIS was regarded by all participants as a better simulator for laparoscopic skills training on all tested features7 ProMIS surgical simulator • “Only [ProMIS] was able to distinguish between advanced trainees and beginners (and) was ProMIS Modules range from Basic Laparoscopic Skills to graded more realistic (70% vs 33%) and more MIS procedures like LapColectomy. For more information useful (83% vs 62%)” than Xitact8 on ProMIS, please contact us at: • The ability of performance metrics of [ProMIS] to Email: promis@haptica.com discern predicted performance differences U.S. tel: +1 617 342 7270 between experts and non-experts was better RoW tel: +353 (0)1 676 7310 than for SimSurgery’s robotic surgery simulator and 1 SAGES 2006 S064 Ritter et al, SurgicalSIM 9 2 SAGES 2006 P237 McCluney MD, et al, 3 4 SAGES 2007 P279 McCluney et al, ProMIS: the preferred Simulator SAGES 2006 S065 Vuong et al, 5 SAGES 2007 ETP057 Young et al, • ProMIS can be used effectively with the DaVinci 6 SAGES 2006 P224 Fellinger, et al, 7 robot to obtain performance data with robotic World J Surg. 2007 Apr;31(4):764-72. Botden et al, 8 SAGES 2007 P270 Hahnloser et al, instrumentation10 9 SAGES 2007 S077 Lin et al, 10 SAGES 2006 Narula et al, • Residents believe that ProMIS is easy to use and 11 24 SAGES 2006 P219 Chang et al, improved their operative skills11 Heinrichs, et al 2007 25 Hungness, et al 2008
    • Validation 1. SAGES 2006 Scientific Session S064 2. SAGES 2006 Education/Outcomes–P237 CONCURRENT VALIDITY OF AUGMENTED REALITY METRICS VALIDATION OF THE PROMIS HYBRID SIMULATOR USING APPLIED TO THE FUNDAMENTALS OF LAPAROSCOPIC A STANDARD SET OF LAPAROSCOPIC TASKS SURGERY (FLS) A L McCluney MD, L S Feldman MD, G M Fried, Steinberg- E. Matt Ritter MD, Tamara W Kindelan MD, Curtis Michael, Bernstein Centre for Minimally Invasive Surgery, McGill Elisabeth A Pimentel BA, Mark W Bowyer MD, 1NCA Medical University Health Centre, Montreal, QC, Canada Simulation Center, Department of Surgery, Uniformed Services Introduction University, 2Division of General Surgery, National Naval Medical Center, Bethesda Maryland SAGES Fundamentals of Laparoscopic Surgery (FLS) tasks are validated measures of technical skills, however FLS scoring Objective requires a trained proctor. The ProMIS simulator (Haptica; Current skills assessment in the Fundamentals of Laparoscopic Dublin, IR) is a ‘hybrid’ system with physical and virtual Surgery (FLS) program is labor intensive requiring one proctor for reality tasks. It has the flexibility to incorporate any every 1-2 subjects. The ProMIS Augmented Reality (AR) simulator physical task and score it with ProMIS metrics. Metrics are (Haptica, Dublin IRE) allows for objective assessment of physical automated and report motion analysis data as instrument tasks through instrument tracking technology. We hypothesized path length (PL) and instrument smoothness (IS). The that the ProMIS metrics could differentiate between ability purpose of this study was to test for construct and groups as well as standard FLS scoring with fewer personnel concurrent validity using FLS tasks in the ProMIS simulator. requirements. Methods Methods 5 laparoscopic novices and 5 experts performed FLS tasks in We recruited 60 volunteer subjects. Subjects were stratified both the standard FLS simulator box and the ProMIS based on their laparoscopic surgical experience. Those who had simulator. Assessments were made based on FLS metrics, as performed more than 100 laparoscopic procedures were well as PL and IS. Student’s t-test was used to compare the considered experienced (n=8). Those with less than 10 mean (SD) of total scores for novices and experts. Pearson’s laparoscopic procedure were considered novices (n=44). The rest correlations were calculated for standard FLS scores in were intermediates (n=8). All subjects performed up to 5 trials relation to ProMIS FLS scores, total PL, and total IS. of the peg transfer task from FLS in the ProMIS simulator. FLS Significance was defined as p < 0.01 (*). score, instrument path length, and instrument smoothness Results assessment were generated for each trial. Standard FLS scores correlated strongly with ProMIS FLS Results scores (r=0.90), total PL (r=-0.83), and total IS (r=-0.78) For each of the 5 trials, experienced surgeons outperformed (p&lt; 0.01). intermediates who in turn out performed novices. Statistically Conclusions significant differences were seen between the groups across all trials for FLS score (p < 0.001), ProMIS path length (p <0.001) and FLS tasks performed in ProMIS, when scored by either ProMIS smoothness (p < 0.001). When the FLS score was traditional FLS metrics or by intrinsic ProMIS metrics, compared to the path length and smoothness metrics, a strong discriminate effectively between novices and experts. Based relationship between the scores was apparent for novices (r = on the observed correlations, ProMIS FLS scores, total PL, 0.78, r = 0.94 , p < 0.001) respectively), intermediates (r = 0.5, p and total IS are excellent predictors of scores in the = 0.2 , r = 0.98, p < 0.001), and experienced surgeons (r = 0.86, p standard FLS simulator. = 0.006, r = 0.99, p < 0.001) Conclusions The construct that the standard scoring of the FLS peg transfer task can discriminate between experienced, intermediate, and novice surgeons is validated. The same construct is valid when the task is assessed using the metrics of the ProMIS. The high correlation between these scores establishes the concurrent validity of the ProMIS metrics. The use of AR for objective assessment of FLS tasks could reduce the personnel requirements of assessing these skills while maintaining the objectivity. www.haptica.com | email: promis@haptica.com | tel (NAm) +617 342 7270 | tel (RoW) +353 (0) 1 676 7310
    • Validation 3. SAGES 2007 Education/Outcomes – P279 4. SAGES 2006 Scientific Sessions S065 AUTOMATED PROMIS SIMULATOR METRICS PREDICT WHAT CAN MOTION DERIVATIVES TELL US ABOUT SKILL READINESS FOR FLS CERTIFICATION PERFORMANCE? Anthony L McCluney MD, J Cao, G N Polyhronopoulos MD, D D Laurel N Vuong BS, Steven D Schwaitzberg MD, Caroline G Stanbridge, L S Feldman MD, G M Fried MD, Steinberg-Bernstein Cao PhD, Tufts University School of Medicine, Cambridge Centre for Minimally Invasive Surgery, McGill University, Health Alliance, Tufts University School of Engineering Montreal, QC, Canada Surgical simulators are a popular topic of discussion on Introduction training in laparoscopic surgery. They reduce the need to use human cadavers or animal models for skills SAGES Fundamentals of Laparoscopic Surgery (FLS) tasks are development. A subset of the MISTELS methodology has validated measures of technical skills. Certification requires been employed in the manual skills assessment for the travel to a testing site and a fee, thus a reliable method of Fundamentals Laparoscopic Skills (FLS) program because it predicting readiness for the exam would be advantageous. The was shown to be a valid discriminator of surgical ProMIS simulator (Haptica) provides automated scoring. FLS tasks experience. Pure performance outcome, such as time to can be placed in the ProMIS simulator and scored using time task completion and number of errors, is used for scoring, (TT), as well as motion analysis metrics: instrument path length which is dependent on the consistency of the scorer. A new (PL) and instrument smoothness (IS). This study was designed to simulator environment has been created which uses motion evaluate these automated ProMIS metrics and their ability to tracking for measurement of performance outcome predict readiness for FLS certification. measures and motion derivatives such as smoothness and Methods efficiency. 33 subjects (12 students, 16 residents PGY 1-4, and 5 experts) The purpose of the study was to determine if the motion performed FLS tasks in the standard simulator and in ProMIS. derivatives can be used to automatically and objectively Tasks were scored by FLS and ProMIS metrics. For each ProMIS discriminate experience levels. Twenty-one subjects (6 metric, the total score was calculated by summing the scores for medical students, 14 surgical residents, and 1 expert the 5 FLS tasks. Pearson’s correlations were calculated for surgeon) were recruited to perform the following tasks: peg ProMIS metrics versus standard FLS scores. Multivariate transfer, pattern cutting, pre-tied loop placement, regression analysis identified independent predictors of standard extracorporeal and intracorporeal knot-tying in the new FLS performance. These variables were then used for sensitivity simulator environment. Subjects were evaluated on time to and specificity calculations in order to establish a ProMIS pass- completion, errors, smoothness and total path length (used fail score for predicting readiness for FLS certification. to calculate efficiency). Significance was defined as p<0.05. Results show that experience level is still distinguishable when using task-dependent parameters to evaluate Results performances during peg transfer (p= 0.035), pre-tied loop TT (r= -0.82), PL (r= -0.56), and IS (r= -0.75) all correlated placement (p= 0.022), extracorporeal (p= 0.0006) and significantly with standard FLS score. Multivariate regression intracorporeal (p= 0.025) knot tying in this new simulator analysis identified TT as the strongest predictor of FLS score. A environment. TT score of 1000 maximizes sensitivity and specificity and was Evaluation of performance using task-independent identified as the pass-fail for reliably predicting FLS parameters significantly distinguished training level in three performance. tasks: (1) smoothness of the left instrument was significant Conclusions as a function of experience level in extracorporeal knot- tying (p= 0.016), (2) efficiency (total path length divided by Automated ProMIS metrics correlate well with standard FLS time to completion) was also significant in the movement of performance. In this study sample, a TT score less than 1000 the right tool as a function of experience level in peg reliably predicted a passing FLS certification score. transfer (p= 0.0011) and (3) pre-tied loop placement (p= 0.013979). This preliminary analysis shows that automatic and objectively measured motion derivatives can be associated with the level of experience. These results indicate a potential for the application of an automatic and objective means of skills evaluation. www.haptica.com | email: promis@haptica.com | tel (NAm) +617 342 7270 | tel (RoW) +353 (0) 1 676 7310
    • Validation 5. SAGES 2007 ETP057 6. SAGES 2006 Education/Outcomes–P224 OBJECTIVE MEASUREMENT OF FLS PRECISION CUTTING TASK COMPLEX LAPAROSCOPIC TASK PERFORMANCE ON TWO Derek Young, Fiona Slevin, Derek Cassidy, Donncha Ryan, Haptica NEW COMPUTER-BASED SKILLS TRAINING DEVICES Inc Erika K Fellinger MD, Michael E Ganey MD, Anthony G Gallagher PhD, Daniel J Scott MD, Ron W Bush BS, Neal E The Precision-Cutting Task in the SAGES/ACS FLS Program Seymour MD, Department of Surgery, Baystate Medical requires the user to dissect a circle of specific size and shape Center, Springfield, MA from a marked piece of mesh. Currently, measurement of the accuracy and area dissected is done by observation and by Introduction measuring the dissected mesh on a measurement grid. Using New computer-based skills training devices can simulate and advanced vision-tracking, the ProMIS surgical simulator takes an measure performance of complex surgical tasks. The aim of image of the dissected mesh and automatically generates a this study is to determine basic face and construct validity metric, indicating the accuracy of the shape and area dissected. characteristics of two new devices configured for Method laparoscopic suturing and knot-tying tasks. 1. Once the user has completed the Precision-Cutting Task, Methods ProMIS takes an image of the dissected mesh. The image is At the 2005 SAGES meeting, Learning Center attendees converted to binary image and then scanned using a blob evaluated two computer-based skills training platforms: detection algorithm which produces a list of blobs. SurgicalSIM (SS), a virtual reality (VR) device (METI, 2. The blob with the largest area is taken as the cut out area and Sarasota, FL; SimSurgery, Oslo, Norway) and ProMIS (PM), a the number of pixels are counted inside this area. computer-enhanced video trainer (Haptica, Ltd., Dublin, Ireland). Demographic and training data were collected 3. The actual measurement for area is given in cm2. This is from 73 subjects. All were asked to perform 2 iterations of calculated by counting the number of pixels in a known area of laparoscopic suturing and intracorporeal knot-tying (10- the image and then using ratios to determine the area of the cut minute time limit) on each device. A 6-question survey was out. (Note: this calibration step is achieved by taking the tissue used to define impressions of task realism, relevance, and off the tray and running the blob detection and pixel count on execution using a 5-point Likert scale. Performance data the uncovered black foam of which the exact area is known) (SS: time, path length, errors; PM: time, path length, Results smoothness) were collected on both devices and comparisons made between user-defined expert and Initial trials of the metrics on the FLS Precision- Cutting Task nonexpert (intermediate and novice) groups (ANOVA and show that ProMIS is as accurate — and frequently more accurate Mann Whitney U test). — than the current human observation method. Results 46 subjects used SS and 56 used PM. Task completion rate was 80% for SS and 93% for PM. Experts performed better than nonexperts for all performance measures on SS (composite score 496±41 vs 699±60, p < 0.005) and PM (974±111 vs 1466±89, p < 0.005). Post-task survey scores for PM were significantly higher for perceived realism (overall realism and thread behavior), reflection of clinical ability, and overall educational value. Perception of educational value was not significantly different between the devices among subjects with prior VR experience. Conclusions Using subject-defined expert and nonexpert groups, construct validity was demonstrated for all performance measures on both training devices. Surveyed face validity measures favored the non-VR device, but results also suggest that subjects with prior VR training experience are more apt to accept a new VR surgical training platform. www.haptica.com | email: promis@haptica.com | tel (NAm) +617 342 7270 | tel (RoW) +353 (0) 1 676 7310
    • Validation 7. World J Surg. 2007 Apr;31(4):764-72. 8. SAGES 2007 Education/Outcomes – P270 AUGMENTED VERSUS VIRTUAL REALITY LAPAROSCOPIC COMPARISON AND VALIDATION OF TWO DIFFERENT SIMULATION: WHAT IS THE DIFFERENCE? : A COMPARISON SURGICAL SKILLS SIMULATORS OF THE PROMIS AUGMENTED REALITY LAPAROSCOPIC Dieter Hahnloser MD,Rachel Rosenthal MD,Christian SIMULATOR VERSUS LAPSIM VIRTUAL REALITY Hammel,Daniel Oertli,Markus Müller,Pierre-Alain Clavien, LAPAROSCOPIC SIMULATOR. Department of Visceral and Transplantation Surgery, Botden SM, Buzink SN, Schijven MP, Jakimowicz JJ. Catharina University Hospital Zurich, Switzerland Hospital, Eindhoven, The Netherlands. Background Background Simulators are increasingly incorporated in surgical training Virtual reality (VR) is an emerging new modality for laparoscopic and validation is important. The simulations need to skills training; however, most simulators lack realistic haptic resemble the task they are based upon (face validity) and feedback. Augmented reality (AR) is a new laparoscopic the simulator should be able to differentiate between levels simulation system offering a combination of physical objects and of experience (construct validity). VR simulation. Laparoscopic instruments are used within an Aim hybrid mannequin on tissue or objects while using video tracking. This study was designed to assess the difference in realism, To assess two different types of computer-based simulators: haptic feedback, and didactic value between AR and VR the fully computerised virtual reality (VR) simulator Xitact laparoscopic simulation. LS500 (VR-simulator) and the hybrid ProMisTM simulator. Methods: 146 participants (61%) of the 22nd Davos Methods International Gastrointestinal Surgery Workshop performed The ProMIS AR and LapSim VR simulators were used in this study. on a voluntary basis three similar exercises (camera The participants performed a basic skills task and a suturing task navigation, clip and cut, and dissection) on the two on both simulators, after which they filled out a questionnaire different simulators. Objective performance parameters about their demographics and their opinion of both simulators recorded by either simulator and subjective evaluation by scored on a 5-point Likert scale. The participants were allotted questionnaire were compared between beginner (n=73) and to 3 groups depending on their experience: experts, advanced participants (n=73). intermediates and novices. Significant differences were Results calculated with the paired t-test. The camera navigation exercise was completed by 52% of Results the participants on the VR- and by 47% on the hybrid There was general consensus in all groups that the ProMIS AR simulator with no difference in performance parameters laparoscopic simulator is more realistic than the LapSim VR between beginners and advanced trainees. The hybrid laparoscopic simulator in both the basic skills task (mean 4.22 simulator was graded more realistic (70% vs. 20%, p=.001) resp. 2.18, P < 0.000) as well as the suturing task (mean 4.15 and more useful (65% vs. 36%, p=.043) than the VR- resp. 1.85, P < 0.000). The ProMIS is regarded as having better simulator. Participation was higher at the clip and cut haptic feedback (mean 3.92 resp. 1.92, P < 0.000) and as being exercise (75% VR- and. 52% hybrid simulator) and advanced more useful for training surgical residents (mean 4.51 resp. 2.94, trainees performed significantly better (shorter tool-tip- P < 0.000). travel distance, smoother, quicker and with higher score) on both simulators compared to beginners. The clip and cut Conclusions exercise was graded more realistic on the hybrid (81% vs. In comparison with the VR simulator, the AR laparoscopic 44%, p=.007) and similar useful on both simulators (77% vs. simulator was regarded by all participants as a better simulator 72%). The dissection exercise was completed more often on for laparoscopic skills training on all tested features. the hybrid simulator (47% vs. 23%, p=0.002). Only the hybrid simulator was able to distinguish between advanced trainees and beginners, with significantly higher scores for all performance parameters for the latter. The hybrid simulator was graded more realistic (70% vs. 33%, p=.016) and more useful (83% vs. 62%, p=.12). Overall, acceptance of requirement to train on and to be evaluated by such simulators is still low (53% and 50%, respectively). Conclusion Fully computerized VR- or hybrid simulator performance parameters can distinguish between beginner and advanced trainees for perceptual motor skills (proving construct www.haptica.com | email: promis@haptica.com | tel (NAm) +617 342 7270 | tel (RoW) +353 (0) 1 676 7310
    • Validation validity), but not for visuo-spatial exercises such as the camera __________________________________________ navigation. 10. __________________________________________ 11. SAGES 2006 Scientific Sessions S096 A COMPUTERIZED ANALYSIS OF ROBOTIC VERSUS LAPAROSCOPIC TASK PERFORMANCE 9. SAGES 2007: S077 V K Narula MD, W C Watson MD, S S Davis MD, K Hinshaw BS, COMPUTER-BASED LAPAROSCOPIC AND ROBOTIC SURGICAL B J Needleman MD, D J Mikami MD, J W Hazey MD, J H SIMULATORS: PERFORMANCE CHARACTERISTICS AND Winston MD, P Muscarella MD, M Rubin, V Patel MD, W S PERCEPTIONS OF NEW USERS Melvin MD, The Ohio State University. CMIS. Columbus, OH David W Lin MD, John R Romanelli MD, Renee E Thompson MD,Michael E Ganey MD, Ron W Bush BS, Neal E Seymour MD, Introduction Baystate Medical Center, Department of Surgery Robotic technology has been postulated to improve The expanding inventory of advanced surgical training devices performance in advanced surgical skills. We utilized a novel now includes simulators for laparoscopic and robotic surgery. In computerized assessment system to objectively describe the order to define perceptions of the need and value of such technical enhancement in task performance comparing devices, we evaluated the initial experience of surgeons using robotic and laparoscopic instrumentation. both in the course of performance of an advanced laparoscopic Methods and Procedures skill. Advanced laparoscopic surgeons (2- 10 yrs experience) Methods performed 3 unique task modules using laparoscopic and At the 2006 SAGES meeting, 62 Learning Center attendees evaluated a Telerobotic Surgical Instrumentation (Intuitive Surgical, new virtual reality (VR) robotic surgery simulator (RS) [SimSurgery, Oslo, Sunnyvale, CA). Performance was evaluated using a Norway] as well as either a computer-enhanced laparoscopic [ProMIS computerized assessment system (ProMIS, Dublin, Ireland) (PM), Haptica, Ltd, Dublin, Ireland] or a VR simulator [SurgicalSIM (SS), and results were recorded as time (sec), total path (mm) SimSurgery and METI, Inc, Sarasota, FL]. Demographic and training data and precision. Each surgeon had an initial training session were collected and all were assessed during one iteration of followed by two testing sessions for each module. A Paired laparoscopic suturing and knot-tying on RS and either PM or SS. An 8- Student’s T-Test was used to analyze the data. question survey was used to determine users? impressions of task Results realism, interface quality, and educational value (5-point Likert scale). Performance data [time, path length, smoothness (PM), errors (SS/RS)] 10 surgeons completed the study. Objective assessment of were collected and comparisons made between user-defined groups and the data is presented in the table below. 8/10 surgeons had different simulation platforms (Mann-Whitney Test, ANOVA). significant technical enhancement utilizing robotic technology Laparoscopic vs Robotic Time (sec) Total Results Path(mm) Precision Module 1210 vs 161 # 11649 vs 5571 * Task completion rate was greater for experts than nonexperts on all 1434 vs 933 * Module 2119 vs 68 * 5573 vs 1949 * 853 vs 406 * platforms (PM 100% vs 75%; SS 100% vs 36%; RS 93% vs 36%). Experts Module 377 vs 55 * 4488 vs 2390 * 552 vs 358 * # = p < 0.009 * performed better than nonexperts on all performance measures on PM = p<0.001 Conclusions: The ProMIS computerized assessment (p<0.05: time 154±16 vs 205±12; path length 820±97 vs 1287±97; system can be modified to objectively obtain task smoothness 952±111 vs 1582±127). There were no significant differences performance data with robotic instrumentation. All the between experts and nonexperts for SS and RS performance measures. tasks were performed faster and with more precision using Perception of value of haptic features was less for subjects with prior robot the robotic technology than standard laparoscopy. experience (n=10; p<0.05). Otherwise realism, interface quality, and educational value scores did not differ on the basis of prior simulator or robot use. Nonexperts found that robotic simulation better reflected clinical skill than did experts. Overall, subjective quality was scored higher for PM than for SS or RS. Conclusions The ability of performance metrics of the computer-enhanced simulator to discern predicted performance differences between experts and nonexperts was better than for VR devices with a single task iteration. Initial use of VR devices was associated with a lower overall perception of realism and educational value as compared to use of physical objects in the non-VR simulator. This may reflect the need for familiarization with the computer-generated environment before the educational potential of VR can be realized. www.haptica.com | email: promis@haptica.com | tel (NAm) +617 342 7270 | tel (RoW) +353 (0) 1 676 7310
    • Validation 12. SAGES 2006 Education/Outcomes–P219 13. SAGES 2007: S080 INTEGRATING SIMULATION LAB TRAINING INTO A SURGICAL THE IMPACT OF KNOWLEDGE OF RESULTS IN SURGICAL RESIDENCY PROGRAM: IS VOLUNTARY PARTICIPATION SKILLS TRAINING EFFECTIVE A. O’Connor MD, C. Cao PhD, S. Schwaitzberg MD, Lily Chang MD, James Petros MD, Donald Hess MD, Caroline Department of Mechanical Engineering, Tufts University. Rotondi BA, Timothy Babineau MD, Boston Medical Center Background Objective Concerns about the adequacy of advanced laparoscopic Surgical training programs nationwide are struggling with the training continue to be raised despite a proliferation of integration of simulation training into their curriculum given the training systems exist. The manner in which the training constraints of the 80-hour work week. We examine the modules are structured to maximize learning has not been effectiveness of voluntary training in a simulation lab as part of examined. There are many aspects to the accumulation of the surgical curriculum. Methods: The ProMIS simulator was laparoscopic skills during training, one of which is introduced into the general surgery residency at Boston Knowledge of Results (KR), i.e. the information provided to University Medical Center. All categorical residents (28) were individuals about the outcomes of their motor responses in required to attend a 2-hour training session and curriculum their environment. We studied the effects of KR on the review. Non-categorical residents (23) were given the option to learning curve of laparoscopic suturing and knot tying. complete training. After the introductory session, time spent in Aims the lab was encouraged, but voluntary. Use of the simulator was tracked for all residents. Participation in the simulation We evaluated the learning curves of 9 medical students with curriculum was defined as 3 or more uses of the simulator. After no previous laparoscopic surgical experience under three 3 months, all residents completed a survey regarding the different conditions, each with different levels of simulation lab and their simulator usage. Results: 26 (93%) knowledge of results. categorical residents and 3 (6%) non-categorical residents completed the introductory simulator training session. Over a 3 Methods month period, use of the simulator at least once was 31% among Subjects were randomly assigned to one of three groups. all eligible residents; 80% of PGY1, 40% of PGY2, 60% of PGY3, Each subject attended a training session for 1 hour each and 0% of PGY4 and PGY5. Four residents (14%) participated in day, 6 days a week for 4 weeks. Group 1 (No feedback) the simulation curriculum. 71% of simulator usage was during received no knowledge of results (KR) and no performance working hours while 29% was completed post-call or off duty. feedback. Group 2 (feedback only) received factual KR Most residents agreed that the simulator was easy to use and following each training session, but no coaching. Group 3 improved their operative skills, but did not think it was a good (feedback and coaching) received KR and coaching. Learning substitute for actual operative experience. Reported reasons for curves were plotted based on task time, smoothness of not using the simulator included off-site rotation (44%), no time instruments and instrument’s path length. The task used (30%), and no interest (11%). was an intracorporeal suture/knot tying in the ProMIS laparoscopic simulator. Perceived workload for each session Conclusions was recorded using a standardized NASA TLX workload Voluntary use of a surgical simulation lab leads to minimal score. participation in a training curriculum. Participation should be mandatory if it is to be an effective part of a residency Results curriculum. The variability across each session for each student was calculated for each of the three parameters. There was statistical significance between the groups for all parameters (p-values 0.0002, 0.0002 and 0.009). Significant differences were found between groups 2 and 3 and group 1 (p values 0.0314-0.0410) Groups 2 and 3 learned significantly faster than those in Group 1, reaching performance plateaus at earlier sessions. There were no significant differences between groups 2 and 3 (p-values 0.1211, 0.1758 and 0.1375). Providing individuals with knowledge of results lowered their perceived workload, adding instructional feedback lowered this even further. These results demonstrate that KR is essential for efficient surgical skill acquisition. Individual coaching, a labor intensive proposition, reduces workload but has NO added beneficial effect on the speed of learning. These results www.haptica.com | email: promis@haptica.com | tel (NAm) +617 342 7270 | tel (RoW) +353 (0) 1 676 7310
    • Validation provide a useful basis for developing efficient and cost effective 15. SAGES 2007 Education/Outcomes – P284 surgical skills training curriculum. ABSTRACT VIRTUAL REALITY TRAINING DEVELOPS CORE ____________________________________________________ LAPAROSCOPIC SKILLS COMPARABLE TO EXPERIENCED LAPAROSCOPIC SURGEONS: RESULTS OF A PROSPECTIVE 14. SAGES 2007: Education/Outcomes – P267 RANDOMIZED TRIAL COMPARING TWO VIRTUAL REALITY THE CONSTRUCT VALIDITY OF COMPUTER-DERIVED TRAINERS PERFORMANCE METRICS FOR SELECTED SIMULATED E. Matt Ritter MD, Elisabeth A Pimentel BA, Ryan E Earnest LAPAROSCOPIC TASKS BS, Randy S Haluck MD, Mark W Bowyer MD, National Capital J. A Oostema MD, Matthew Abdel BS, Jon C Gould MD, University Area Medical Simulation Center, Uniformed Services of Wisconsin School of Medicine and Public Health, Department University, Bethesda, Maryland / Department of Surgery, of Surgery Pennsylvania State College of Medicine, Hershey, Pennsylvania Introduction A surgical skills assessment tool is said to demonstrate evidence Introduction of construct validity if users with more experience, and by While simulation is becoming more widely accepted in inference more skill, perform better or more efficiently. surgical training, comparative trials on the training Computer derived motion metrics such as smoothness (the effectiveness of these simulators are lacking. We sought to number of times an instrument tip changes velocity during a compare the effectiveness of two abstract virtual reality task) and path length may be more sensitive measures of skill for trainers to train laparoscopic skills as assessed by the a particular task than traditional metrics such as time. Fundamentals of Laparoscopic Surgery (FLS). We then compared the post training performance of the novice Methods subjects with a group of experienced surgeons. Twenty-four medical students (third year), 19 surgical residents (PG1-5), and 3 attending surgeons were asked to perform four Methods and Results different tasks 3 times in a hybrid computer-based physical 20 novice medical students were recruited. Each subject laparoscopic trainer (ProMIS, Haptica Inc., Dublin). The 4 tasks in performed a pre-test consisting of 3 FLS tasks - Peg Transfer order of complexity were laparoscopic orientation (Task 1), (PT), Pattern Cut (PC) and Intracorporeal Suture (IS) - object positioning (Task 2), sharp dissection (Task 3), and intra- placed in the ProMIS augmented reality simulator (Haptica, corporeal knot tying (Task 4). Metrics recorded were time, path Ireland). They were then randomized to train to length, and smoothness. Laparoscopic operative experience for predetermined levels of proficiency on 3 tasks of the each user was quantified using case logs. Correlations were Minimally Invasive Surgical Trainer-Virtual Reality (MIST-VR) determined using regression analysis and ANOVA. (Mentice, Sweden) or the Rapid Fire/Smart Tutor (RFST) (Verefi, Elizabethtown, PA). After reaching the proficiency Results levels, both groups then took a post test consisting of 3 A statistically significant correlation was observed between trials of the same tasks used for the pre-test. Post test experience and performance for all three metrics for tasks 2-4 performance by both groups was then compared to a control (all p< 0.01). Smoothness was the only metric to correlate in the group, composed of 10 experienced surgeons who had laparoscopic orientation task. Within tasks, time and smoothness completed the same post test. correlate much more strongly with experience and to a similar MIST-VR and RFST groups demonstrated statistically degree. The strongest correlation was observed for the knot significant improvement from the pre-test to the post test tying task (r2=0.60 for time and 0.59 smoothness). An r2=1.0 on all 3 FLS tasks (p < 0.0001). There was no significant would represent a perfect correlation between experience and difference in post test performance between the MIST-VR the specified metric. and RFST groups. When the simulation trained groups were Conclusions compared to experienced controls there was no significant difference in performance with respect to PT. The The computer-derived metrics measured by the hybrid trainer experienced controls did significantly outperformed the correlate with laparoscopic experience. Further study is MIST-VR group in PC (p<0.01) and IS (p<0.05), but necessary to determine if specific metrics are better indicators differences between the experienced controls and the RFST of actual skill. group did not reach statistical significance. Conclusion Simulation based training on either the MISTVR or the RFST simulator improves the skill level of novices as assessed by FLS. The post training skill level of these novices compares favorably with a group of experienced surgeons. Virtual Reality trainers, such as RFST and MISTVR, train www.haptica.com | email: promis@haptica.com | tel (NAm) +617 342 7270 | tel (RoW) +353 (0) 1 676 7310
    • Validation fundamental laparoscopic skills equally and to a level 17. SAGES 2006 Emerging Technologies P036 comparable to a group of experienced practicing surgeons. AUGMENTED REALITY SIMULATOR FOR HAND-ASSISTED ____________________________________________________ LAPAROSCOPIC COLECTOMY Derek Young, Derek Cassidy, Fiona Slevin, Donncha Ryan, Haptica Ltd, Dublin, Ireland. 16. SAGES07 Ergonomics/Instrumentation P307 Training in Hand-Assisted Laparoscopic Colectomy (HALC) A COMPUTERIZED ANALYSIS OF STANDARD VERSUS HIGH has largely been done using cadavers and porcine models. DEXTERITY LAPAROSCOPIC INSTRUMENTATION IN TASK These have drawbacks in terms of realism, logistics and lack PERFORMANCE of performance measurement. A Simulator would provide V K Narula MD,K M Reavis MD,D R Renton MD,D J Mikami MD,B J consistent instruction and practice and provide feedback on Needleman MD,J W Hazey MD,K E Hinshaw BS,W S Melvin MD, performance. However, given the range of instruments used THE OHIO STATE UNIVERSITY HOSPITAL, CENTER FOR MINIMALLY in the procedure, and especially the use of a hand, pure INVASIVE SURGERY virtual reality could not be considered as a solution. Introduction The ProMIS Augmented Reality simulator platform – by combining physical and virtual reality - enables interaction Minimally invasive surgery is becoming the standard of care for and tracking of real instruments with a physical model. And the majority of abdominal procedures. Laparoscopic because of its technological approach (vision-tracking) also instrumentation is constantly undergoing improvements to give enables the hand to be tracked. In the new ProMIS HALC surgeons an advantage. Articulated instrumentation provides a simulator, 3D models or graphical objects are overlaid on distinct advantage in the field of robotic surgery. Applying the the physical model to provide instruction and guidance. For same principles to standard laparoscopic instrumentation could example, a 3D animation may be used to demonstrate how offer increased degrees of freedom to make complex to complete a step; a graphical guideline ‘A – B’ may be laparoscopic tasks easier to perform. We utilized a novel used to indicate a target area for dissection. ProMIS HALC computerized assessment system to objectively evaluate task measures surgical skill by gathering data on the movement performance comparing Standard and High Dexterity (HD) of commercial laparoscopic instruments while completing a laparoscopic instrumentation. standardized task. The main performance metrics are time Methods taken, total path length and economy of movement. Additionally metrics specific to a step are calculated to Advanced laparoscopic surgeons (2-12yrs experience) performed measure performance associated with a specific instruction 3 unique task modules utilizing Standard and HD laparoscopic in a specific region of the physical model. Following the instrumentation (Novare Surgical Systems, Cupertino, CA). simulated procedure, the user completes a self-assessment Performance was evaluated using a computerized assessment which contributes to the metrics for the full procedure. A system (ProMIS, Dublin, Ireland) and results were recorded as full analysis is of performance is presented to the user on time (sec), path (mm), and precision. Each surgeon had an initial completion of the procedure and self-assessment. training session followed by two testing sessions for each module. A Paired Student’s T-Test was used to analyze the data. Results Results: Nine surgeons completed the study. Objective Initial trials of the ProMIS HALC simulator indicate that assessment of the data is presented in the table below. Module 1 practice on the simulator improves performance as was statistically significant, whereas Module 2 and 3 showed no measured by the metrics gathered by the simulator. While difference in task performance with the HD instrumentation. detailed validation studies remain to be done, initial Conclusion indicators are that the HALC simulator represents an “unparalleled opportunity to practice, step by step, a Hand- HD instrumentation is in its infancy. Results showed no assisted laparoscopic sigmoid resection” and “a huge step advantage using HD instrumentation. This could be due to the forward in surgical training”. learning curve associated with new instrumentation and technology. With future developments in HD technology and training, the user interface will improve and may offer an advantage over standard laparoscopic instrumentation. www.haptica.com | email: promis@haptica.com | tel (NAm) +617 342 7270 | tel (RoW) +353 (0) 1 676 7310
    • Validation 18. May 2006 North of England Surgical Society Annual 19. EAES 2006, Poster Registrar’s Meeting Winner of the George Feggeter Gold Medal DEVELOPING PSYCHOMETRIC ASSESSMENT OF LAPAROSCOPIC SKILLS ACQUISITION: IS PSYCHOMETRIC LAPAROSCOPIC SKILLS USING THE PROMIS SIMULATOR MOTION ANALYSIS A VALID ASSESSMENT TOOL? Pellen MGC1;2, Barton JR2, Horgan LF1, Attwood SE1 1;2 2 1 1 Pellen MGC , Barton JR , Horgan LF , Attwood SE Northumbria Northumbria Upper Gastrointestinal Team of Surgeons1; Upper Gastrointestinal Team of Surgeons1; University of University of Newcastle upon Tyne2, United Kingdom Newcastle upon Tyne2, Newcastle upon Tyne, United Kingdom Aims Aims Reliable and validated methods of objective skills training In an evolving climate of competency-based assessment, reliable and assessment are required for trainee surgeons. The and validated methods of objective skills assessment are ProMIS Simulator (Haptica, Ireland) potentially offers a required for trainee surgeons. We aimed to assess whether the method of assessing laparoscopic psychomotor performance. ProMIS Simulator (Haptica, Dublin, Ireland) offers a method of We present initial data from our Centre and Royal College of assessing laparoscopic psychomotor performance. Surgeons Basic Surgical Skills (BSS) Courses. Methods Methods and results Volunteers comprising 17 experienced laparoscopists ( >100 Volunteers comprising 17 experienced laparoscopists (>100 laparoscopic cholecystectomies) and 38 medical students novices laparoscopic cholecystectomies) and 38 medical students (no laparoscopic experience) performed 3 simulated tasks novices (no laparoscopic experience) were assessed on a comprising virtual reality camera navigation, object transfer and complex sharp dissection task (glove over balloon). A sharp dissection task (glove over balloon). A further group of 28 further group of 28 basic surgical trainees (experience basic surgical trainees (experience limited to 1st assistant) limited to 1st assistant) attending BSS Courses were assessed attending BSS Courses were assessed on the same tasks before on the same task before and after training in laparoscopic and after training in laparoscopic skills. Data metrics of time, skills. Data metrics of time, smoothness and path length smoothness and path length were measured via optical tracking were measured via optical tracking of instrum-ent of instrument movement. Objective observations of specific movement as well as observations of specific errors. errors were also recorded. Data analysis (ANOVA) demonstrated experienced Results laparoscopists performed target dissection at least 50% faster, smoother and with more economy of instrument Non-parametric analysis demonstrated experienced movement than students (p<0.05). Experienced participants laparoscopists performed all 3 tasks significantly faster, performed sharp dissection more accurately (p<0.01) smoother and with more economy of movement (p<0.05), although no difference in balloon puncture frequency was excluding camera navigation path length. Experienced seen. Similarly significantly better performance over participants performed sharp dissection more accurately (p<0.01) trainees was demonstrated. Trainees showed only although no difference in balloon puncture was seen. Repeat significantly smoother instrument handling when compared assessment of BSS Course Trainees showed significant to students, possibly reflecting greater baseline dexterity in improvements in simulator metrics (Paired T test, P<0.05), this selective group. Repeat assessment following course although smaller yet significant improvements in “untrained” training showed significant improvements in all metrics by student performance was also seen. 32-40% (Paired T test, P<0.05). Whilst significant Conclusions improvements were also demonstrated in repeat assessment of the untrained student group, these were less marked (15- Gross analysis of these metrics can distinguish between 18%). experience levels supporting construct validity of these simulator tasks. These results suggest potential for objectively measuring Conclusions baseline skill level and response to training. Further work will The gross analysis of these metrics can distinguish between examine the effect of interface familiarisation and defining experience levels supporting the construct validity of this target levels of performance in simulated tasks. simulator task. These results suggest a potential role for objectively measuring baseline skill level and response to training in distinct psychomotor challenges. Further work in progress is examining the effect of interface familiarisation and repeated task performance on novice learning curves and defining target levels of performance in a range of simulated tasks. www.haptica.com | email: promis@haptica.com | tel (NAm) +617 342 7270 | tel (RoW) +353 (0) 1 676 7310
    • Validation 20. Surg Endosc (May 2006) 20: 900–904 21. Surgical endoscopy ISSN: 0930-2794 (Paper) 1432- CONSTRUCT VALIDATION OF A NOVEL HYBRID SURGICAL 2218 (Online) SIMULATOR CONSTRUCT VALIDATION OF THE PROMIS SIMULATOR D. Broe, P. F. Ridgway, S. Johnson, S. Tierney, K. C. Conlon USING A NOVEL LAPAROSCOPIC SUTURING TASK Department of Surgery, Professorial Surgical Unit, Level 4, The K. R. Van Sickle1, D. A. McClusky III1, A. G. Gallagher and Adelaide and Meath Hospital, incorporating the National C. D. Smith1 Children’s Hospital, Tallaght, Dublin 24, Ireland Background Background The use of simulation for minimally invasive surgery (MIS) Simulated minimal access surgery has improved recently as both skills training has many advantages over current traditional a learning and assessment tool. The construct validation of a methods. One advantage of simulation is that it enables an novel simulator, ProMis, is described for use by residents in objective assessment of technical performance. The training. purpose of this study was to determine whether the ProMIS augmented reality simulator could objectively distinguish Methods between levels of performance skills on a complex ProMis is a surgical simulator that can design tasks in both virtual laparoscopic suturing task. and actual reality. A pilot group of surgical residents ranging Methodology from novice to expert completed three standardized tasks: orientation, dissection, and basic suturing. The tasks were tested Ten subjects — five laparoscopic experts and five for construct validity. Two experienced surgeons examined the laparoscopic novices — were assessed for baseline recorded tasks in a blinded fashion using an objective structured perceptual, visio-spatial, and psychomotor abilities using assessment of technical skills format (OSATS: task-specific validated tests. After three trials of a novel laparoscopic checklist and global rating score) as well as metrics delivered by suturing task were performed on the simulator, measures the simulator. for time, smoothness of movement, and path distance were analyzed for each trial. Accuracy and errors were evaluated Results separately by two blinded reviewers to an interrater The findings showed excellent interrater reliability (Cronbach_s reliability of >0.8. Comparisons of mean performance a of 0.88 for the checklist and 0.93 for the global rating). The measures were made between the two groups using a Mann- median scores in the experience groups were statistically Whitney U test. Internal consistency of ProMIS measures was different in both the global rating and the task-specific assessed with coefficient α. checklists (p < 0.05). The scores for the orientation task alone Results did not reach significance (p = 0.1), suggesting that modification is required before ProMis could be used in isolation as an The psychomotor performance of the experts was superior assessment tool. at baseline assessment (p < 0.001). On the laparoscopic suturing task, the experts performed significantly better Conclusions than the novices across all three trials (p < 0.001). They The three simulated tasks in combination are construct valid for performed the tasks between three and four times faster (p differentiating experience levels among surgeons in training. < 0.0001), had three times shorter instrument path length This hybrid simulator has potential added benefits of marrying (p < 0.0001), and had four times greater smoothness of the virtual with actual, and of combining simple box traits and instrument movement (p < 0.009). Experts also showed advanced virtual reality simulation. greater consistency in their performance, as demonstrated by SDs across all measures, which were four times smaller than the novice group. Observed internal consistency of ProMIS measures was high (α = 0.95, p < 0.00001). Conclusions Preliminary results of construct validation efforts of the ProMIS simulator show that it can distinguish between experts and novices and has promising psychometric properties. The attractive feature of ProMIS is that a wide variety of MIS tasks can be used to train and assess technical skills. www.haptica.com | email: promis@haptica.com | tel (NAm) +617 342 7270 | tel (RoW) +353 (0) 1 676 7310
    • Validation 22. EAES 2004 Abstract nr.: O207 23. EAES 2004 abstract nr.: O208 RELATIONSHIP BETWEEN MOTION ANALYSIS, TIME, PSYCHOMOTOR SKILLS ASSESSMENT IN PRACTICING ACCURACY, AND ERRORS DURING PERFORMANCE OF A SURGEONS PERFORMING ADVANCED LAPAROSCOPIC LAPAROSCOPIC SUTURING TASK ON AN AUGMENTED REALITY PROCEDURES II: DEMOGRAPHICS AND PERFORMANCE SIMULATOR PROFILES Author: D.A.M. McClusky, Emory University School of Medicine, Author: A.G. Gallagher, Emory University, Atlanta, United Atlanta Georgia, United States of America. Co-author(s): K. Van States of America. Co-author(s): C.D. Smith, Emory Sickle, Emory University School of Medicine, Atlanta Georgia, University, Atlanta, United States of AmericaR.M. Satava, United States of America University of Washington, Seattle, United States of America A.G. Gallagher, Emory University School of Medicine, Atlanta Background Georgia, United States of America This study reports on the objectively assessed psychomotor Background performance of minimally invasive surgeons on a box-trainer Time, efficient movement, accuracy, and safety are reliable and and a virtual reality (VR) task as a function of handedness, discriminative metrics of proficiency during virtual reality and gender, sight corrected status, and age. Methods: Two box-trainer based minimally invasive surgical (MIS) training. The hundred and ten surgeons attending the 2001 annual role these metrics may serve during more advanced skills training meeting of the American College of Surgeons (ACS) in New are not well understood. Using a novel augmented reality Orleans who reported having completed more than 50 simulator, we sought to gain an understanding of the relationship laparoscopic procedures participated. Subjects completed a between these metrics during an advanced MIS suturing task. box-trainer laparoscopic cutting task and a similar virtual Methods: Eleven subjects completed 3 trials of a suturing task reality task twice. Demographic and laparoscopic designed for a box-trainer and adapted for the ProMIS (Haptica, experience data was also collected. Results: There were no Dublin, Ireland) simulator. Time, tool path, and smoothness of significant differences between subjects performance on movement were assessed using computer algorithms. Measures of either tasks in terms of handedness, gender or whether they accuracy during suture placement and errors in performance were sight corrected or not. A clear and consistent linear were assessed by two blinded reviewers trained to assess trend emerged in terms of age. Older subjects (ages 60 – 69) performance with inter-rater reliability > 0.8. A Pearson’s performed significantly worse than younger subjects (ages correlation coefficient was used to assess the strength of the 30 – 39, 40 – 49) on the box-trainer task for correct incisions relationship between ProMIS metrics and suturing task (13.1 Vs 19.3, p < 0.008) and incorrect incisions (12.3 Vs 2.5, performance. Results: Of the ProMIS metrics, time correlated p > 0.05). They also performed worse on the VR task for with tool path distance and smoothness of movement in three time (132 Vs 71, p < 0.05), error (99 Vs 41, p < 0.05) and trials (range 0.914 – 0.957, p < 0.0001). When the suturing task economy of movement (22.8 Vs 11.7, p < 0.05). Conclusions: was analyzed, accuracy and error score demonstrated an equally Increasing age was found to be associated with a decline in strong relationship (range -0.726 - -0.84, all p < 0.0001). objectively assessed psychomotor performance on two well Combining all metrics, path distance correlated strongest with validated laparoscopic tasks. accuracy (2 trials significant, range -0.67 - -0.93), and error Acknowledgements score (3 trials significant, range 0.54 – 0.61). Smoothness of movement significantly correlated with accuracy in 2 trials This study was supported with grants from the ACS, SAGES, (range -0.63 - -0.88), and time correlated with error score in 2 SLS, TATRC, and Emory University Endosurgery Unit. trials (range 0.56 – 0.60). Conclusion Metrics based on movement efficiency and time, and those based on task accuracy and error scores strongly correlate when grouped independently. At this time, a proficiency curriculum should incorporate both forms of analysis, however further validation work is needed to replicate these findings and give further insight into how ProMIS metrics relate to real-world performance. www.haptica.com | email: promis@haptica.com | tel (NAm) +617 342 7270 | tel (RoW) +353 (0) 1 676 7310
    • Validation 24. JSLS, Journal of the Society of Laparoendoscopic 25. SAGES 2008 P174 Surgeons, Vol. 11, No. 3. (September 2007), pp. 273- FLS TEST IS TRANSFERABLE TO PROMIS SIMULATOR 302. Eric S Hungness MD, Albert Amini BA, Deb E Rooney MS, Eric CRITERION-BASED TRAINING WITH SURGICAL SIMULATORS: T Volckman MD, Nathaniel J Soper MD, Feinberg School of PROFICIENCY OF EXPERIENCED SURGEONS Medicine, Northwestern University, Chicago, IL Heinrichs, Wm, Lukoff, Brian, Youngblood, Patricia, Dev, Parvati, Introduction Shavelson, Richard, Hasson, M Harrith, Satava, M Richard, The McGill Inanimate System for Training and Evaluation Mcdougall, M Elspeth, Wetter, Paul Alan of Laparoscopic Skills (MISTELS) comprises five tasks with Objective an objective scoring system, and has been incorporated by SAGES in their Fundamentals of Laparoscopic Surgery In our effort to establish criterion-based skills training for (FLS) program. MISTELS has high inter-rater and test- surgeons, we assessed the performance of 17 experienced retest reliability and correlates with operative skill. laparoscopic surgeons on basic technical surgical skills recorded However, the FLS program is labor intensive, requiring a electronically in 26 modules selected in five commercially trained proctor. The ProMIS simulator allows for available, computer-based simulators. assessment of physical tasks (instrument path length and instrument smoothness) through instrument tracking Methods/Procedures technology. We hypothesized that the FLS scores obtained in the ProMIS simulator as well as ProMIS Performance data were derived from selected surgeons randomly metrics would correlate with standard FLS scoring. assigned to simulator stations, and practicing repetitively during three one-half day sessions on five different simulators. We Methods measured surgeon proficiency defined as efficient, error-free Twenty general surgery residents (13 junior and 7 senior) performance and developed proficiency score formulas for each had baseline laparoscopic skills assessed using MISTELS in module. Demographic and opinion data were also collected. the standard FLS and ProMIS simulators (pre-test). Nine junior and 4 senior residents had a post-test after four Results weeks of training. Tasks were scored by FLS and ProMIS metrics. Total path length (TP) and total smoothness (TS) Surgeons’ performance demonstrated a sharp learning curve with were calculated by adding the path lengths and the most performance improvement seen in early practice smoothness of each individual task. ANOVA was used to attempts. Median scores and performance levels at the 10th, compare the mean (SD) of total and individual task scores 25th, 75th, and 90th percentiles are provided for each module. for pre- and post-tests in the FLS and ProMIS simulators. Construct validity was examined for two modules by comparing Student's t-test was used to compare ProMIS metrics. experienced surgeons’ performance with that of a convenience Pearson's correlations were calculated for standard FLS sample of lessexperienced surgeons. scores in relation to ProMIS FLS scores, TP and TS. Significance was defined as p < 0.01. Conclusions Results A simple mathematical method for scoring performance is All residents showed statistically significant improvement in applicable to these simulators. Proficiency levels for training post-test total and individual task FLS scores on either the courses can now be specified objectively by residency directors FLS or ProMIS simulator. 100% and 88% of residents achieved and by professional organizations for different levels of training passing post-test scores on the FLS and ProMIS simulator, or post-training assessment of technical performance. But data compared to 30% and 29.2% on the pre-test. There was no users should be cautious due to the small sample size used in this difference in junior and senior resident posttest scores (87.6 study and the need for further study into the reliability and vs 79.1). ProMIS path length and smoothness were validity of the use of surgical simulators as assessment tools. significantly reduced across all tasks (range 14- 68%). Total ProMIS FLS scores (0.729), TP (-0.753) and TS (- 0.769) Summary comment significantly correlated with total standard FLS simulator The simulators used included pure Virtual Reality Simulators and scores. All residents with TP < 4000mm or TS < 6000 two simulators with real haptics (including ProMIS): Lap Mentor achieved a passing total FLS score. from Simbionix (pure Virtual Reality); LapSim from Surgical- Conclusions Science AB (pure Virtual Reality); SurgicalSIM from METI (pure Virtual Reality); ProMIS from Haptica (real haptics with metrics); All surgical residents achieved a passing FLS score after a LTS2000 ISM60 from RealSim (real haptics with metrics). 4-week laparoscopic skills curriculum. FLS tasks are transferable to the ProMIS simulator with traditional FLS ProMIS out-performed the virtual reality simulators with a mean scoring and intrinsic ProMIS metrics being good effectiveness rating of 3.56 versus 3.22 and 3.11 for LapSim and measurement tools. A ProMIS total path length <4000 mm SurgicalSim (LapMentor also scored 3.56). or total smoothness <6000 reliably predicts a passing FLS score. www.haptica.com | email: promis@haptica.com | tel (NAm) +617 342 7270 | tel (RoW) +353 (0) 1 676 7310