Evaluating the Impact of Virtual Reality on Liver Surgical Planning Procedures 211However, such techniques provide surgeons with just 2D images leaving surgeons tobuild their own 3D model of liver, tumor, and vasculature which is a challenging taskfor even experienced surgeons. Moreover, anatomical variability due to the lesionmay occur making the situation more challenging. Consequently, some importantinformation could be missed leading in non-optimal or even wrong treatmentdecisions. On the other hand, 3D while might appear as a candidate solution to theaforementioned challenges, its visualization in such medical context is not evensufficient if presented on conventional displays. Moreover, surgical planning for liverresection requires a qualitative analysis of volumes and distances related to the liverstructure. Examples include the relative size (volume) of the tumor to the overall livertissues which is currently estimated on basis of the gained data collected via CTimages. Currently, surgeons rely on their built mental model without having possibletruthful simulation of the actual resection.2 Potential Impact of Image Guided Systems to the Surgical DomainThe continuous progress in technology has made transition to clinical practicereplacing traditional open surgical procedures with minimally invasive techniques. Incontrast to open surgery, image guided procedures (IGP), physicians identifyanatomical structures in images (segmentation) and mentally establish the spatialrelationship between the imagery and the patient registration. To gain an insight intothe major IGP technologies involved in surgical procedures, it becomes important tostate first the three main phases of surgical planning procedures: (i) pre-operativeplanning, (ii) intra-operative plan execution and (iii) post-operative assessment. First,the key technologies involved in pre-operative planning are: (1) medical imagingincluding the correction of geometric and intensity distortions in images, (2) datavisualization and manipulation of image and patients’ data, (3) segmentation orclassification of image data into anatomically meaningful structures and (4)registration: alignment of data into a single coordinate system. Second, thetechnologies related to the inter-operative plan execution include the aforementionedtechnologies in addition to tracking systems for localizing the spatial position andorientation of anatomy and tools and the human computer interaction (HCI). To sumup the key technologies involved are: • Medical imaging and image processing • Data visualization • Segmentation • Registration • Tracking systems • Human Computer Interaction (HCI)
212 N. El-Khameesy and H. El-Wishy3 The Proposed Virtual Liver Surgery Planning ModelIts our claim that VR presents a potential promising solution to enhance the tasksinvolved in surgical planning procedures as it efficiently counterpart the challengesand difficulties incurred by the CT data. The main idea of VLSPS is to supportradiologists while preparing data sets of patient’s liver and to provide surgeon withmore insight and to navigate along 3D images. The addressed gain is to minimize thetime needed to collect data during the pre-surgical planning stage and to automate thework done. While, a fully automated segmentation is not yet available due to the largevariability of shape and gray level distribution of normal or diseased tissue, it’s nowpossible to adopt a semi-automated approach.3.1 Frame Work of the VLSP ModelThe main idea of our proposed model is to employ the VLSPS in a hybrid userinterface (e.g. both 2D and 3D) to enhance the user interaction and still takeadvantage of the gained information of the 2D data set which is to some extent amature technique. The proposed system can be identified as three major subsystems:(1) medical image analysis, (2) interactive segmentation refinement and (3) resectionplanning. The following comments highlight some of the implied tasks in the adoptedmodel as shown in figure (1): • First part employs robust algorithms for segmentation of liver/tumor, vessel extraction and Voxel -based segment approximation • Second part, aims at filtering, refining and verifying results gained from segmentation of first part via a hybrid interface. • Third part provides the necessary components for a VR-based resection planning environment. It employs measurements tools, volumetric partitioning and general resection tools. • Many VR hardware tools, in addition to high specification workstation of high resolution monitor, are utilized including shuttle glasses, head mount display (HMD), tracking pencil and transparent personal interaction pen (PIP). • Actual surgical planning is completely gained via desktop by enabling surgeons gain 3D user interface at six levels of degree of freedom.On the other hand, the image analysis part of the system aims at preparing raw data ofthe CT input in order to be used for the 3D visualization. Such objectives have to facea number of challenging tasks which can be summarized as: (1) difficulties of propersegmenting liver and tumor because in some cases local borders of the liver toneighboring structures (e.g. heart) are virtually not present in the image source, (2)
Evaluating the Impact of Virtual Reality on Liver Surgical Planning Procedures 213 Fig. 1. Main subsystems of the Proposed Liver Surgical Planning System (VLPS)challenges due to proper extraction of portal vein tree as good as possible, which isimportant for precise liver segment approximation and (3) challenges facingdevelopment of algorithms for generating a correct liver segment approximationsbased on a given portal labeling information. In order to counteract the precedingchallenges, first task provides interaction with the segmentation refinement block ofthe system as fully automated approach is not viable yet due to the inhomogeneousstructure of liver and tumor tissues. In the second task, interaction is limited to thespecification of correct parameters for tweaking segmentation results. Liverpartitioning requires surgical knowledge; therefore it’s related to the resectionplanning block of the VLSPS. An interactive labeling technique of segment-feedingvessel branches is carried out, which can easily be done using direct 3D interactions.Segment classification of liver can’t be visible in CT data but can be achieved in themodel via segment approximation algorithms. Such algorithms can provide surgeonswith automatically generated eight different liver segments labeled according to thelabeled segment feeding vessel branches. Figure (2) shows an example forpartitioning the liver into its main segments based on a labeled portal vein tree. Figure(2-a) shows the formal portal tree representation while Figure (2-b) shows the labeledsurface based representation of the portal vein tree. Figures (2-c, d) show a front and aback view of the resulting liver segments displayed color-coded as surfaces.3.2 VR-Based Segmentation Refinement ToolsThe radiologist’s task is to deliver correct segmentation results to surgeons before theactual resection planning takes place. The VLSPS therefore integrates different tools
214 N. El-Khameesy and H. El-WishyFig. 2. A liver Segment Approximation based on a labeled portal vein tree.2(a) Formal portaltree representation, 2(b) Labeled portal tree model, 2(c) Classified Liver Segments usingnearest neighbor approximation (front view) and 2(d) Classified Liver Segments using nearestneighbor approximation (back view).for segmentation verification and editing. The original CT data are projected astextures on the backside of the PIP which is a required information source during thewhole refinement process. It is possible to make CT snapshots of arbitrarily orientedplanes which can be positioned in space. In order to enable interactive segmentation editing, a set of tools are embedded intothe VR system for true 3D interaction. However, for specifying precise landmarkpoints, 2D inputs may be useful and therefore, a hybrid user interface is currentlydeveloped which allows a combined 2D and 3D interaction. Figure (3) implies ideasbehind the VR-based segmentation refinement using free-form deformation. Asegmentation error can easily be corrected in a VR setup using the pencil for surfacedragging and the PIP for showing additional context information.
Evaluating the Impact of Virtual Reality on Liver Surgical Planning Procedures 215Fig. 3. Example of a VR-based Interactive Segmentation Refinement deforming the LiverSurface: (a ) Detecting the region where segmentation failed. (b) Locating the best refinementposition. (c) Starting the refinement process using context information on the PIP.(d) Observing the segmentation refinement result.4 The Proposed Surgical Planning WorkflowTypically, the common practice for surgical planning starts after the radiologist assurescorrect segmentation of liver, tumor, and portal vein then surgeons start with the actualresection planning process. Meanwhile, in contrast to the hybrid user interface, thesurgeons favor 3D interaction for planning, since they’re highly 3D-oriented in theirclinical routine. During an intervention, all movements with surgical devices take placein 3D, and the objects to interact with, are also 3D. The suggested surgical planningworkflow has been elaborated based on collaborative efforts with surgeons; andsuch workflow is feasible for planning an intervention for patients suffering from HCCconsidered here as an empirical case study. The elaborated workflow is shown in Figure( 4 ). It’s assumed here, that each required object (i.e. liver, tumors, and vessel tree) hasbeen already correctly segmented and approved by a radiologist. After radiologicalvalidation, each object is stored as an individual surface mesh (i.e. simplex mesh) and isthen transformed into a triangular model. By using the mesh generation algorithms, atetrahedral data model is generated including the liver boundary and all tumors. Thevessel tree is treated separately, since its complexity would have negative effect on theoverall performance of the tetrahedral mesh. A suitable number of elements, for
216 N. El-Khameesy and H. El-Wishytetrahedral meshes modeling a liver dataset, is about 100k. Since the vessel tree is verycomplex in geometry (e.g. 30k triangles are necessary), this number of target tetrahedralmeshes cannot be reached without loss of information. Once the model is generated, the surgeons can start their planning process byinspecting the data, especially the location of the tumor and the arrangement of theportal vessel tree. For malignant tumors, segmentation approximation is necessary inorder to retrieve the information about affected liver segments. Therefore, the vessel treeis labeled according to anatomical knowledge about segment-feeding branches. Byapplying a preview (segment approximation only applied on the liver surface), labelingcan be altered until the surgeon is satisfied with the result. In a next step, liver segmentsare calculated based on the underlying volumetric tetrahedral mesh. The result is againinspected by the surgeon, in order to find a decision which resection strategy isadequate. In case of an anatomical resection, the automated generated resection proposalcan optionally be applied to perform collision detection with affected liver segments.The result of this proposal is verified and can be edited by adjusting the safety marginaround the tumor. Additionally, measurement tools can be used for further quantitativeanalysis in an iterative process. If a typical resection is the only solution, one of thebuilt-in resection tools is used allowing a classification of liver tissue into: resected andremaining regions and again, a validation using the oncological safety margin must beperformed. Further adjustments of the resected region are also possible by applyingadditional partitioning operations. The outcome of the surgical planning consists of aresection plan and quantitative indices about resected and remaining liver tissue. Fig. 4. The Proposed Surgical Planning Workflow
Evaluating the Impact of Virtual Reality on Liver Surgical Planning Procedures 2175 Results and ConclusionsThis research has demonstrated the added value of collaborative research combiningauthors of both informatics interest and medical surgeons. Coupling their viewsenabled achieving applicable results in the medical domain of liver surgical planning.The proposed model has been highly accepted as a result of analyzing the response ofover 20 surgeons in the largest six liver medical centers in Egypt. The research hashighlighted and verified the potential advantages and the added value of using VRtechniques and tools in the surgical planning domain in general and in liver surgicalplanning in particular. Results of adopting the proposed work flow of the VLPS in thesurgical planning domain have shown quite an appreciation when demonstrated,tested and validated according to the considered surgeon set leading to a refinedapplicable liver surgical planning work flow. A significant gain has been provenfrom perspective of surgical planning task completion as it enabled trained surgeon toachieve a resection plan in less than 30 minutes including the calculation time forFig. 5. The results gained from the proposed model provide surgeons with many valuablequantitative indices such as volume of liver segments, resected and remaining liver tissues.Consequently, the VLSPS would enable surgeons to elaborate more efficiently a better surgicalplanning strategy.
218 N. El-Khameesy and H. El-Wishyliver segments. Moreover, the adoption of such VR based models can be also adoptedin enhancing clinical routines. However, cost as well as the need for more integratedVR tools still presents a challenge for spreading out such systems especially in smallclinics and/or by liver specialists in their own offices.References 1. Myers: Quantitative Research in Information Systems. Sage Publications (2009) 2. Bernard, R., Alexander, B., Beichel, R., Schmalstieg, D.: Liver Surgery Planning using Virtual Reality. IEEE Journal of Computer Graphics and Applications 26(6), 36–47 (2006) 3. Beichel, R.: Virtual Liver Surgery Planning: Segmentation of CT Data., PhD thesis, Graz University of Technology (2005) 4. Burdea, G., Coiffet, P.: Virtual Reality Technology, 2nd edn. Wiley-Interscience Pub. (2003) 5. Feiner, S.: Augmented Reality: A New way of Seeing. Journal of Scientific American Science and Technology (2002) 6. Preim, B., Tietjen, C., Spindler, W., Peitegen, H.: Integration of Measurement Tools in Medical 3D Visulaizations. In: IEEE Visualization 2002, pp. 21–28 (2002b) 7. Blackwell, M., Nikou, C., DiGioia, A., Kanade, T.: An Image Overlay System for Medical Visulaization. Journal of Medical Image Analysis 4(1), 67–72 (2000) 8. Maintz, J.B., Viergever, M.A.: A Survey of Medical Image Registration. Journal of Medical Image Analysis 2(1), 1–37 (1998) 9. Robb, R.A.: VR Assisted Surgery Planning. IEEE Mag. of Engineering Medicine and Biology 15(1), 60–69 (1996)10. Robb, R.: Three Dimensional Biomedical Imaging: Principles and Practice. VCH Pub., New York (1994)11. Delingette, H.: Simplex Meches: A General Representation for 3D Shape reconstruction. In: Proceedings the Int. Conf. on Computer Vision and Pattern Recognition (CVPR 1994), Seattle, USA, pp. 856–857 (1994)