Visuo-Haptic Augmented Reality Runtime and Authoring Environment for Medical Training Ulrich Eck Supervisor: Dr. Christian Sandor, University of South Australia Co-Supervisor: Dr. Hamid Laga, University of South Australia Associate Supervisor: Prof. Nassir Navab, TU Munich 14/02/2013, Research Proposal Presentation 1Thank you for joining the presentation of my research proposal with the title:Visuo-haptic Augmented Reality Runtime and Authoring Environment for Medical Training.My supervisor is: Dr. Christian Sandor, my co-supervisor is: Dr. Hamid Laga,and my associate supervisor is: Prof. Nassir Navab from TU Munich.
Visuo-Haptic Augmented Reality Applications Previous Work: Rapid Prototyping Vision: Medical Training Authoring Environment User Interface Runtime Environment Development System Haptics Augmented Relality Simulation 2Im starting my presentation with an overview on the proposed research project.The movie on the left side shows one example of my previous work on VHAR applications for rapid prototyping. The user uses a haptic device to paint on a virtual shoe.We have evaluated such a painting application with kids as young as 6 years and found, that the user interface is an intuitive way to interact with virtual object.My vision is to enable and motivate developers and researchers in the domain of medical procedures, to use VHAR user interfaces for their training simulators.A mockup of such a training scenario is shown on the right.In order to enable developers to build their applications with VHAR user interfaces, a new User Interface Development System is needed.It consists of a runtime environment, which provides the required functionality, and an authoring environment, which allows developers to create and modify content and behaviorinteractively.In order to motivate researchers to use this technology in the medical domain, I will design and build medical training application prototypes and evaluate them with domain experts.So lets look a bit closer to what VHAR is, related work in this young ﬁeld of research, and the challenges in building applications with this user interface technology.
VHAR Properties See and touch digital information: embedded in the real world Precisely co-located with haptic devices Improved performance and realism for manual tasks [P. Rhienmora et al., VR 2010] 3The important properties of VHAR are:- It enables users to see and touch digital information, which is embedded in the real world- that haptic feedback and visual output are co-located- previous research has shown, that VHAR improves user performance and realism for manual tasksSome VHAR systems have been developed as part of research projects during the last decade.I picked two interesting examples ...
Related Work [Sandor, C. et al., IEICE 2007] 4[Sandor and colleagues.] in 2007 presented a VHAR system which allowed usersto see and touch a virtual car, which is tracked using a real world object.
Related Work [Harders, M. et al., TVCG 2009] 5In 2009, Harders and colleagues presented a prototype for a medical training simulatorwith physics-based simulation of soft tissue cutting.Building such systems is challenging ...
Challenges in VHAR Accurate co-location of visual rendering and haptic interaction Precise calibration of every component and complete system Low latency, realtime operation 6The visual rendering and haptic interaction need to be accurately co-located.Precise calibration of every component and the complete system is a necessary precondition.Furthermore, VHAR applications need to run in realtime with low latency.But there is more ...
More Challenges ... Model representation and transformation: Model simpliﬁcation (haptic rendering requires simpler geometry than visual rendering) Simulation of deformable bodies (mass-spring systems, ﬁnite element method) Most VHAR applications have been built using shared-data, multi-threaded architectures, which is difﬁcult to get right 7Virtual models are needed in multiple representations at the same time for haptic rendering, physics-based simulation, and visual rendering.Deformations of virtual objects need to be synchronized between all representations in realtime.Finally, VHAR applications have been built using shared-data, multi-threaded architectures, which is difﬁcult to get right.In order to simplify and promote the development of such user interfaces, I want to answer the following research questions:
Research Questions Is it possible to design and implement a widely applicable VHAR runtime? What is a suitable VHAR application authoring environment for the stakeholders: programmers, designers, usability engineers, and users? What are measurable beneﬁts of applications with VHAR user interfaces in general, and speciﬁcally for medical training simulators? 8Is it possible to design and implement a widely applicable VHAR runtime?What is a suitable VHAR application authoring environment for the stakeholders: programmers, designers, usability engineers, and users?What are measurable beneﬁts of applications with VHAR user interfaces in general, and speciﬁcally for medical training simulators?
Approach Overview Applications Medical ... controls executes Authoring Environment User Interface Development uses System Runtime Environment Haptics Augmented Relality Simulation 9In order to answer these questions, I will ﬁrst develop a runtime environment, which can be used to execute VHAR applications.An authoring environment, which uses the runtime, enables users interactively create and modify content and behavior.Finally, these applications will be evaluated to show measurable beneﬁts in task performance.Let’s have a closer look on a VHAR system ..
VHAR System Decomposition Tracking Simulation Tracker Simulation World Model Engine Capture Haptic Rendering Video Haptic Collision Force Visual Rendering Device Detection Response Graphics Control Engine Algorithms [Eck, U., Honours Thesis 2012] 10This simpliﬁed decomposition shows the main components of a typical VHAR application.The haptic device sends sensor readings to the haptic rendering component and receives feedback forces.The haptic rendering component determines collisions with virtual objects and calculates feedback forces based on the penetration depth.The virtual object’s behavior is simulated based on physical laws.Cameras from the head-mounted display capture the environment, which is used a background for the rendered objects and for tracking.The users viewpoint and poses of other real objects are tracked in the video using ﬁducial markers and potentially fused with poses received from an external tracking system.These poses are then used as input to the simulation engine and for visual rendering.
Simplify VHAR Development with a User Interface Development System Similar approach as early graphical user interface development systems [Myers, B., IEEE Software 1989] Dataﬂow process network architecture with support for parallel execution [Lee, E., Proc. of IEEE 1995] Runtime environment, that connects all components and manages optimal scheduling of tasks on multi-core CPU systems with multiple GPUs [Hermann et al., Euro-Par 2010] Authoring environment, that enables developers to create and modify content and behaviour at runtime [MacWilliams, 2005] 11Building such applications is difﬁcult.I propose to create a UIDS for VHAR applications, similar to early research in graphical user interfaces.As shown by Myers in 1989, UIDS can simplify the development of GUIs by providing appropriate communication patterns and clean apis to developers.The communication pattern in VHAR applications is streams with varying update rates and throughput.A suitable architecture for a set of processing nodes connected via streams is the dataﬂow process network architecture.The dataﬂow architecture, which requires side-effect free processing components, decouples algorithms from communication and schedules.The runtime environment implements the dataﬂow network and provides default implementations for all required components.The authoring environment will use the ﬂexible runtime to provide an interactive development system for creating and modifying content and behavior.Let’s have a closer look onto the concurrency of VHAR systems ...
Concurrency of VHAR Subsystems Continuos Time Discrete Time Parallel scheduling of haptic External 100Hz Tracking Sensor rendering, simulation, sensor Tracker Fusion fusion, visual rendering, and computer vision Tracking/ Camera 30Hz Computer Vision Distribution of workload on multi- core CPUs and multiple GPUs Haptic Haptic Simulation Human 1Khz Device Rendering 100-200Hz Efﬁcient data exchange between concurrent subsystems Visual Visual Meet latency requirements for 60Hz Display Rendering realtime operation 12The diagram shows physical devices which provide an interface to the real world, from analog to digital as well as from continuous time to discrete time.All devices operate at different update rates and are normally not synchronized.The update rates vary from 30Hz to 1kHz and the packet sizes range from small pose updates to large image buffers or geometric models.In order to achieve maximum performance, all processing needs to be scheduled optimally for execution on all available CPU cores and GPUs.Furthermore, the cost of communication between components needs to be taken into account, to achieve minimum latency and realtime operation.A unique feature of the runtime environment will be the dynamic optimization of execution schedules.
Self Optimizing Dataﬂow Network Static inputs: - timing requirements (deadlines, latency) - quality requirements (jitter, error) Dataﬂow Speciﬁcation - execution requirements (cpu, gpu, ...) iteratively map dataﬂow network optimally to available resources Scheduling Algorithm Runtime Environment Dynamic inputs: - node: processing time - node: total error - edge: cost of communication Intra-process Connector CPU1 GPU1 CPU2 Inter-process Connector 13This diagram shows the dynamic schedule optimization for dataﬂow process networksThe dataﬂow graph connects processing nodes and provides information on static requirements, like timing, quality attributes, or execution context.A scheduling algorithms segments the graph into a partition for every available CPU core and GPU.During runtime, the dataﬂow runtime provides dynamic information about actual processing time, accumulated errors, and cost of communication.This information is the used to iteratively optimize the graph segmentation until an optimal solution has been found.
Authoring Environment Interactive creation and modiﬁcation of content and behavior Support for development at runtime to [MacWilliams, Thesis 2005] Live code editor with just-in-time compiler [Victor, B., Cusec 2012, Storer, J., Projucer 2012] 14The proposed authoring environment allow the developer to create and modify content and behavior interactively.This can be done in two ways:- Using the development at runtime process as presented by MacWilliams in 2005- And via a live coding environment as demonstrated by Bret Victor in 2012A short video for both approaches follows..
Development at Runtime [Sandor et al., ISMAR 2005] 15This clip shows a system developed by Sandor and colleagues in 2005, where users can deﬁne the behavior of interaction components at runtime.
Live Coding [Victor, B., Cusec 2012] 16This clip shows a live code editor developed by Bret Victor in 2012.I’m planning to create such an environment for either C++ using the LLVM/Clang compiler suite or by using a jit-compiled language such as Racket, Clojure, Julia, or PyPy.Once such an UIDS for VHAR exists, many applications can be built ...
Applications for VHAR Medical procedures [Coles, T., PhD Thesis 2011] Training [Knoerlein, B., PhD Thesis 2011] Rapid prototyping [Sandor et al., IEICE 2007] [Eck, Honours Thesis 2012] many applications with haptic interaction 17VHAR user interfaces can improve the user’s experience and performance in many domains, but this has not been studied extensively.As previously shown, medical procedures, training, and rapid prototyping are good candidates.Many haptic enabled applications can beneﬁt from VHAR user interfaces.Although, there are many options - I will focus on medical training scenarios during my research project.
State of the Art for Medical Simulation [Ullrich and Kuhlen, VGC 2012] 18This clip shows a typical setup of a VR-based medical training application.In this demo, users can practice palpation and needle insertion using two haptic devices.The visual output is presented on a 3D screen, but the haptic interaction is not co-located with the visual appearance.I think, the user interface should be improved...
Improve User Interfaces with VHAR State of the Art My Vision [Ullrich and Kuhlen, VGC 2012] 19As seen before, the haptic interaction and visual rendering are not co-located in current state of the art medical simulators.I propose the use of VHAR user interfaces for medical procedures ...
VHAR can Improve Medical Training Beneﬁts: Reduced cognitive load Improved realism Greater ﬂexibility than mockup based simulators Problem: Formal evaluation of beneﬁts is missing 20because they can reduce cognitive load, improve realism, and provide greater ﬂexibility than mockup based simulators.But, in order to successfully deploy VHAR-enabled medical simulators, their beneﬁts need to be formally evaluated.
Show Beneﬁts of VHAR User Interfaces Develop medical training prototypes using UIDS Collaboration with TU Munich and German Space Agency DLR Evaluation with medical experts Within-subject user study comparing task performance to haptic-enabled VR training 21Medical training simulators are complex to build and need be evaluated with medical experts.Therefore, we have set up collaborations with TU Munich and the German Space Agency DLR.This collaboration will help me- to deﬁne appropriate scenarios for the evaluation of VHAR user interfaces in medical training,- to build them using best-of-breed components for tracking, sensor fusion, collision detection, and haptic rendering,- and to evaluate them with with domain experts in hospitals in Munich.I plan to perform two within-subject user studies comparing task performance of VHAR-enabled simulators with haptic-enabled VR simulators.I’m summarizing the expected research contributions as follows ..
Expected Contributions Creation of the ﬁrst widely applicable dataﬂow kernel for VHAR applications and a reusable and extensible VHAR runtime environment The design and prototype implementation of the ﬁrst integrated development and authoring environment for VHAR, which supports the development at runtime process Show measurable beneﬁts of VHAR user interfaces through evaluation of VHAR-enabled applications with a focus on medical training 22The creation of the ﬁrst widely applicable dataﬂow kernel for VHAR applications and a reusable and extensible VHAR runtime environmentThe design and prototype implementation of the ﬁrst integrated development and authoring environment for VHAR, which supports the development at runtime process.To show measurable beneﬁts of VHAR user interfaces through evaluation of VHAR-enabled applications with a focus on medical training.
Research Plan and Collaborations DLR MVL TUM 2013 Collision Detection VHAR Runtime Tracking and Haptics with integrate integrate Prototype Sensor Fusion Rigid Bodies Torque / Force TorqueViz Medical VHAR consult Visualisations Evaluation Requirements specify Authoring Env. specify Prototype 2014 Medical VHAR Medical VHAR collaborate Prototyp 1 Prototype 1 Collisions and Improved Authoring Prototype 1 Haptics with integrate reﬁne and Runtime Env. Evaluation Deformable Bodies 2015 Medical VHAR Medical VHAR collaborate Prototype 2 Prototype 2 Prototype 2 Evaluation 23A approximate timeline and an overview on our collaboration with TUM and DLR is shown in this diagram.
References Selected References (14 out of 128): Coles, T.R., 2011. Investigating Augmented Reality Visio- Haptic Techniques for Medical Training. Wales: Bangor University. Eck, U., 2011. HARP: A Framework for Visuo-Haptic Augmented Reality Research Projects. Adelaide: University of South Australia. Harders, M. et al., 2009. Calibration, Registration, and Synchronization for High Precision Augmented Reality Haptics. IEEE Transactions on Visualization and Computer Graphics, 15(1), pp.138–149. Hermann, E. et al., 2010. Multi-GPU and multi-CPU parallelization for interactive physics simulations. Euro-Par 2010- Parallel Processing, pp.235–246. Lee, E.A. & Parks, T.M., 1995. Dataﬂow Process Networks. Proceedings of the IEEE, 83(5), pp.773–801. MacWilliams, A., 2005. A Decentralized Adaptive Architecture for Ubiquitous Augmented Reality Systems. Technische Universität München. Myers, B.A., 1989. User-Interface Tools: Introduction and Survey. IEEE Software, 6(1), pp.15–23. Rhienmora, P. et al., 2010. Augmented Reality Haptics System for Dental Surgical Skills Training. In Proceedings of the 17th ACM Symposium on Virtual Reality Software and Technology. Hong Kong: ACM, pp. 97–98. Sandor, C. et al., 2005. Immersive Mixed-Reality Conﬁguration of Hybrid User Interfaces. In Proceedings of IEEE and Sandor, C. et al., 2007. Exploring Visuo-Haptic Mixed Reality, IEICE. Sandor, C., 2010. Talk at TEDxAdelaide: The Ultimate Display, 2010, Last accessed on 20 November 2012. ACM International Symposium on Mixed and Augmented Reality. Vienna, Austria, pp. 110–113. Storer, J., 2012. Projucer Demo. youtu.be. Available at: http://youtu.be/imkVkRg-geI [Accessed December 6, 2012]. Ullrich, S. & Kuhlen, T., 2012. Haptic Palpation for Medical Simulation in Virtual Environments. Visualization and Computer Graphics, pp.1–9. Victor, B., 2012. Inventing on Principle. worrydream.com. Available at: http://worrydream.com/#!/InventingOnPrinciple [Accessed February 6, 2013]. 24These are the references used in this presentation - 14 out of 128 citations in my research proposal.
Applications Previous Work: Rapid Prototyping Vision: Medical Training Authoring Environment User Interface Runtime Environment Development System Questions ? Haptics Augmented Relality Simulation Thank You! Expected Contributions: Creation of the ﬁrst widely applicable dataﬂow kernel for VHAR applications and a reusable and extensible VHAR runtime environment The design and prototype implementation of the ﬁrst integrated development and authoring environment for VHAR, which supports the development at runtime process Show measurable beneﬁts of VHAR user interfaces through evaluation of VHAR-enabled applications with a focus on medical training 25Thank you for listening - Any Questions ?