Your SlideShare is downloading. ×
0
PhD Proposal Presentation
PhD Proposal Presentation
PhD Proposal Presentation
PhD Proposal Presentation
PhD Proposal Presentation
PhD Proposal Presentation
PhD Proposal Presentation
PhD Proposal Presentation
PhD Proposal Presentation
PhD Proposal Presentation
PhD Proposal Presentation
PhD Proposal Presentation
PhD Proposal Presentation
PhD Proposal Presentation
PhD Proposal Presentation
PhD Proposal Presentation
PhD Proposal Presentation
PhD Proposal Presentation
PhD Proposal Presentation
PhD Proposal Presentation
PhD Proposal Presentation
PhD Proposal Presentation
PhD Proposal Presentation
PhD Proposal Presentation
PhD Proposal Presentation
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

PhD Proposal Presentation

5,858

Published on

My PhD Proposal Presentation from Februrary 14th, 2013 at University of South Australia, School of Computer and Information Science.

My PhD Proposal Presentation from Februrary 14th, 2013 at University of South Australia, School of Computer and Information Science.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
5,858
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Visuo-Haptic Augmented Reality Runtime and Authoring Environment for Medical Training Ulrich Eck Supervisor: Dr. Christian Sandor, University of South Australia Co-Supervisor: Dr. Hamid Laga, University of South Australia Associate Supervisor: Prof. Nassir Navab, TU Munich 14/02/2013, Research Proposal Presentation 1Thank you for joining the presentation of my research proposal with the title:Visuo-haptic Augmented Reality Runtime and Authoring Environment for Medical Training.My supervisor is: Dr. Christian Sandor, my co-supervisor is: Dr. Hamid Laga,and my associate supervisor is: Prof. Nassir Navab from TU Munich.
  • 2. Visuo-Haptic Augmented Reality Applications Previous Work: Rapid Prototyping Vision: Medical Training Authoring Environment User Interface Runtime Environment Development System Haptics Augmented Relality Simulation 2Im starting my presentation with an overview on the proposed research project.The movie on the left side shows one example of my previous work on VHAR applications for rapid prototyping. The user uses a haptic device to paint on a virtual shoe.We have evaluated such a painting application with kids as young as 6 years and found, that the user interface is an intuitive way to interact with virtual object.My vision is to enable and motivate developers and researchers in the domain of medical procedures, to use VHAR user interfaces for their training simulators.A mockup of such a training scenario is shown on the right.In order to enable developers to build their applications with VHAR user interfaces, a new User Interface Development System is needed.It consists of a runtime environment, which provides the required functionality, and an authoring environment, which allows developers to create and modify content and behaviorinteractively.In order to motivate researchers to use this technology in the medical domain, I will design and build medical training application prototypes and evaluate them with domain experts.So lets look a bit closer to what VHAR is, related work in this young field of research, and the challenges in building applications with this user interface technology.
  • 3. VHAR Properties See and touch digital information: embedded in the real world Precisely co-located with haptic devices Improved performance and realism for manual tasks [P. Rhienmora et al., VR 2010] 3The important properties of VHAR are:- It enables users to see and touch digital information, which is embedded in the real world- that haptic feedback and visual output are co-located- previous research has shown, that VHAR improves user performance and realism for manual tasksSome VHAR systems have been developed as part of research projects during the last decade.I picked two interesting examples ...
  • 4. Related Work [Sandor, C. et al., IEICE 2007] 4[Sandor and colleagues.] in 2007 presented a VHAR system which allowed usersto see and touch a virtual car, which is tracked using a real world object.
  • 5. Related Work [Harders, M. et al., TVCG 2009] 5In 2009, Harders and colleagues presented a prototype for a medical training simulatorwith physics-based simulation of soft tissue cutting.Building such systems is challenging ...
  • 6. Challenges in VHAR Accurate co-location of visual rendering and haptic interaction Precise calibration of every component and complete system Low latency, realtime operation 6The visual rendering and haptic interaction need to be accurately co-located.Precise calibration of every component and the complete system is a necessary precondition.Furthermore, VHAR applications need to run in realtime with low latency.But there is more ...
  • 7. More Challenges ... Model representation and transformation: Model simplification (haptic rendering requires simpler geometry than visual rendering) Simulation of deformable bodies (mass-spring systems, finite element method) Most VHAR applications have been built using shared-data, multi-threaded architectures, which is difficult to get right 7Virtual models are needed in multiple representations at the same time for haptic rendering, physics-based simulation, and visual rendering.Deformations of virtual objects need to be synchronized between all representations in realtime.Finally, VHAR applications have been built using shared-data, multi-threaded architectures, which is difficult to get right.In order to simplify and promote the development of such user interfaces, I want to answer the following research questions:
  • 8. Research Questions Is it possible to design and implement a widely applicable VHAR runtime? What is a suitable VHAR application authoring environment for the stakeholders: programmers, designers, usability engineers, and users? What are measurable benefits of applications with VHAR user interfaces in general, and specifically for medical training simulators? 8Is it possible to design and implement a widely applicable VHAR runtime?What is a suitable VHAR application authoring environment for the stakeholders: programmers, designers, usability engineers, and users?What are measurable benefits of applications with VHAR user interfaces in general, and specifically for medical training simulators?
  • 9. Approach Overview Applications Medical ... controls executes Authoring Environment User Interface Development uses System Runtime Environment Haptics Augmented Relality Simulation 9In order to answer these questions, I will first develop a runtime environment, which can be used to execute VHAR applications.An authoring environment, which uses the runtime, enables users interactively create and modify content and behavior.Finally, these applications will be evaluated to show measurable benefits in task performance.Let’s have a closer look on a VHAR system ..
  • 10. VHAR System Decomposition Tracking Simulation Tracker Simulation World Model Engine Capture Haptic Rendering Video Haptic Collision Force Visual Rendering Device Detection Response Graphics Control Engine Algorithms [Eck, U., Honours Thesis 2012] 10This simplified decomposition shows the main components of a typical VHAR application.The haptic device sends sensor readings to the haptic rendering component and receives feedback forces.The haptic rendering component determines collisions with virtual objects and calculates feedback forces based on the penetration depth.The virtual object’s behavior is simulated based on physical laws.Cameras from the head-mounted display capture the environment, which is used a background for the rendered objects and for tracking.The users viewpoint and poses of other real objects are tracked in the video using fiducial markers and potentially fused with poses received from an external tracking system.These poses are then used as input to the simulation engine and for visual rendering.
  • 11. Simplify VHAR Development with a User Interface Development System Similar approach as early graphical user interface development systems [Myers, B., IEEE Software 1989] Dataflow process network architecture with support for parallel execution [Lee, E., Proc. of IEEE 1995] Runtime environment, that connects all components and manages optimal scheduling of tasks on multi-core CPU systems with multiple GPUs [Hermann et al., Euro-Par 2010] Authoring environment, that enables developers to create and modify content and behaviour at runtime [MacWilliams, 2005] 11Building such applications is difficult.I propose to create a UIDS for VHAR applications, similar to early research in graphical user interfaces.As shown by Myers in 1989, UIDS can simplify the development of GUIs by providing appropriate communication patterns and clean apis to developers.The communication pattern in VHAR applications is streams with varying update rates and throughput.A suitable architecture for a set of processing nodes connected via streams is the dataflow process network architecture.The dataflow architecture, which requires side-effect free processing components, decouples algorithms from communication and schedules.The runtime environment implements the dataflow network and provides default implementations for all required components.The authoring environment will use the flexible runtime to provide an interactive development system for creating and modifying content and behavior.Let’s have a closer look onto the concurrency of VHAR systems ...
  • 12. Concurrency of VHAR Subsystems Continuos Time Discrete Time Parallel scheduling of haptic External 100Hz Tracking Sensor rendering, simulation, sensor Tracker Fusion fusion, visual rendering, and computer vision Tracking/ Camera 30Hz Computer Vision Distribution of workload on multi- core CPUs and multiple GPUs Haptic Haptic Simulation Human 1Khz Device Rendering 100-200Hz Efficient data exchange between concurrent subsystems Visual Visual Meet latency requirements for 60Hz Display Rendering realtime operation 12The diagram shows physical devices which provide an interface to the real world, from analog to digital as well as from continuous time to discrete time.All devices operate at different update rates and are normally not synchronized.The update rates vary from 30Hz to 1kHz and the packet sizes range from small pose updates to large image buffers or geometric models.In order to achieve maximum performance, all processing needs to be scheduled optimally for execution on all available CPU cores and GPUs.Furthermore, the cost of communication between components needs to be taken into account, to achieve minimum latency and realtime operation.A unique feature of the runtime environment will be the dynamic optimization of execution schedules.
  • 13. Self Optimizing Dataflow Network Static inputs: - timing requirements (deadlines, latency) - quality requirements (jitter, error) Dataflow Specification - execution requirements (cpu, gpu, ...) iteratively map dataflow network optimally to available resources Scheduling Algorithm Runtime Environment Dynamic inputs: - node: processing time - node: total error - edge: cost of communication Intra-process Connector CPU1 GPU1 CPU2 Inter-process Connector 13This diagram shows the dynamic schedule optimization for dataflow process networksThe dataflow graph connects processing nodes and provides information on static requirements, like timing, quality attributes, or execution context.A scheduling algorithms segments the graph into a partition for every available CPU core and GPU.During runtime, the dataflow runtime provides dynamic information about actual processing time, accumulated errors, and cost of communication.This information is the used to iteratively optimize the graph segmentation until an optimal solution has been found.
  • 14. Authoring Environment Interactive creation and modification of content and behavior Support for development at runtime to [MacWilliams, Thesis 2005] Live code editor with just-in-time compiler [Victor, B., Cusec 2012, Storer, J., Projucer 2012] 14The proposed authoring environment allow the developer to create and modify content and behavior interactively.This can be done in two ways:- Using the development at runtime process as presented by MacWilliams in 2005- And via a live coding environment as demonstrated by Bret Victor in 2012A short video for both approaches follows..
  • 15. Development at Runtime [Sandor et al., ISMAR 2005] 15This clip shows a system developed by Sandor and colleagues in 2005, where users can define the behavior of interaction components at runtime.
  • 16. Live Coding [Victor, B., Cusec 2012] 16This clip shows a live code editor developed by Bret Victor in 2012.I’m planning to create such an environment for either C++ using the LLVM/Clang compiler suite or by using a jit-compiled language such as Racket, Clojure, Julia, or PyPy.Once such an UIDS for VHAR exists, many applications can be built ...
  • 17. Applications for VHAR Medical procedures [Coles, T., PhD Thesis 2011] Training [Knoerlein, B., PhD Thesis 2011] Rapid prototyping [Sandor et al., IEICE 2007] [Eck, Honours Thesis 2012] many applications with haptic interaction 17VHAR user interfaces can improve the user’s experience and performance in many domains, but this has not been studied extensively.As previously shown, medical procedures, training, and rapid prototyping are good candidates.Many haptic enabled applications can benefit from VHAR user interfaces.Although, there are many options - I will focus on medical training scenarios during my research project.
  • 18. State of the Art for Medical Simulation [Ullrich and Kuhlen, VGC 2012] 18This clip shows a typical setup of a VR-based medical training application.In this demo, users can practice palpation and needle insertion using two haptic devices.The visual output is presented on a 3D screen, but the haptic interaction is not co-located with the visual appearance.I think, the user interface should be improved...
  • 19. Improve User Interfaces with VHAR State of the Art My Vision [Ullrich and Kuhlen, VGC 2012] 19As seen before, the haptic interaction and visual rendering are not co-located in current state of the art medical simulators.I propose the use of VHAR user interfaces for medical procedures ...
  • 20. VHAR can Improve Medical Training Benefits: Reduced cognitive load Improved realism Greater flexibility than mockup based simulators Problem: Formal evaluation of benefits is missing 20because they can reduce cognitive load, improve realism, and provide greater flexibility than mockup based simulators.But, in order to successfully deploy VHAR-enabled medical simulators, their benefits need to be formally evaluated.
  • 21. Show Benefits of VHAR User Interfaces Develop medical training prototypes using UIDS Collaboration with TU Munich and German Space Agency DLR Evaluation with medical experts Within-subject user study comparing task performance to haptic-enabled VR training 21Medical training simulators are complex to build and need be evaluated with medical experts.Therefore, we have set up collaborations with TU Munich and the German Space Agency DLR.This collaboration will help me- to define appropriate scenarios for the evaluation of VHAR user interfaces in medical training,- to build them using best-of-breed components for tracking, sensor fusion, collision detection, and haptic rendering,- and to evaluate them with with domain experts in hospitals in Munich.I plan to perform two within-subject user studies comparing task performance of VHAR-enabled simulators with haptic-enabled VR simulators.I’m summarizing the expected research contributions as follows ..
  • 22. Expected Contributions Creation of the first widely applicable dataflow kernel for VHAR applications and a reusable and extensible VHAR runtime environment The design and prototype implementation of the first integrated development and authoring environment for VHAR, which supports the development at runtime process Show measurable benefits of VHAR user interfaces through evaluation of VHAR-enabled applications with a focus on medical training 22The creation of the first widely applicable dataflow kernel for VHAR applications and a reusable and extensible VHAR runtime environmentThe design and prototype implementation of the first integrated development and authoring environment for VHAR, which supports the development at runtime process.To show measurable benefits of VHAR user interfaces through evaluation of VHAR-enabled applications with a focus on medical training.
  • 23. Research Plan and Collaborations DLR MVL TUM 2013 Collision Detection VHAR Runtime Tracking and Haptics with integrate integrate Prototype Sensor Fusion Rigid Bodies Torque / Force TorqueViz Medical VHAR consult Visualisations Evaluation Requirements specify Authoring Env. specify Prototype 2014 Medical VHAR Medical VHAR collaborate Prototyp 1 Prototype 1 Collisions and Improved Authoring Prototype 1 Haptics with integrate refine and Runtime Env. Evaluation Deformable Bodies 2015 Medical VHAR Medical VHAR collaborate Prototype 2 Prototype 2 Prototype 2 Evaluation 23A approximate timeline and an overview on our collaboration with TUM and DLR is shown in this diagram.
  • 24. References Selected References (14 out of 128): Coles, T.R., 2011. Investigating Augmented Reality Visio- Haptic Techniques for Medical Training. Wales: Bangor University. Eck, U., 2011. HARP: A Framework for Visuo-Haptic Augmented Reality Research Projects. Adelaide: University of South Australia. Harders, M. et al., 2009. Calibration, Registration, and Synchronization for High Precision Augmented Reality Haptics. IEEE Transactions on Visualization and Computer Graphics, 15(1), pp.138–149. Hermann, E. et al., 2010. Multi-GPU and multi-CPU parallelization for interactive physics simulations. Euro-Par 2010- Parallel Processing, pp.235–246. Lee, E.A. & Parks, T.M., 1995. Dataflow Process Networks. Proceedings of the IEEE, 83(5), pp.773–801. MacWilliams, A., 2005. A Decentralized Adaptive Architecture for Ubiquitous Augmented Reality Systems. Technische Universität München. Myers, B.A., 1989. User-Interface Tools: Introduction and Survey. IEEE Software, 6(1), pp.15–23. Rhienmora, P. et al., 2010. Augmented Reality Haptics System for Dental Surgical Skills Training. In Proceedings of the 17th ACM Symposium on Virtual Reality Software and Technology. Hong Kong: ACM, pp. 97–98. Sandor, C. et al., 2005. Immersive Mixed-Reality Configuration of Hybrid User Interfaces. In Proceedings of IEEE and Sandor, C. et al., 2007. Exploring Visuo-Haptic Mixed Reality, IEICE. Sandor, C., 2010. Talk at TEDxAdelaide: The Ultimate Display, 2010, Last accessed on 20 November 2012. ACM International Symposium on Mixed and Augmented Reality. Vienna, Austria, pp. 110–113. Storer, J., 2012. Projucer Demo. youtu.be. Available at: http://youtu.be/imkVkRg-geI [Accessed December 6, 2012]. Ullrich, S. & Kuhlen, T., 2012. Haptic Palpation for Medical Simulation in Virtual Environments. Visualization and Computer Graphics, pp.1–9. Victor, B., 2012. Inventing on Principle. worrydream.com. Available at: http://worrydream.com/#!/InventingOnPrinciple [Accessed February 6, 2013]. 24These are the references used in this presentation - 14 out of 128 citations in my research proposal.
  • 25. Applications Previous Work: Rapid Prototyping Vision: Medical Training Authoring Environment User Interface Runtime Environment Development System Questions ? Haptics Augmented Relality Simulation Thank You! Expected Contributions: Creation of the first widely applicable dataflow kernel for VHAR applications and a reusable and extensible VHAR runtime environment The design and prototype implementation of the first integrated development and authoring environment for VHAR, which supports the development at runtime process Show measurable benefits of VHAR user interfaces through evaluation of VHAR-enabled applications with a focus on medical training 25Thank you for listening - Any Questions ?

×