Matera Summer School AI&XR 2023.pdf
•
0 likes•79 views
The document discusses approaches and challenges related to using virtual and augmented reality in healthcare. It provides an introduction and motivation for using medical AR/VR. Some healthcare scenarios discussed include using VR for radiologists in diagnostic reading rooms, virtual colonoscopy, and augmented reality for surgery. Challenges addressed include building networks of healthcare professionals, linguistic barriers, and "automagic worshippers" who expect too much without understanding specifications. The document emphasizes the importance of evaluation and usability testing with medical professionals.
Recommended
Recommended
More Related Content
Similar to Matera Summer School AI&XR 2023.pdf
Similar to Matera Summer School AI&XR 2023.pdf (19)
More from Joaquim Jorge
More from Joaquim Jorge (20)
Recently uploaded
Recently uploaded (20)
Matera Summer School AI&XR 2023.pdf
- 1. Approaches and Challenges to Virtual and Augmented Reality in Health Care Joaquim Jorge Anderson Maciel INESC-ID & IST / U Lisboa Catarina Moreira UTS, Sydney Nuno Figueiredo Grupo Lusíadas, Lisboa Portugal
- 2. Joaquim Jorge Instituto Superior Técnico Universidade de Lisboa Visualization and Multimodal Interfaces @ INESC-ID Lisboa http://web.ist.utl.pt/jorgej ACM Distinguished Member Editor Computers & Graphics UNESCO Chair AI & XR Research Interests: Virtual & Augmented Reality, Multimodal Interfaces, Graphical Modeling
- 3. Catarina Moreira University Technology Sydney Associate Professor XAMI & INESC-ID Co-PI UNESCO Chair AI&XR Co-PI XAVIER Project Research Interests: Explainable AI Data Science Virtual & Augmented Reality,
- 4. Anderson Maciel Graphics and Interaction @INESC-ID Instituto Superior Técnico Universidade de Lisboa Virtual Reality Haptics Interactive Computer Graphics, Computer-Aided Healthcare, Interactive Visualization
- 5. Nuno Figueiredo MD, PhD, FRCS Head of Surgical Center Lusiadas Knowledge Center www.lusiadas.pt Research Interests: Image Guided Surgery Augmented Reality Computer-Aided Healthcare,
- 6. Introduction and Motivation Introduction and Motivation
- 8. Reality–Virtuality Continuum Real Reality Immersive VR S e m i - i m m e r s i v e V R D i m i n i s h e d R e a l i t y S p a t i a l A R S e e - T h r o u g h A R A u g m e n t e d V i r t u a l i t y Mixed Reality Augmented Reality Extended Reality
- 9. Challenges Building a network of healthcare professionals Identifying current practices Linguistic Barrier Big Picture but no Specs “Automagic worshippers”
- 11. Motivation
- 12. Motivation
- 13. Extend the Reality: Learning
- 17. VRRRRoom: Virtual Reality for Radiologists in the Reading Room CHI 2017
- 18. VR for Radiologists in the Reading ROOM
- 19. VR for Radiologists in the Reading Room
- 20. VR for Radiologists in the Reading Room
- 21. VR for Radiologists in the Reading Room https://www.youtube.com/watch?v=bsNwV4BhJOw
- 22. VR 4 Radiologists in the Reading ROOM Evaluation Radiology Neuroradiology Gynecology Surgery Dental Implantology Obstetrics Medical Specialities 5 no previous experience in VR
- 25. Problem
- 26. Problematic
- 27. Interaction Techniques for Immersive CT Colonography: A Professional Assessment MICCAI / VCBM 2018
- 29. Professional Assessment 5M+2F Doctors Diagnostic: Minutes << HOURS!
- 30. Evaluation + immersion + freedom of movement + World In Miniature navigation - limited camera control - Improvements needed
- 35. New design :VR Features: Navigation Minimap Controls Speed control Bookmarking
- 36. New design :VR Features: Navigation Minimap Controls Speed control Bookmarking Measurement
- 37. New design :VR Features: Navigation Minimap Controls Speed control Bookmarking Measurement Coverage: eye tracking
- 38. New design :VR Features: Navigation Minimap Controls Speed control Bookmarking Measurement • Coverage: eye tracking
- 39. New design :VR Features: Navigation Minimap Controls Speed control Bookmarking Measurement Coverage: eye tracking Slice view
- 40. New design :VR Features: Navigation Minimap Controls Speed control Bookmarking Measurement Coverage: eye tracking Slice view
- 41. :VR
- 42. Towards Human-Centered AI: What Can Machines Learn from Eye- Tracking Data? Catarina Moreira Human Centred AI / Brisbane, 2022
- 43. Motivation – AI and the search for Intelligence
- 44. Goal: “Simulate each of the different faculties of human intelligence using machines”
- 46. Machine Learning vs. Deep Learning Source: https://www.boomlibrary.com/blog/machine-learning-and-deep-learning-for-audio/
- 47. Can we trust machine learned predictions?
- 48. Machines can make the right predictions, but not for the right reasons!
- 49. Machines can make the right predictions, but not for the right reasons!
- 50. Machines can make the right predictions, but not for the right reasons!
- 51. Machines can make the right predictions, but not for the right reasons!
- 52. Machines can make the right predictions, but not for the right reasons! This is no Atelectasis! It is just Fat!
- 53. Machines can make the right predictions, but not for the right reasons! Sporta et al (2021), Benchmarking saliency methods for chest X-ray interpretation. Medrxiv:2021.02.28.21252634v2
- 54. Machines can make the right predictions, but not for the right reasons! Sporta et al (2021), Benchmarking saliency methods for chest X-ray interpretation. Medrxiv:2021.02.28.21252634v2 Findings: - Saliency methods (machine) perform worse when compared to humans - Errors between machine vs human were larger in pathologies that were smaller in size
- 55. Predictive algorithms widely used in healthcare exhibited racial biases that prevented minority societal groups from receiving extra care. https://www.scientificamerican.com/article/racial-bias-found-in-a-major- health-care-risk-algorithm/
- 56. INPUT classification Explainable AI – An Ideal Scenario for Chest-Xrays Pneumonia
- 57. Explainable AI – An Ideal Scenario for Chest-Xrays INPUT OUTPUT classification Prediction (pneumonia) Explanation
- 58. XAVIER: eXplainable AI & VR for Enhanced Radiology
- 59. Eye Tracking – Different Types of Data
- 60. Eye Tracking Collecting radiologists eye movements can open new research directions in terms of deep learning approaches: Study radiologists’ cognitive maps Investigate fatigue / cognitive load Investigate how radiologists trained in different countries assess these images / Radiologists’ degree of expertise
- 62. Eye Tracking – Estimating Expertise from Eye Patterns Borys and Plechawska-W ó jcik (2017), Eye-Tracking metrics in perception and visual attention research. European Journal of Medical Technologies, (3):11-23.
- 63. Eye Tracking Data Local Annotations
- 64. Eye Tracking Data can also Promote Explainability
- 65. Possibility to Study Radiologists’ Search Patterns
- 66. Less Data Driven -> More Human Focus!
- 67. No Agreement in the Literature Eye Tracking Improves Accuracy in DL Eye Tracking Does Not Improve Accuracy in DL
- 68. Towards a Human-Centered Deep Learning Architecture
- 69. Eye Tracking Data – Noise from Interface Interaction
- 70. Eye Tracking Data – Subjectivity Noise Subjectivity of the diagnosis (5 radiologists labeled the same images differently)
- 71. How we Think Radiologists Look at an Image vs How They Actually Look! What we were expecting to find What we found
- 72. Fixations are correlated with regions of uncertainty Bullets Regions in need for extra protection
- 73. Starting From the Human Side
- 74. The Importance of Clinical Data Interviews with radiologists suggest that they cannot make accurate diagnosis without knowing the patients’ clinical data.
- 75. MDF – Multimodal Dual Fusion Net
- 76. MDF – Multimodal Dual Fusion Net
- 77. Do Radiologists Have a Pattern? NORMAL (healthy subject) Congestive Heart Failure Pneumonia First Gazes in Silence Gazes when Speaking
- 78. Do Radiologists Have a Pattern? NORMAL (healthy subject) Congestive Heart Failure Pneumonia First Gazes in Silence Gazes when Speaking PATTERNS FOUND!
- 79. Thank you to all contributors Mr. Chihcheng (Richard) Hsieh Mr. André Luís Mr. Diogo Alvito PhD candidate at Centre for Data Science @ QUT MSc at Instituto Superior Técnico MSc at Instituto Superior Técnico
- 80. Augmented Dissection Table Augmented Dissection Table
- 83. Anatomy Studio: Augmented Virtual Diss
- 87. Anatomy Studio: Augmented Virtual Dissection Sessions
- 89. P 0 P 2 P 4 P 1 P 3 P 5 P 0 R0 R2 R1 R2 R0 P 0 P 2 P 4 P 1 P 3 P 5 P 0 R0 R2 R1 R2 R0
- 92. Anatomy Studio: Augmented Virtual Dissection Sessions
- 93. Anatomy Studio: Augmented Virtual Dissection Sessions Usability Testing 10 participants (1 female), 8 medical professionals 2 students
- 94. Anatomy Studio: Augmented Virtual Dissection Sessions Interactions Tablet MR Preview MR Exploration MR Interaction Collaboration
- 95. Healthcare Scenarios UNESCO Chair: Digital Anatomy Université Paris Descartes Unité de recherche en Développement, Imagerie et Anatomie - Prof. Vincent Delmas et Dr. J.F Uhl
- 96. Springer Book Digital Anatomy Jean-François Uhl Daniel Simões Lopes Joaquim Jorge Pedro F. Campos (Editors) Published in 2021
- 98. XR in the Operating Room
- 100. Source: O’Hara et al., 2014 VOXEL Explorer: Asepsis
- 101. VOXEL Explorer: Simple Gestures
- 102. Novel Spatial Interaction Techniques for Exploring 3D Medical Images 102 VOXEL Explorer: Track your Body
- 103. VOXEL Explorer: Gameboy Metaphor
- 106. Augmented Surgery More than meets the scalpel
- 107. The Promise of Augmented Reality
- 108. Motivation
- 110. Motivation
- 111. Motivation
- 112. Motivation
- 113. Laparoscopic Head Mounted Display
- 115. SURVIVRS Surround Video-Based Virtual Reality for Surgery Guidance
- 116. Motivation Problems Lack of specialized surgeons in remote, low-income regions Robotic Teleoperation not currently an option for open surgery Task shifting/task sharing in LMIC as a last resort– high risks
- 117. How to get specialized critical care where care is not available? Specialized surgeon in developed country Live Link General surgery
- 118. Three Streams Spot camera feed of surgical site Stream of vitals monitor 360o immersive live feed of surgery room
- 119. 360o immersive live feed of surgery room
- 120. Spot camera feed of surgical site
- 121. Stream of vitals monitor
- 122. Prototype Implementation Surgery Room PC Server Surgery Room Monitor VR Application Surgeon side Send live video feed Send annotated video feed 360o camera Spot Camera Vitals data Show annotated video Send recorded videos After closing the call ✓Interacts with video ✓Annotates videos ✓Video is Recorded ✓Audio
- 123. Prototype Implementation • Surgeon annotates on surgical site video. • Annotations are sent to local surgery team • Each annotation is saved • Two most recent are accessible
- 124. Video demo
- 126. Evaluation + found to be highly usable + viewed as potentially beneficial + high sense of presence – evaluation not (yet) done with MDs – minor UX/design issues
- 127. Most recent design iteration
- 133. Next steps Short term test tool with surgeons improve surgery room interface Long term AR in surgery room with real-time fitting of annotations to real organ Closing the telepresence loop by adding AR representation of remote surgeon
- 135. Raskar | MIT Media Lab Advanced UIs XAI IoT Perception Ergo, Hygiene Full-Body Language
- 136. Challenges Ahead
- 139. Disorientation
- 140. Fatigue
- 141. Nausea
- 142. Bright Future Ahead
- 144. Acknowledgments VRRRRoom Maurício Sousa, Daniel Mendes CAVE COLON Pedro Borges, Daniel Medeiros, João Serras Voxel TIPs Rita Mendes, Pedro Parreira, Soraia Paulo Anatomy Studio Maurício Sousa, Daniel Mendes, Rafael Anjos, Soraia Paulo Voxel Explorer Pedro Parreira, Soraia Paulo TOOTHFAIRY Filipe Relvas, Soraia Paulo Laparoscopic HUD Miguel Belo ImplantAR Tiago Jerónimo SURVIVRS Regis Kopper (UNC)
- 145. Special Thanks to Ramesh Raskar, MIT Anders Ynnerman, Linkoping U. Maurício de Sousa, U Toronto Daniel Pires de Sá Medeiros, U Glasgow Daniel Tavares Mendes, U Porto Rafael Kuffner dos Anjos, UCL, London, UK Soraia Figueiredo Paulo, INESC-ID Beatriz Peres, U Madeira, INESC-ID
- 146. THE NEED FOR NEW DESIGN DISCIPLINES
- 148. Design is Hard Society T e c h n o l o g y P e o p l e UI UX UE
- 149. Challenges in New Modalities
- 150. Speech Interfaces 50 years ago 2001 – A Space Odissey 1969
- 152. Errors = Design Opportunities
- 154. Synergistic Parsing
- 155. Machine Learning to the Rescue
- 156. Task Analysis à Learning UM
- 160. Digital Anatomy: Applications of Virtual, Mixed and Augmented Reality Editors: Uhl, J.-F., Jorge, J., Lopes, D.S., Campos, P.F. Springer: Human-Computer Interaction Series, 2021 ISBN 978-3-030-61905-3 Laparoscopy with augmented reality adaptations ER Zorzal, JMC Gomes, M Sousa, P Belchior, PG da Silva, N Figueiredo, DS Lopes, Joaquim Jorge, 2020 Journal of biomedical informatics 107, 103463 Camera Travel for Immersive Colonography SF Paulo, D Medeiros, P Borges, J Jorge, DS Lopes arXiv preprint arXiv:2010.07798 2020 Improving Camera Travel for Immersive Colonography SF Paulo, D Medeiros, PB Borges, J Jorge, DS Lopes 2020 IEEE Conference on Virtual Reality and 3D User Interfaces
- 161. Soraia Figueiredo Paulo, Filipe Relvas, Hugo Nicolau, Yosra Rekik, Vanessa Machado, João Botelho, José João Mendes, Laurent Grisoni, Joaquim Jorge, Daniel Simões Lopes, Touchless interaction with medical images based on 3D hand cursors supported by single-foot input: A case study in dentistry, Journal of Biomedical Informatics, Volume 100, 2019, 103316, DOI: 10.1016/j.jbi.2019.103316 Daniel Simões Lopes, Filipe Relvas, Soraia Paulo, Yosra Rekik, Laurent Grisoni, Joaquim Jorge. 2019. FEETICHE: FEET Input for Contactless Hand gEsture Interaction. In The 17th International Conference on Virtual Reality Continuum and its Applications in Industry (VRCAI ’19), November14–16, 2019, Brisbane, QLD, Australia. ACM, New York, NY, USA, 10 pages. DOI: 10.1145/3359997.3365704 Ezequiel R. Zorzal, Maurício Sousa, Daniel Mendes, Rafael Kuffner dos Anjos, Daniel Medeiros, Soraia Figueiredo Paulo, Pedro Rodrigues, José João Mendes, Vincent Delmas, Jean-Francois Uhl, José Mogorrón, Joaquim Armando Jorge, Daniel Simões Lopes, Anatomy Studio: a Tool for Virtual Dissection Through Augmented 3D Reconstruction, Computers & Graphics, DOI: 10.1016/j.cag.2019.09.006
- 162. D.S. Lopes, D. Medeiros, S.F. Paulo, P.B. Borges, V. Nunes, V. Mascarenhas, M. Veiga, J.A. Jorge, Interaction Techniques for Immersive CT Colonography: A Professional Assessment, In: Frangi A., Schnabel J., Davatzikos C., Alberola-López C., Fichtinger G. (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI 2018. MICCAI 2018. Lecture Notes in Computer Science, vol 11071, Pages 629–637, Springer, Cham, 2018. DOI: 10.1007/978-3-030-00934-2_70 M. Sousa, D. Mendes, S. Paulo, N. Matela, J. Jorge, D.S. Lopes, VRRRRoom: Virtual Reality for radiologists in the reading room, Proceedings of the 35th Annual ACM Conference on Human Factors in Computing Systems (CHI 2017), New York: ACM Press, 2017. DOI: 10.1145/3025453.3025566 D.S. Lopes, P.F. Parreira, S.F. Paulo, V. Nunes, P.A. Rego, M.C. Neves, P.S. Rodrigues, J.A. Jorge, On the utility of 3D hand cursors to explore medical volume datasets with a touchless interface, Journal of Biomedical Informatics, 72, Pages 140–149, 2017. DOI: 10.1016/j.jbi.2017.07.009 Daniel Simões Lopes, Pedro F. Parreira, Ana R. Mendes, Vasco M. Pires, Soraia F. Paulo, Carlos Sousa, Joaquim A. Jorge, Explicit design of transfer functions for volume-rendered images by combining histograms, thumbnails, and sketch-based interaction, The Visual Computer, December 2018, Volume 34, Issue 12, pp 1713–1723. DOI: 10.1007/s00371-017-1448-8
- 163. R Tang, XD Yang, S Bateman, J Jorge, A Tang, Physio@ Home: Exploring visual guidance and feedback techniques for physiotherapy exercises, ACM SIGCHI 2015 João Vieira, Maurício Sousa, Artur Arsénio and Joaquim Jorge SleeveAR: Augmented Reality for Rehabilitation using Realtime Feedback, ACM IUI 2016 Beatriz Peres, Pedro F. Campos, Aida Azadegan, A Persuasive Approach in Using Visual Cues to Facilitate Mobility Using Forearm Crutches. BCSS@PERSUASIVE 2019 Beatriz Peres, Pedro F. Campos, Aida Azadegan , Digitally Augmenting the Physical Ground Space with Timed Visual Cues for Crutch-Assisted Walking. CHI Extended Abstracts 2019 Beatriz Peres, Pedro F. Campos, Aida Azadegan , A Digitally-Augmented Ground Space with Timed Visual Cues for Facilitating Forearm Crutches’ Mobility, Full paper , Interact 2019 .
- 164. Thank You!