Empathic Computing Research for the Broader Metaverse
•
0 likes•263 views
1) The document discusses the concept of empathic computing and its application to designing for the broader metaverse. 2) Empathic computing aims to develop systems that allow people to share what they are seeing, hearing, and feeling with others through technologies like augmented reality, virtual reality, and physiological sensors. 3) Potential research directions are explored, like using lifelogging data in VR, bringing elements of the real world into VR, and developing systems like "Mini-Me" avatars that can convey non-verbal communication cues to facilitate remote collaboration.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Empathic Computing Research for the Broader Metaverse
Similar to Empathic Computing Research for the Broader Metaverse (20)
Recently uploaded
Recently uploaded (20)
Empathic Computing Research for the Broader Metaverse
- 1. EMPATHIC COMPUTING Mark Billinghurst mark.billinghurst@auckland.ac.nz April 23rd 2023 Designing for the Broader Metaverse
- 2. The Metaverse is Hot!
- 3. Google Searches on the Metaverse • Jan – Dec 2021 • Metaverse (blue) vs. Virtual Reality (red)
- 4. Publications about the Metaverse • Metaverse in the abstract • 2021 – 46, 2022 – 1065, 2023 – 422 so far
- 5. But what is the Metaverse? • Real-time, 3D, Interactive, Social, Persistent AWE 2022 John Riccitiello CEO, Unity Technologies
- 7. The Broader Metaverse • Neal Stephenson’s “Snow Crash” (1992) • The Metaverse is the convergence of: • 1) virtually enhanced physical reality • 2) physically persistent virtual space
- 8. Metaverse Taxonomy • Four Key Components • Virtual Worlds • Augmented Reality • Mirror Worlds • Lifelogging • Metaverse Roadmap • http://metaverseroadmap.org/
- 9. Mirror Worlds • Simulations of external space/content • Capturing and sharing surroundings • Photorealistic content • Digital twins Matterport Deep Mirror Google Street View Soul Machines
- 10. Lifelogging • Measuring user’s internal state • Capturing physiological cues • Recording everyday life • Augmenting humans Apple Fitbit Shimmer OpenBCI
- 13. Possible Research Directions • Lifelogging to VR • Bringing real world actions into VR, VR to experience lifelogging data • AR to Lifelogging • Using AR to view lifelogging data in everyday life, Sharing physiological data • Mirror Worlds to VR • VR copy of the real world, Mirroring real world collaboration in VR • AR to Mirror Worlds • Visualizing the past in place, Asymmetric collaboration • And more..
- 14. Modern Communication Trends 1. Improved Content Capture • Move from sharing faces to sharing places 2. Increased Network Bandwidth • Sharing natural communication cues 3. Implicit Understanding • Recognizing behaviour and emotion
- 17. Empathic Computing Research Focus Can we develop systems that allow us to share what we are seeing, hearing and feeling with others?
- 18. Key Elements of Empathic Systems •Understanding • Emotion Recognition, physiological sensors •Experiencing • Content/Environment capture, VR •Sharing • Communication cues, AR
- 19. Example: Sharing Communication Cues • Measuring non-verbal cues • Gaze, face expression, heart rate • Sharing in Augmented Reality • Collaborative AR experiences
- 20. Empathy Glasses • Combine together eye-tracking, display, face expression • Implicit cues – eye gaze, face expression + + Pupil Labs Epson BT-200 AffectiveWear Masai, K., Sugimoto, M., Kunze, K., & Billinghurst, M. (2016, May). Empathy Glasses. In Proceedings of the 34th Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems. ACM.
- 21. Remote Collaboration • Eye gaze pointer and remote pointing • Face expression display • Implicit cues for remote collaboration
- 23. Example: Connecting between Spaces • Augmented Reality • Bringing remote people into your real space • Virtual Reality • Bringing elements of the real world into VR • AR/VR for sharing communication cues • Sharing non-verbal communication cues Piumsomboon, T., Dey, A., Ens, B., Lee, G., & Billinghurst, M. (2019). The effects of sharing awareness cues in collaborative mixed reality. Frontiers in Robotics and AI, 6, 5.
- 24. Sharing Virtual Communication Cues • AR/VR displays • Gesture input (Leap Motion) • Room scale tracking • Conditions • Baseline, FoV, Head-gaze, Eye-gaze
- 26. Sharing: Communication Cues (2018) • What happens when you can’t see your colleague/agent? Piumsomboon, T., Lee, G. A., Hart, J. D., Ens, B., Lindeman, R. W., Thomas, B. H., & Billinghurst, M. (2018, April). Mini-me: An adaptive avatar for mixed reality remote collaboration. In Proceedings of the 2018 CHI conference on human factors in computing systems (pp. 1-13). Collaborating Collaborator out of View
- 27. Mini-Me Communication Cues in MR • When lose sight of collaborator a Mini-Me avatar appears • Miniature avatar in real world • Mini-Me points to shared objects, show communication cues • Redirected gaze, gestures
- 29. Example: Sensor Input into AR/VR • Augmented Reality • Bringing remote people into your real space • Virtual Reality • Bringing elements of the real world into VR • AR/VR for sharing communication cues • Sharing non-verbal communication cues
- 30. NeuralDrum • Using brain synchronicity to increase connection • Collaborative VR drumming experience • Measure brain activity using 3 EEG electrodes • Use PLV to calculate synchronization • More synchronization increases graphics effects/immersion Pai, Y. S., Hajika, R., Gupta, K., Sasikumar, P., & Billinghurst, M. (2020). NeuralDrum: Perceiving Brain Synchronicity in XR Drumming. In SIGGRAPH Asia 2020 Technical Communications (pp. 1-4).
- 31. Set Up • HTC Vive HMD • OpenBCI • 3 EEG electrodes
- 33. Results "It’s quite interesting, I actually felt like my body was exchanged with my partner." Poor Player Good Player
- 35. MiniMe Virtual Cues Enhanced Emotion Brain Synchronization Emotion Recognition Scene Capture AI
- 36. Accessibility and the Broader Metaverse • Combine elements from different quadrants • Augmenting, Immersing, Capturing, Sensing • Example Projects • Sharing emotions in VR • Using VR for remote TBI therapy • Intelligent agents for depression support • Haptics for remote emotion sharing • MiniMe agents for communication cue sharing
- 37. Non-visual Metaverse (2018) • Using haptics/audio for remote collaboration • Example: DualPanto (2018) • Haptic only VR experiences • Designed for blind interaction Schneider, O., Shigeyama,... & Baudisch, P. (2018, October). DualPanto: a haptic device that enables blind users to continuously interact with virtual worlds. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (pp. 877-887).
- 38. Empathic AuRea (2022) • Person in video see-through HMD (‘decoder’) looks at user wearing ECG (‘encoder’) • AR aura showing emotional state of the encoder Valente, A., Lopes, D. S., Nunes, N., & Esteves, A. (2022, March). Empathic AuRea: Exploring the Effects of an Augmented Reality Cue for Emotional Sharing Across Three Face-to-Face Tasks. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR) (pp. 158-166). IEEE.
- 39. Delivering the Entire Metaverse
- 40. Empathic Tele-Existence • Based on Empathic Computing • Creating shared understanding • Covering the entire Metaverse • AR, VR, Lifelogging, Mirror Worlds • Transforming collaboration • Observer to participant • Feeling of doing things together • Supporting Implicit collaboration
- 41. Google Searches on the Metaverse • Jan – Dec 2021 • Metaverse (blue) vs. Virtual Reality (red)
- 42. Searches to April 2023 Dec 2021 Apr 2021
- 44. VR Winter (2002-2013) • April 2007 Computer World • VRVoted 7th on list of 21 biggest flops • MS Bob #1
- 45. What Happened After That • Growth of internet • Natural user interfaces • Increase in graphics performance • Explosion in mobile phones • Pokemon Go!
- 46. Increase in VR Research • Papers grew over fivefold..
- 47. AR/VR Growth • 171 million VR user worldwide • > 750 million MAU Snap AR Snap AR DAU
- 48. Lessons we can Learn • Take advantage of new technology available • Eye tracking, physiological sensing, HMDs, etc • Significant decrease in costs • Design for real needs • Increasing number of people with accessibility needs • Growing elderly, health issues, etc • Focus on the research • More people with skills available to do research • Especially multi-disciplinary researchers
- 49. Conclusions • Broader Metaverse • Augmenting, Immersing, Capturing, Sensing • Empathic Computing • Systems that enhance understanding • Combining AR, VR, Physiological sensors • Opportunities for Research • Accessibility, Empathic Tele-Existence • Take advantage of the Metaverse winter • Design for the whole metaverse