Downloaded 23 times
Uploaded byMark Billinghurst
Reach Out and Touch Someone: Haptics and Empathic Computing
Keynote speech given by Mark Billinghurst at the 2025 IEEE World Haptics Conference. This talk was given on July 10th 2025. It describes the use of haptic interfaces for remote touch and social collaboration.
More Related Content
Similar to Reach Out and Touch Someone: Haptics and Empathic Computing
Reach Out and Touch Someone: Haptics and Empathic Computing
- 1. REACH OUT ANDTOUCH SOMEONE Mark Billinghurst mark.billinghurst@unisa.edu.au IEEE Haptics 2025 July 10th 2025 HAPTICS AND EMPATHIC COMPUTING
- 3. “Only through communication can HumanLife hold meaning.” Paulo Freire
- 5. Modern Communication TechnologyTrends 1. Improved Content Capture • Move from sharing faces to sharing places 2. Increased Network Bandwidth • Sharing natural communication cues 3. Implicit Understanding • Recognizing behaviour and emotion
- 6.
- 7. “Empathy is Seeingwith the Eyes of another, Listening with the Ears of another, and Feeling with the Heart of another..” Alfred Adler
- 8. Empathic Computing ResearchFocus Can we develop systems that allow us to share what we are seeing, hearing and feeling with others?
- 9. Key Elements ofEmpathic Systems •Understanding • Emotion Recognition •Experiencing • Content/Environment capture •Sharing • Communication cues Physiological sensors Virtual Reality Augmented Reality
- 10. Example: Empathy Glasses •Combine together eye-tracking, display, face expression • Implicit cues – eye gaze, face expression + + Pupil Labs Epson BT-200 AffectiveWear Masai, K., Sugimoto, M., Kunze, K., & Billinghurst, M. (2016, May). Empathy Glasses. In Proceedings of the 34th Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems. ACM.
- 11. AffectiveWear – EmotionSensing Glasses • Photo sensors to recognize expression • User calibration • Machine learning • Recognizing 8 face expressions
- 12. Remote Collaboration • Eyegaze pointer and remote pointing • Face expression display • Implicit cues for remote collaboration
- 14. Shared Sphere –360 Video Sharing Shared Live 360 Video Host User Guest User Lee, G. A., Teo, T., Kim, S., & Billinghurst, M. (2017). Mixed reality collaboration through sharing a live panorama. In SIGGRAPH Asia 2017 Mobile Graphics & Interactive Applications (pp. 1-4).
- 16. AR/VR for RemoteCollaboration
- 17.
- 18. The Importance ofTouch “Touch is a fundamental language of human connection” Dacher Keltner "After years spent immersed in the science of touch, I can tell you that they are far more profound than we usually realize: They are our primary language of compassion..”
- 19. Research on Touchfor Remote Communication • Papers in Scopus • Haptics – 11,158 • Haptics AND AR/VR – 3,290 (30%) • Haptics AND AR/VR AND Communication – 329 (3%) • Haptics AND AR/VR AND Remote Communication - 5 (<0.05%) • Papers in IEEE Haptics 2025 ~3-4% • All Long/Regular papers 3/75 • WIP 4/144 • Demos 1/70
- 20. Research Topics • Devicesfor remote touch • Wearable, lightweight, responsive • Using haptics to convey touch from agents • AI agent giving touch • Sharing physiological cues • Haptics for sharing heart rate • Using haptics to modulate heart rate • Interoception • Using haptics to trigger memories • Haptics + emotion recognition
- 21.
- 22. Example Use: HapticsVR Painting Hardware Motion-Coupled Haptic Rendering VR Art Yuan, R., Tang, A., Zou, Q., Mahmoudinezhad, M. H., Zhang, Y., & Anderson, I. (2024). Finger Painting in VR: Multi-Dynamic Gestural Input for VR Painting. In SIGGRAPH Asia 2024 XR (pp. 1-2).
- 24. Conversational agent Intelligent VirtualAgents (IVAs) Embodied in 2D Screen Embodied in 3D space
- 25. Empathic Mixed RealityAgent Chang, Z., Bai, H., Zhang, L., Gupta, K., He, W., & Billinghurst, M. (2022). The impact of virtual agents’ multimodal communication on brain activity and cognitive load in Virtual Reality. Frontiers in Virtual Reality, 3, 179.
- 26. Empathic Mixed RealityAgents
- 27. Haptic Controller Electronics Capacitance Samplingboard 5 Channel Piezo Driver 10 Channel LRA Driver
- 28. Tactile Touch withMixed Reality Agents • Using vibrotactile pad with MR agent • Feel touch on forearm • Desert survival task • Agent recommends objects
- 29. Experiment • Character provides recommendations •Touch/No touch conditions • Subjective feedback
- 30. Results • Agents withvisual-tactile touch preferred and perceived as more realistic • Physiological data showed no differences, so touch did not elicit notable physiological response
- 31. Sharing Physiological Cues •Does sharing heart rate in VR create connection? Dey, A., Piumsomboon, T., Lee, Y., & Billinghurst, M. (2017). Effects of sharing physiological states of players in a collaborative virtual reality gameplay. In Proceedings of CHI 2017 (pp. 4045-4056).
- 32. Sharing Heart Ratein VR • HTC Vive HMD • Heart rate sensor • Empatica E4
- 33. VR Environments • ButterflyWorld: calm scene, collect butterflies • Zombie Attack: scary scene, fighting zombies
- 35. Experiment Design • KeyQuestion • What is the impact of sharing heart rate feedback? • Two Independent Variables • Game Experience (Zombies vs butterflies) • Heart Rate Feedback (On/Off) • Measures • Heart rate (player) • PANAS Scale (Emotion) • Inclusion of other in self scale (Connection)
- 36. Results • Results • Significantdifference in Heart Rate • Sharing HR improves positive affect (PANAS) • Sharing HR created subjective connection between collaborators Heart Rate Data Likert Data
- 37. Viewing Heart Ratein VR • How should heart rate be represented? • Audio, visual, haptic cues? Chen, H., Dey, A., Billinghurst, M., & Lindeman, R. W. (2017). Exploring the design space for multi-sensory heart rate feedback in immersive virtual reality. In Proceedings of the 29th Australian conference on computer-human interaction (pp. 108-116).
- 38. VR Experience • VRsafari scene designed to create five emotions • Happiness, anxiety, fear, disgust, and sadness • Provide representation of heart rate • None, Audio-Visual, Visio-Haptic, Audio-Haptic, and Audio-Visual-Haptic.
- 40. Experiment • Within subjectsstudy • 20 participants (5 female), average 32 years old • Experience 5 scenes in counter balanced order • Measures • SAM, PANAS subjective scale • Ranking questions • Likert questions • Interviews
- 41. Results • Audio-haptic rankedbest • Heart rate significantly increased in VR • Visual feedback was labelled as distracting • Participants appreciated hearing their own heart rate
- 42. Sharing Heart Ratein VR • Does sharing HR using haptics improve collaboration/connection? Dey, A., Chen, H., Zhuang, C., Billinghurst, M., & Lindeman, R. W. (2018, October). Effects of sharing real-time multi-sensory heart rate feedback in different immersive collaborative virtual environments. In 2018 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) (pp. 165-173). IEEE.
- 43. Sharing HR • SharingHeart Rate in 3 different environments • Escape room, Exploration, Furniture arrangement • HR represented using Audio-Haptic cues HR on/off
- 44. Results • Sharing HRfeedback improved feeling the other person’s presence • Sharing HR feedback improved perceived feeling of emotional state • The type of task had a significant effect on results Co-presence Emotional State “. . . it is great to feel my collaborators heart rate ... makes me feel I am not alone!”
- 45. Manipulating Heart Rate •What happens if your share a fake heart rate? Dey, A., Chen, H., Billinghurst, M., & Lindeman, R. W. (2018, October). Effects of manipulating physiological feedback in immersive virtual environments. In Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play (pp. 101-111).
- 46. Manipulating Heart Rate •What is effect of manipulated HR in VR? • 20 subjects (within subjects design) • Stimuli • 5 VR scenes – jungle safari • Audio-haptic cues showing HR • HR Manipulations • Capture HR (Polar H7) • -30%, -15%, 0%, +15%, 30% • Measures • PANAS scale, SAM, HR, GSR, Survey
- 47. Results • Significant effectof HR manipulation on emotions • Significant effect on recorded excitement, scariness • No effect of HR manipulation on physiological cues • Manipulating HR • Increased interest, excitement, scariness, nervousness, and fear • Participants noticed large manipulations (+30%)
- 48. Modulating Heart Rate •Can haptics affect your heart rate? • Where should haptic feedback be placed? Valente, A., Lee, D., Choi, S., Billinghurst, M., & Esteves, A. (2024, October). Modulating Heart Activity and Task Performance using Haptic Heartbeat Feedback: A Study Across Four Body Placements. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (pp. 1-13).
- 49.
- 50. Changing Heart Rate •Place audio transducer at four different locations • chest, wrist, ankle, and neck • Pulse at two different frequencies • 50 bpm, 110 bpm • Measure feedback while under cognitive load • N-back task Audio signal (50 bpm and 110 bpm) Frequency: 80Hz
- 51. Results • 20 participantsx 12 trials • High frequencies were able to increase heart rate • Significant effect of haptic placement in participants’ heart rate/HRV • Participants found the chest location more enjoyable and reduced anxiety • Wrist placement showed no significant difference compared to baseline
- 52. ReTouch: Enhancing MemoryRecall • Using emotionally adaptive VR and haptic feedback • Aims to explore impact of haptics on emotional and physiological responses Gunasekaran, T. S., Ju, Y., Barbareschi, G., Minamizawa, K., Pai, Y. S., & Billinghurst, M. (2024). Re-Touch: A VR Experience for Enhancing Autobiographical Memory Recall Through Haptic and Affective Feedback. In SIGGRAPH Asia 2024 XR (pp. 1-2).
- 54. Three Step Process •1) Calibration: users are presented with emotionally evocative scenarios (happy or horror) to elicit strong emotional responses (PPG, EDA) and train machine learning models to predict arousal levels. • 2) Memory Collection: Users describe memories, converted into images, 3D VR scenes and vibrational feedback, adjusted to enhance emotional impact. • 3) Interactive Exploration: Users explore VR environment with tactile sensation and high arousal levels trigger additional content.
- 55.
- 56. Research Opportunities • Advanceddisplays • Wide FOV, high resolution • Real time space capture • 3D scanning, stitching, segmentation • Natural gesture interaction • Hand tracking, pose recognition • Robust eye-tracking • Gaze points, focus depth • Emotion sensing/sharing • Physiological sensing, emotion mapping
- 57. Multiple Physiological Sensorsinto HMD - Galea • Incorporate range of sensors on HMD faceplate and over head • EMG – muscle movement • EOG – Eye movement • EEG – Brain activity • EDA, PPG – Heart rate
- 59. • Measure physiologicalcues • Brain activity • Heart rate • Eye gaze • Show user state • Cognitive load • Attention Showing Cognitive Load in Collaboration
- 60.
- 61. User Study • Aim •How visual cues of physiological state affect collaboration and awareness • Task (28 people/ 14 pairs) • Motorbike repair • Different levels of complexity • Found • Users had a preference for monitoring their partner’s attentional state, • but paid little attention to physiological cues and unsure of how to interpret % of time looking at physiological cues User preference ranking
- 62. Research Challenges • Representingemotion using haptics • Red = angry, which vibration = sad? • Multimodal representation • Combination visual/audio/haptic cues • Representing physiological cues • Cognitive load, sweating, etc. • Causing emotional states • Interoception • Integrating Haptics with other senses • Emotion recognition/physiological sensing
- 63. Conclusions • Empathic Computing •Combines AR, VR, Sensing for improved connection • Little research using Haptics for remote connection • < 5% published papers • Significant research opportunities • Devices • Conveying touch • Emotion sharing • Emotion Representation • Combining with physiological sensors • Looking for collaborators • Interns, visiting researchers, joint projects, etc..
- 65.