Skip to main content
UploadLanguage (EN)Support
BusinessMobileSocial MediaMarketingTechnologyArt & PhotosCareerDesignEducationPresentations & Public SpeakingGovernment & NonprofitHealthcareInternetLawLeadership & ManagementAutomotiveEngineeringSoftwareRecruiting & HRRetailSalesServicesScienceSmall Business & EntrepreneurshipFoodEnvironmentEconomy & FinanceData & AnalyticsInvestor RelationsSportsSpiritualNews & PoliticsTravelSelf ImprovementReal EstateEntertainment & HumorHealth & MedicineDevices & HardwareLifestyle
Mark Billinghurst, profile picture
Uploaded byMark Billinghurst
138 views

2026 BrainLink Presentaton - Empathic Computing: Designing Human-Centric Spatial AI

Invited talk given by Mark Billinghurst at the 2026 BrainLink workshop on Omnimodal AI-XR Convergence for Future Intelligent Spaces: Cognition, Empathy, and Augmented Interaction. The talk was titled Empathic Computing: Designing Human-Centric Spatial AI, and discussed how omnimodal AI could be used to enhance communication. This talk was given on August 9th, 2026.

Technology◦
Related topics:
Insights on Augmented Reality•
Download to read offline
Mark Billinghurst mark.billinghurst@adelaide.edu.au August 9th 2026 Empathic Computing: Designing Human-Centric Spatial AI Adelaide University
“Only through communication can Human Life hold meaning.” Paulo Freire
Modern Communication Technology Trends 1. Improved Content Capture • Move from sharing faces to sharing places 2. Increased Network Bandwidth • Sharing natural communication cues 3. Implicit Understanding • Recognizing behaviour and emotion
Natural Collaboration Experience Capture Implicit Understanding Empathic Computing
Empathic Computing Research Focus Can we develop systems that allow us to share what we are seeing, hearing and feeling with others?
Key Elements of Empathic Systems •Understanding • Emotion Recognition •Experiencing • Content/Environment capture •Sharing • Communication cues Physiological sensors Virtual Reality Augmented Reality
Live 3D Scene Capture and Sharing (2020) Scene Reconstruction Remote Expert Local Worker Bai, H., …& Billinghurst, M. (2020). A user study on mixed reality remote collaboration with eye gaze and hand gesture sharing. In Proceedings of the 2020 CHI conference (pp. 1-13).
AR View Remote Expert View
Empathic Aurea (2022) • Person in video see-through HMD (‘decoder’) looks at user wearing ECG (‘encoder’) • Decoder tries to guess emotional state of the encoder
Sharing Gaze Cues (2021) How sharing gaze behavioural cues can improve remote collaboration in Mixed Reality environment. ➔ Developed eyemR-Vis, a 360 panoramic Mixed Reality remote collaboration system ➔ Showed gaze behavioural cues as bi-directional spatial virtual visualisations shared between a local host (AR) and a remote collaborator (VR). Jing, A., May, K. W., Naeem, M., Lee, G., & Billinghurst, M. (2021). eyemR-Vis: Using Bi-Directional Gaze Behavioural Cues to Improve Mixed Reality Remote Collaboration. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-7).
Gaze Representation Browse Focus Mutual Fixed Circle-map
Natural Collaboration Experience Capture Implicit Understanding Empathic Computing AI
Omni-modal AI • What is omnimodal AI? • Processes images, sounds, videos, and physical environments simultaneously • An intelligence that can truly interact with its environment. • Trend to Omnimodal AI • Training on text (LLMs) - acquire deep linguistic knowledge • Integrating multiple senses (multi-modality) – train on each modality separately, fuse later • Learning all modes simultaneously (omni-modality) - Simultaneous learning, aligning all sensory inputs within a shared coordinate system.
Empathic Computing and Omnimodal AI •Understanding • Personal Context •Experiencing • User Space •Sharing • Communication Onmimodal AI Empathic Communication
Integration of Omnimodal AI with Empathic Computing to create a novel approach for remote collaboration, Empathic Communication, where AI facilitates human connection and synchronisation by operating on both the content cues and relational process of communication. Research Vision
• Current remote conferencing • Limited sharing of conscious cues (speech, gesture, face expression, etc.) • No sharing of unconscious cues (micro-expressions, pupil dilation, etc.) • Poor process feedback (Grounding/Synchronisation) Current Conferencing
• AI enhanced remote conferencing • Using AI to recognize communication cues/emotional state • AI generates communication cues, representing unconscious cues • AI facilitates Grounding and Synchronisation AI Enhanced Empathic Communication
PPG GSR Video Audio Multimodal Input (3 Participants) Signal Processing and Feature Extraction Cognitive State and Affect State Prediction - ML Model Cognitive and Affective Module LLM - Meta’s LLAMA3.1 Speech to Text Conversation Module Group Awareness Module Digital Human Agent Control Group Aware Digital Human Video Conference Meeting Behavior Module Early Demo (2026) • Adding support for multimodal emotion recognition • Existing system for meeting support/facilitation • Detect cognitive load, affect, and adapt character feedback
Empathic Mixed Reality Agents
Empathic Mixed Reality Agent (2020) Chang, Z., Bai, H., Zhang, L., Gupta, K., He, W., & Billinghurst, M. (2022). The impact of virtual agents’ multimodal communication on brain activity and cognitive load in Virtual Reality. Frontiers in Virtual Reality, 3, 179.
Tactile Touch with Mixed Reality Agents • Using vibrotactile pad with MR agent • Feel touch on forearm • Desert survival task • Agent recommends objects
Experiment • Character provides recommendations • Touch/No touch conditions • Subjective feedback
Technology Trends • Advanced displays • Wide FOV, high resolution, light • Real time space capture • 3D scanning, stitching, gaussian splats • Natural gesture interaction • Hand tracking, pose recognition • Robust eye-tracking • Gaze points, focus depth • Emotion sensing/sharing • Physiological sensing, emotion mapping • Artificial Intelligence • Semantic understanding, AR agents
HMDs with Integrated Sensing • Latest HMDs support physiological sensing • Gaze tracking, heart rate, brain activity, etc.. • Enable capture and sharing implicit cues • Cognitive load, emotion, stress, etc.. • Many possible application • Remote collaboration • Enhanced training • Gaming Galea, Open BCI Cognixion
Empathic Shared MR Experiences (2020) • Using brain synchronicity to increase connection • Collaborative VR drumming experience • Measure brain activity using 3 EEG electrodes • Use PLV to calculate synchronization • More synchronization increases graphics effects/immersion Pai, Y. S., Hajika, R., Gupta, K., Sasikumar, P., & Billinghurst, M. (2020). NeuralDrum: Perceiving Brain Synchronicity in XR Drumming. In SIGGRAPH Asia 2020 Technical Communications (pp. 1-4).
Results "It’s quite interesting, I actually felt like my body was exchanged with my partner." Poor Player Good Player
• Measure physiological cues • Brain activity • Heart rate • Eye gaze • Show user state • Cognitive load • Attention Showing Cognitive Load in Collaboration (2024) Sasikumar, P.,... & Billinghurst, M. (2024). A user study on sharing physiological cues in vr assembly tasks. In 2024 IEEE VRlity (pp. 765-773).
Demo
Key Research Directions • Scene capture and understanding • Semantic and context recognition • Emotion/Cognitive State Recognition/Sharing • Reliable estimate from multimodal sources • AI enhanced communication • Empathic communication • Training Omnimodal AI for communication • How to build communication experts • Agent representation • How to represent omnimodal AI agents • Privacy and Security • How to keep secrets
Conclusions • Empathic Computing • Creating systems that increase understanding • Key Aspects of Empathic Computing • Scene capture, Emotion recognition, Sharing communication cues • Empathic Computing + Onmimodal AI • Empathic Communication • Many Opportunities for Research • Emotion recognition, Agent Representation, Omnimodal training, etc.
Empathic Computing Journal • Looking for submissions • Any topic relevant to Empathic Computing • Open Access, free to publish currently
www.empathiccomputing.org @marknb00 mark.billinghurst@adelaide.edu.au