This document describes research into using spatial audio and visual cues to enhance remote collaboration. It discusses prior work exploring spatialized audio and video calls. Two user studies were conducted to test how spatialized voice, auditory beacons, hand gestures, and seeing the remote helper's field of view impacted collaboration on an object search task. The results showed that spatial audio cues helped guide the task and increased social presence, while adding visual cues further improved performance and workload. The research demonstrates the potential for spatial audio and visuals to augment remote collaboration.

1. UnConference 2019
1
25th October 2019
Prasanth Sasikumar
Augmented Human Lab

2. What is Remote
Collaboration?
Remote collaboration refers to the
ability to engage as a team from
anywhere in the world.
2
Image Source: https://www.viewsonic.com

3. 3
Why Remote Collaboration?
● Productivity
● Workforce Diversity
● Reduces cost for workers
● Helps close gender gap in Tech
● Reduces Stress
● Prolong careers
● Benefits the environment
● Increases talent pool
● Employee Retention

4. Why do we need
better remote
Collaboration?
According to Saleforce, 86% of
employees and executives cite lack
of collaboration or ineffective
communication for workplace
failures.
4
Image Source: https://www.viewsonic.com
https://blog.bit.ai/collaboration-statistics/

5. Multimodal input in Remote
Collaboration with Dense Scene
Reconstruction
Prasanth Sasikumar
5
Image Source: CathyHackl

6. Multimodal Input
6
Dense Scene
Reconstruction
● Physiological Sensors :
EEG, EOG, EDA, ECG, EMG,BVP
● Emotion Detection
● Emotion Representation
● Scene Reconstruction:
Real Time 3D reconstruction
3D Representation
● Depth Cameras
● Portable remote collaboration
solutions

7. 7
Previous Work
● Spatial Audio Navigation for large scale remote
collaboration
● Augmented Collaborative Application for car
showroom
● 3D Dense reconstruction using Depth cameras
● Comparing device and natural communication
cues in remote collaboration
● Hyperscanning in VR

8. The Effect of Spatial Auditory
and Visual Cues in Mixed Reality
Remote Collaboration
Jing Yang, Prasanth Sasikumar, Huidong Bai, Mark Billinghurst
8
Image Source: CathyHackl

9. Introduction
9
While visual cues have been
investigated in several collaborative
MR systems, spatial auditory
cues remain underexplored
Image Source: Techcrunch

10. 10
face-to-face
communication
Most people still prefer face-to-face communication over
current audio video-conferencing solutions
Partly because they usually fail to convey the subtle
nonverbal cues that are important in face-to-face
conversations.
Image Source: Freshbooks

11. Navigation over a large area
● Using spatial audio to augment visual
position and orientation
● Comparison with non spatialized audio.
11
Level 7, ECL NZ

12. Related Work
12
Since the first two-way video
telephony system back in
the 1960’s, remote collaboration
has become more and more
convenient with commercial multi-
party video-conference systems
that are currently used in our office
life.
AT&T Picture phone 1964 (wikipedia)

13. Related Work
● Kinected Conference: Augmenting Video
Imaging with Calibrated Depth and Audio -
Anthony DeVincenzi (MIT ML)
● Shared Sphere - Gun A Lee
13
Synthetic focusing used to create a “Talk to
Focus” effect. Background blur is generated
based on depth map
Shared Sphere

14. ● An MR remote collaboration system that spatializes user’s voice and auditory beacons.
● One of the first user studies that explores how spatial auditory cues can help an MR remote collaboration
task and enhance the local worker’s experience.
● One of the first user studies that explores how hybrid spatial auditory and visual cues can help an MR
remote collaboration task and enhance the local worker’s experience.
● An analysis of the users’ verbal communication in an MR remote collaboration task under different
conditions.
Our Novel Contribution
14

15. Implementation &
User Study
15
+ Magic Leap AR headset
+ Remote Helper - Actor
+ Unity 3D game engine
+ Spatialization Plugin
+ Second generation HTC Vive HMD System Overview

16. Study 1: Spatial Auditory
Cues
● Conditions
○ Non-spatialized voice (A1): Both users talked to
each other like in a normal phone or video call.
○ Spatialized voice (A2): The remote expert’s voice
was spatialized to the local worker.
○ Non-spatialized voice + spatialized auditory beacon
(A3): Only the auditory beacon (a 2 s long but
looping, wide band musical sound) was spatialized
from the target object. Voice was delivered as in
A1.
○ Spatialized voice + spatialized auditory beacon
Asked people to recall character N before the
currently displayed one.
16
Finding cues with audio
Lego Placement

17. Study 2: Spatial Auditory
Cues + Visual Cues
● Conditions
○ Spatialized voice + spatialized auditory beacon
(H1)
○ H1 + hand gestures (H2): The remote expert would
use hand gestures to precisely indicate the target
block when the block entered the local worker’s
FoV.
○ H1 + head frustum (H3): The head frustum of the
remote expert would always be visible in the AR
display.
○ H1 + hand gestures + head frustum (H4): All spatial
auditory and visual cues were shared to the local
worker.
17
Finding cues with audio
Lego Placement

18. Results
18
We found that the spatialized
remote expert’s voice and auditory
beacons could efficiently guide an
object search task in a large area.
Social Presence (Exp 1)
Ranking (Exp 2)
Workload (Exp 2)

19. Discussion
19
The results show that different
auditory cues influenced
participants’ performance, social
experience, and spatial perception
in certain ways Number of words spoken

20. Conclusion &
Future Works
20
We presented a novel MR remote
collaboration system that features
both spatial auditory and visual
cues. Through two user studies,
we found that the spatialized
remote expert’s voice and auditory
beacons could efficiently guide an
object search task in a large area.
Image Source: Techcrunch

21. Thank you!
21
UnConference 2019
Prasanth Sasikumar
Augmented Human Lab