The document discusses using virtual avatars to improve remote collaboration. It provides background on communication cues used in face-to-face interactions versus remote communication. It then discusses early experiments using augmented reality for remote conferencing dating back to the 1990s. The document outlines key questions around designing effective virtual bodies for collaboration and discusses various technologies that have been developed for remote collaboration using augmented reality, virtual reality, and mixed reality. It summarizes several studies that have evaluated factors like avatar representation, sharing of different communication cues, and effects of spatial audio and visual cues on collaboration tasks.

1. TALK TO ME
USING VIRTUAL AVATARS TO IMPROVE
REMOTE COLLABORATION
Mark Billinghurst
May 2022

2. Which is a better avatar for selling a business?

4. “Only through
communication
can Human Life
hold meaning.”
Paulo Freire
Philosopher

5. A wide variety of communication cues used.
Speech
Paralinguistic
Para-verbals
Prosodics
Intonation
Audio
Gaze
Gesture
Face Expression
Body Position
Visual
Object Manipulation
Writing/Drawing
Spatial Relationship
Object Presence
Environmental
Face to Face Communication

6. Face to Face Collaboration
Task Space
Communication Space

7. Face to Face Communication
Audio Cues
Visual Cues
Environmental Cues

8. My Workplace in 2020, 2021…

9. Remote Conferencing
• da

10. Remote Communication
Audio Cues
Visual Cues
Environmental Cues

11. Communication Seam
• da Task Space Communication Space

12. Limitations with Current Technology
•Lack of spatial cues
• Person blends with background
•Poor communication cues
• Limited gaze, gesture, non-verbal communication
•Separation of task/communication space
• Can’t see person and workspace at same time

13. Star Wars – Hologram (1977)

14. Connecting at a Distance
• Using AR/VR to connect at a distance
• Restore spatial cues
• Sharing non-verbal cues
• Creating shared spaces

15. Early Experiments (1994 - 2003)
Greenspace (1994)
AR conferencing (1999)
3D Live (2003)

16. AR Video Conferencing (2001)
• Bringing conferencing into real world
• Using AR video textures of remote people
• Attaching AR video to real objects
Billinghurst, M., & Kato, H. (2002). Collaborative augmented reality. Communications of the ACM, 45(7), 64-70.

17. 2001

18. Multi-View AR Conferencing
Billinghurst, M., Cheok, A., Prince, S., & Kato, H. (2002). Real world
teleconferencing. Computer Graphics and Applications, IEEE, 22(6), 11-13.

20. Holoportation (2016)
• Augmented Reality + 3D capture + high bandwidth
• http://research.microsoft.com/en-us/projects/holoportation/

21. Collaboration in Augmented Reality
Magic Leap Avatar Chat

22. Collaboration in Virtual Reality
Meta Workrooms
RecRoom

23. Microsoft Mesh (2021)

24. Bringing 3D into 2D Video Conferencing
https://www.youtube.com/watch?v=OKNRMauZjFY

25. Remote Communication

26. Key Questions
• Do we need a virtual body?
• What should our virtual body look like?
• What enhancements could we provide to virtual bodies?
• What type of bodies do we need for different collaboration tasks?
• What other cues can we provide to enhance remote collaboration?

27. First Person Perspective Remote Collaboration
• View from remote user’s perspective
• Wearable Teleconferencing
• audio, video, pointing
• send task space video
• CamNet (1992)
• British Telecom
• Similar CMU study (1996)
• cut performance time in half

28. AR for Remote Collaboration
• Camera + Processing + AR Display + Connectivity
• First person Ego-Vision Collaboration

29. AR View Remote Expert View

30. Adding Gaze Cues - Empathy Glasses
• Combine together eye-tracking, display, face expression
• Implicit cues – eye gaze, face expression
+
+
Pupil Labs Epson BT-200 AffectiveWear
Masai, K., Sugimoto, M., Kunze, K., & Billinghurst, M. (2016, May). Empathy Glasses. In Proceedings of the
34th Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems. ACM.

31. Remote Collaboration
• Eye gaze pointer and remote pointing
• Face expression display
• Implicit cues for remote collaboration

33. Shared Sphere – 360 Video Sharing
Shared
Live 360 Video
Host User Guest User
Lee, G. A., Teo, T., Kim, S., & Billinghurst, M. (2017). Mixed reality collaboration through sharing a
live panorama. In SIGGRAPH Asia 2017 Mobile Graphics & Interactive Applications (pp. 1-4).

35. Sharing a View

36. 3D Live Scene Capture
• Use cluster of RGBD sensors
• Fuse together 3D point cloud

37. Live 3D Scene Capture

38. Scene Capture and Sharing
Scene Reconstruction Remote Expert Local Worker

39. AR View Remote Expert View

40. 3D Mixed Reality Remote Collaboration (2022)
Tian, H., Lee, G. A., Bai, H., & Billinghurst, M. (2023). Using Virtual Replicas to Improve Mixed
Reality Remote Collaboration. IEEE Transactions on Visualization and Computer Graphics.

41. View Sharing Evolution
• Increased immersion
• Improved scene understanding
• Better collaboration
2D 360 3D

42. Switching between 360 and 3D views
• 360 video
• High quality visuals
• Poor spatial representation
• 3D reconstruction
• Poor visual quality
• High quality 3D reconstruction

43. Swapping between 360 and 3D views
• Have pre-captured 3D model of real space
• Enable remote user to swap between live 360 video or 3D view
• Represent remote user as avatar
Teo, T., F. Hayati, A., A. Lee, G., Billinghurst, M., & Adcock, M. (2019). A technique for mixed reality remote collaboration using
360 panoramas in 3d reconstructed scenes. In 25th ACM Symposium on Virtual Reality Software and Technology (pp. 1-11).

45. • Using AR/VR to share communication cues
• Gaze, gesture, head pose, body position
• Sharing same environment
• Virtual copy of real world
• Collaboration between AR/VR
• VR user appears in AR user’s space
Piumsomboon, T., Dey, A., Ens, B., Lee, G., & Billinghurst, M. (2019). The effects of sharing awareness cues
in collaborative mixed reality. Frontiers in Robotics and AI, 6, 5.
Sharing: Virtual Communication Cues (2019)

46. Sharing Virtual Communication Cues
• Collaboration between AR and VR
• Gaze Visualization Conditions
• Baseline, FoV, Head-gaze, Eye-gaze

48. Results
• Predictions
• Eye/Head pointing better than no cues
• Eye/head pointing could reduce need for pointing
• Results
• No difference in task completion time
• Head-gaze/eye-gaze great mutual gaze rate
• Using head-gaze greater ease of use than baseline
• All cues provide higher co-presence than baseline
• Pointing gestures reduced in cue conditions
• But
• No difference between head-gaze and eye-gaze

49. Multi-Scale Collaboration
• Changing the user’s virtual body scale

51. On the Shoulder of a Giant
Piumsomboon, T., Lee, G. A., Irlitti, A., Ens, B., Thomas, B. H., & Billinghurst, M. (2019, May). On the shoulder
of the giant: A multi-scale mixed reality collaboration with 360 video sharing and tangible interaction.
In Proceedings of the 2019 CHI conference on human factors in computing systems (pp. 1-17).

52. Role of Spatial Cues (2020)
• What is impact of spatial audio/visual cues
over large scale AR/VR collaboration?
Yang, J., Sasikumar, P., Bai, H., Barde, A., Sörös, G., & Billinghurst, M. (2020). The effects of spatial auditory
and visual cues on mixed reality remote collaboration. Journal on Multimodal User Interfaces, 14(4), 337-352.

53. Experiment Environment
Starting point
Local AR space
Remote VR
space
Lego blocks
90 𝑚!

54. Finding Lego Blocks (2 𝑐𝑚!)

55. Project Set-up
Spatialized auditory
beacon from the
object
AR HMD
Controller
Local AR
worker’s field of
view
Remote VR
expert teleports
in mesh
Local Remote
VR HMD

57. User Study Results
• Two studies conducted
• (1) Audio only cues (spatial vs. non-spatial), (2) Different types of visual cues (head, hands)
• Key findings
• Spatial audio significantly increase Social Presence
• Users strongly preferred head, gesture, audio cues, or non-spatial voice, spatial beacon
• Integrating visual cues with the spatial auditory cues significantly improved task performance
• Integrating the remote expert’s head frustum into the spatial auditory cues can provide
significantly better social presence, spatial awareness, and system usability
Spatial Presence Scores
Task Completion Times

58. Sharing Gesture Cues
• What type of gesture cues should be shared in AR/VR collaboration?
Kim, S., Lee, G., Huang, W., Kim, H., Woo, W., & Billinghurst, M. (2019). Evaluating the combination of visual communication cues for HMD-
based mixed reality remote collaboration. In Proceedings of the 2019 CHI conference on human factors in computing systems (pp. 1-13).
Augmented Reality
Virtual Reality

59. Communication Cues
• Four different cues used
• (1) Hands Only (HO), (2) Hands + Pointer (HP)
• (3) Hands + Sketch (HS), (4) Hands + Pointer + Sketch (HPS)
• Three experimental tasks
• Lego assembly, Tangram puzzle, Origami folding

60. Key Results
• Task completion time
• Sketch cues enabled users to complete tasks significantly faster (task dep.)
• Adding pointing didn’t improve task completion time
• Co-Presence
• Adding pointing and sketch cues didn’t improve feeling of co-presence
• User Preference
• User’s overwhelming preferred Hands + Pointer + Sketch condition, Hands Only ranked last
Task completion time
“sketch allowed drawings for accuracy
and hand for general use", “sketch is
pretty useful for describing actions that
was difficult by verbal words and could
express more details".

61. Changing Gaze Cues
How sharing gaze behavioural cues can improve remote collaboration in Mixed Reality environment.
➔ Developed eyemR-Vis, a 360 panoramic Mixed Reality remote collaboration system
➔ Showed gaze behavioural cues as bi-directional spatial virtual visualisations shared
between a local host (AR) and a remote collaborator (VR).
Jing, A., May, K., Lee, G., & Billinghurst, M. (2021). Eye See What You See: Exploring How Bi-Directional Augmented
Reality Gaze Visualisation Influences Co-Located Symmetric Collaboration. Frontiers in Virtual Reality, 2, 79.

62. System Design
➔ 360 Panaramic Camera + Mixed Reality View
➔ Combination of HoloLens2 + Vive Pro Eye
➔ 4 gaze behavioural visualisations:
browse, focus, mutual, fixated circle

63. System Design
Browse Focus
Mutual Fixed
Circle-map

64. System Demo
https://www.youtube.com/watch?v=1liRqdghGN8

65. Experiment
• Participants
• 12 pairs of people (6 women), 60% AR/VR experience, 50% with gaze experience
• Conditions
• 2x2 study design, also no gaze (NG)
• Gaze Behaviour (WB) /No Gaze Behaviour (NB)
• Uni-directional (U) /Bi-directional gaze (B)
• Tasks
• Finding abstract symbols, guiding other person to the symbols
• Measures
• Quantitative measures: gaze behaviour metrics, completion time, questionnaires
• Qualitative measures: open-ended questions and behaviour analysis

66. Research Questions
• (1) Gaze vs no Gaze: compared to no gaze visualisation, gaze cues
would encourage joint focus by providing explicit visual feedback
• (2) Uni-directional gaze style vs bi-directional style: Bidirectional style is
more balanced visually, leading to less confusion of gaze identity
• (3) Gaze behaviours vs no behaviour: Integrating gaze behaviour
visualisations lowers communication cognitive load

67. Results
• RQ1: Behaviour visualisations stimulate frequent joint attention
• RQ2: Bidirectional gaze ensured gaze information was accurately delivered
• RQ3: Gaze made it easier to coordinate on target object location, and made
communication easier and more effective

68. Sharing: Separating Cues from Body
• What happens when you can’t see your colleague/agent?
Piumsomboon, T., Lee, G. A., Hart, J. D., Ens, B., Lindeman, R. W., Thomas, B. H., & Billinghurst, M.
(2018, April). Mini-me: An adaptive avatar for mixed reality remote collaboration. In Proceedings of the
2018 CHI conference on human factors in computing systems (pp. 1-13).
Collaborating Collaborator out of View

69. Mini-Me Communication Cues in MR
• When lose sight of collaborator a Mini-Me avatar appears
• Miniature avatar in real world
• Mini-Me points to shared objects, show communication cues
• Redirected gaze, gestures

71. Results from User Evaluation (16 subjects)
• Collaboration between user in AR, expert in VR
• Hololens, HTC Vive
• Two tasks
• Asymmetric, symmetric collaboration
• Significant performance improvement
• 20% faster with Mini-Me
• Social Presence
• Higher sense of Presence
• Users preferred
• People felt the task was easier to complete
• 60-75% preference
“I feel like I am talking
to my partner”
“The ability to see the small
avatar … enhanced the
speed of solving the task”

72. Avatar Representation
• Pilot study with recorded avatar
• Motorcycle engine assembly
• Avatar types
• (A1) Annotation: Computer-generated lines drawn in 3D space.
• (A2) Hand Gesture: Real hand gestures captured using stereoscopic cameras
• (A3) Avatar: Virtual avatar reconstructed using inverse kinematics.
• (A4) Volumetric Playback: Using three Kinect cameras, the movements of an expert
are captured and played back as a virtual avatar via a see-through headset.

73. Avatar Representation
Remote pointer Realistic hands

74. Representing Remote Users
Virtual Avatar Volumetric Avatar

75. Experiment Design (30 participants)
Performing motorbike assembly task under guidance
- Easy, Medium, Hard task
Hypotheses
- H1. Volumetric playback would have a better sense of social presence in a
remote training system.
- H2. Volumetric playback would enable faster completion of tasks in a remote
training system
Measures
• NMM Social Presence Questionnaire, NASA TLX, SUS

76. Results
• Hands, Annotation significantly faster than avatar
• Volumetric playback induced the highest sense of co-presence
• Users preferred Volumetric or Annotation interface
Performance Time
Average Ranking

77. Results
Volumetric instruction cues exhibits an increase in co-presence and
system usability while reducing mental workload and frustration.
Mental Load (NASA TLX)
System Usability Scale

78. User Feedback
• Annotations easy to understand (faster performance)
• “Annotation is very clear and easy to spot in a 3d environment”.
• Volumetric creates high degree of social presence (working with person)
• “Seeing a real person demonstrate the task, feels like being next to a person”.
• Recommendations
• Use Volumetric Playback to improve Social Presence and system usability
• Using a full-bodied avatar representation in a remote training system is not
recommended unless it is well animated
• Using simple annotation can have significant improvement in performance if
social presence is not of importance.

79. Avatar Representation for Social Presence
• What should avatars look
like for social situations?
• Cartoon vs. realistic?
• Partial or full body?
• Impact on Social Presence?
Yoon, B., Kim, H. I., Lee, G. A., Billinghurst, M., & Woo, W. (2019, March). The effect of
avatar appearance on social presence in an augmented reality remote collaboration. In
2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR) (pp. 547-556). IEEE.

80. Avatar Representations
• Cartoon vs. Realistic, Part Body vs. Whole Body
• Realistic Head & Hands (RHH), Realistic Upper Body (RUB), Realistic Whole Body (RWB),
• Cartoon Head & Hands (CHH), Cartoon Upper Body (CUB), Cartoon Whole Body (CWB).

81. Experiment
• Within-subjects design (24 subjects)
• 6 conditions: RHH, RUB, RWB, CHH, CUB, CWB
• AR/VR interface
• Subject in AR interface, actor in VR
• Experiment measures
• Social Presence
• Networked Mind Measure of Social Presence survey
• Bailenson’s Social Presence survey
• Post Experiment Interview
• Tasks
• Study 1: Crossword puzzle (Face to Face discussion)
• Study 2: Furniture placement (virtual object placement)
AR user
VR user

82. Hypotheses
H1. Body Part Visibility will affect the user’s Social Presence in AR.
H2. The Whole-Body virtual avatars will have the highest Social
Presence among the three levels of visibility.
H3. Head & Hands virtual avatars will have the lowest Social
Presence among the three levels of visibility.
H4. The Character Style will affect the user’s Social Presence.
H5. Realistic avatars will have a higher Social Presence than
Cartoon Style avatars in an AR remote collaboration.

83. Results
• Aggregated Presence Scores
• 1: strongly disagree - 7: strongly agree

84. User Comments
• ‘Whole Body’ Avatar Expression to Users
• “Presence was high with full body parts, because I could notice joints’
movement, behaviour, and reaction.”
• “I didn’t get the avatar’s intention of the movement, because it had only
head and hands.”
• ‘Upper Body’ vs. ‘Whole Body’ Avatar
• “I preferred the one with whole body, but it didn’t really matter because I
didn’t look at the legs much.”,
• “I noticed head and hands model immediately, but I didn’t feel the
difference whether the avatar had a lower body or not.”
• ‘Realistic’ vs ‘Cartoon’ style Avatars
• "The character seemed more like a game than furniture placement in real. I
felt that realistic whole body was collaborating with me more.”

85. Hypotheses Outcome
H1. Body Part Visibility will affect the user’s Social Presence in AR.
H2. The Whole-Body virtual avatars will have the highest Social
Presence among the three levels of visibility.
H3. Head & Hands virtual avatars will have the lowest Social
Presence among the three levels of visibility.
H4. The Character Style will affect the user’s Social Presence.
H5. Realistic avatars will have a higher Social Presence than
Cartoon Style avatars in an AR remote collaboration.

86. Key Lessons Learned
• Avatar Body Part visibility should be considered first when designing for AR remote
collaboration since it significantly affects Social Presence
• Body Part Visibility
• Whole Body & Upper Body: Whole body is preferred, but upper body is okay in some cases
• Head & Hands: Should be avoided
• Character Style
• No difference in Social Presence between Realistic and Cartoon avatars
• However, the majority of participants had a positive response towards the Realistic avatar
• Cartoon character for fun, Realistic avatar for professional meetings

87. Technology Trends
• Advanced displays
• Wide FOV, high resolution
• Real time space capture
• 3D scanning, stitching, segmentation
• Natural gesture interaction
• Hand tracking, pose recognition
• Robust eye-tracking
• Gaze points, focus depth
• Emotion sensing/sharing
• Physiological sensing, emotion mapping

88. Sensor Enhanced HMDs
Eye tracking, heart rate,
pupillometry, and face camera
HP Omnicept Project Galea
EEG, EEG, EMG, EDA, PPG,
EOG, eye gaze, etc.

90. Enhancing Emotion
• Using physiological and contextual cues to enhance emotion representation
• Show user’s real emotion, make it easier to understand user emotion, etc..
Real User
Physiological Cues
Arousal/Valence
Positive
Negative
Avatar
Context Cues

91. System Design

92. Early Results
Face Tracking Positive Affect Avatar Outcome

94. Conversational agent
Intelligent Virtual Agents (IVAs)
Embodied in 2D Screen Embodied in 3D space

95. Photorealistic Characters
• Synthesia
• AI + ML to create videos
• Speech + image synthesis
• Supports >60 languages
• Personalized characters

96. https://www.youtube.com/watch?v=vifHh4WjEFE

97. Empathic Mixed Reality Agents

98. Intelligent Digital Humans
• Soul Machines
• AI digital brain
• Expressive digital humans
• Autonomous animation
• Able to see and hear
• Learn from users

100. Towards Empathic Social Agents
• Goal: Using agents to creating
empathy between people
• Combine
• Scene capture
• Shared tele-presence
• Trust/emotion recognition
• Enhanced communication cues
• Separate cues from representation
• Facilitating brain synchronization

101. Trends..
Time
Human
Touch
Empathic Agents
Digital Humans
Photo Realistic
Chatbots
Voice menus

102. Summary
• Being able to share communication cues is vital
• Focus on the cues needed for task
• Need for body is task dependent
• Physical task - pointer/simple cues okay
• Social task – avatar/volumetric avatar better
• Simple representation may be okay for some tasks
• Legs optional, arms essential
• Using additional communication cues can be beneficial
• Gaze lines, view frustrum, body copy, etc.

103. Remote Communication

104. MiniMe
Virtual Cues Enhanced Emotion
Brain
Synchronization
Trust Recognition
Scene Capture
AI

105. Opportunities for Research
• Adding sensors
• Physiological cues, sharing emotional/mental state
• Behavioral/sensor synchronization
• Novel communication cues
• Scene sharing, gaze, gesture cues
• AI + Virtual Avatars
• Realistic behaviors, visual representation, cultural translation, etc
• Autonomous Mixed Reality Agents
• Awareness of real world, real user behavior

107. www.empathiccomputing.org
@marknb00
mark.billinghurst@auckland.ac.nz