This is a guest lecture given by Mark Billinghurst at the University of Sydney on March 27th 2024. It discusses some future research directions for Augmented Reality.

1. FUTURE RESEARCH DIRECTIONS
FOR AUGMENTED REALITY
Mark Billinghurst
March 27th 2024
mark.billinghurst@unisa.edu.au

2. 1968 2022 2070?

4. “.. the technologies that will significantly
affect our lives over the next 10 years
have been around for a decade. The
future is with us ... The trick is learning
how to spot it”
October 2004
Bill Buxton

5. Augmented Reality Definition
1. Combines Real and Virtual Images
• Both can be seen at the same time
2. Interactive in real-time
• The virtual content can be interacted with
3. Registered in 3D
• Virtual objects appear fixed in space
Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.

6. Sutherland’s Head Mounted Display - 1968
https://www.youtube.com/watch?v=eVUgfUvP4uk

7. AR Key Enabling Technologies
1. Combines Real and Virtual Images
Display Technology
2. Interactive in real-time
Interaction Technologies
3. Registered in 3D
Tracking Technologies

8. Research Grand Challenges
Billinghurst, M. (2021). Grand challenges for
augmented reality. Frontiers in Virtual
Reality, 2, 578080.

9. Grand Challenges
• Research in Enabling Technology
• Display Technology
• Interaction
• Tracking
• Other Key Research Areas
• Perception and Neuroscience
• Collaboration
• Social and Ethical Issues

10. DISPLAY TECHNOLOGY

11. • Past
• Bulky Head mounted displays
• Current
• Handheld, lightweight head mounted
• Future
• Projected AR
• Wide FOV see through
• Retinal displays
• Contact lens
Evolution in Displays

12. North Focals (2020)
• https://www.bynorth.com
• Socially acceptable smart glasses
• $599 USD, small field of view
• Ring input device

13. Wide FOV See-Through Displays
• Waveguide techniques
• Wider FOV
• Thin see through
• Socially acceptable
• Pinlight Displays
• LCD panel + point light sources
• 110 degree FOV
• UNC/Nvidia
Lumus DK40
Maimone, A., Lanman, D., Rathinavel, K., Keller, K., Luebke, D., & Fuchs, H. (2014). Pinlight displays:
wide field of view augmented reality eyeglasses using defocused point light sources. In ACM SIGGRAPH
2014 Emerging Technologies (p. 20). ACM.

14. https://www.youtube.com/watch?v=P407DFm0PFQ
Pinlight Display Demo

15. Virtual Retinal Display. (1993 - )
• Use scanner to scan images into the eye

16. From Prototype to Reality
Tom Furness (1993) Microvision Nomad (2002)

17. Current Microvision Technology
https://www.youtube.com/watch?v=OvHN_bypoNE

18. Occlusive Displays
OST AR View

19. Prototype Implementation
Kiyokawa, Kiyoshi, Yoshinori Kurata, and Hiroyuki Ohno. "An optical see-through display for mutual
occlusion with a real-time stereovision system." Computers & Graphics 25.5 (2001): 765-779.

20. Results

21. Magic Leap 2
• Dimmer LCD panel supports pixel level occlusion

22. More Recent – Occlusion Leak Compensation
• Change occlusion mask to larger than AR object
• Compensate with additional graphics
Itoh, Yuta, Takumi Hamasaki, and Maki Sugimoto. "Occlusion leak compensation for optical see-through displays using a single-
layer transmissive spatial light modulator." IEEE transactions on visualization and computer graphics 23.11 (2017): 2463-2473.

23. Contact Lens Displays
• Contact Lens only
• Unobtrusive
• Significant technical challenges
• Power, data, resolution
• Babak Parviz (2008)
• MojoVision
• Mojo Lens
• Prototype smart contact lens
• https://www.mojo.vision/
http://spectrum.ieee.org/biomedical/bionics/augmented-reality-in-a-contact-lens/

26. INTERACTION

27. Evolution of Interaction
• Past
• Limited interaction
• Viewpoint manipulation
• Present
• Screen based, gesture
• Tangible interaction
• Future
• Multimodal
• Intelligent Interfaces
• Physiological/Sensor based

28. Natural Gesture
• Freehand gesture input
• Depth sensors for gesture capture
• Move beyond simple pointing
• Rich two handed gestures
• Eg Microsoft Research Hand Tracker
• 3D hand tracking, 30 fps, single sensor
• Commercial Systems
• Hololens2, Oculus, Intel, MagicLeap, etc
Sharp, T., Keskin, C., Robertson, D., Taylor, J., Shotton, J., Leichter, D. K. C. R. I., ... & Izadi, S.
(2015, April). Accurate, Robust, and Flexible Real-time Hand Tracking. In Proc. CHI (Vol. 8).

29. https://www.youtube.com/watch?v=7qt6ZSo4-JY

30. Multi-Scale Gesture
• Combine different gesture types
• In-air gestures – natural but imprecise
• Micro-gesture – fine scale gestures
• Gross motion + fine tuning interaction
Ens, B., Quigley, A., Yeo, H. S., Irani, P., Piumsomboon, T., & Billinghurst, M. (2018). Counterpoint:
Exploring Mixed-Scale Gesture Interaction for AR Applications. In Extended Abstracts of the 2018 CHI
Conference on Human Factors in Computing Systems (p. LBW120). ACM.

31. https://www.youtube.com/watch?v=TRfqNtt1VxY&t=23s

32. Multimodal Input
• Gesture and Speech Input Complimentary
• Speech
• modal commands, quantities
• Gesture
• selection, motion, qualities
• Support combined commands
• “Put that there” + pointing
• Previous work found multimodal interfaces
intuitive for 2D/3D graphics interaction
Lee, M., Billinghurst, M., Baek, W., Green, R., & Woo, W. (2013). A usability study of
multimodal input in an augmented reality environment. Virtual Reality, 17(4), 293-305.

33. Multimodal CA Design
Billinghurst, M., Piumsomboon, T., & Bai, H. (2014). Hands in Space: Gesture Interaction with
Augmented-Reality Interfaces. IEEE computer graphics and applications, (1), 77-80.

34. Eye Tracking Input
• HMDs with integrated eye-tracking
• Hololens2, MagicLeap One
• Research questions
• How can eye gaze be used for interaction?
• What interaction metaphors are natural?

35. Gaze and Pinch Interaction
Pfeuffer, K., Mayer, B., Mardanbegi, D., & Gellersen, H. (2017, October). Gaze+ pinch interaction
in virtual reality. In Proceedings of the 5th symposium on spatial user interaction (pp. 99-108).

36. https://www.youtube.com/watch?v=NzLrZSF8aDM

37. Apple Vision Pro

38. Physiological Sensor Input
• Using physiological sensors for implicit input
• Systems that recognize user intent/activity
• EEG
• Measuring brain activity
• EMG
• Measuring muscle activity

39. https://www.youtube.com/watch?v=K34p7RwjWt0

40. Intelligent Interfaces
• Move to Implicit Input vs. Explicit
• Recognize user behaviour
• Provide adaptive feedback
• Move beyond check-lists of actions
• E.g. AR + Intelligent Tutoring
• Constraint based ITS + AR
• PC Assembly (Westerfield, 2015)
• 30% faster, 25% better retention
Westerfield, G., Mitrovic, A., & Billinghurst, M. (2015). Intelligent Augmented Reality Training for
Motherboard Assembly. International Journal of Artificial Intelligence in Education, 25(1), 157-172.

41. TRACKING

42. Evolution of Tracking
• Past
• Location based, marker based,
• magnetic/mechanical
• Present
• Image based, hybrid tracking
• Future
• Ubiquitous
• Model based
• Environmental
• AI Enhanced

43. Depth Sensing Tracking
• Environment capture
• Use depth sensors to capture scene & track from model
• InifinitAM (www.robots.ox.ac.uk/~victor/infinitam/)
• Real time scene capture, dense or sparse capture, open source
• iPad Pro LiDAR
• Scene scanning up to 5m

44. https://www.youtube.com/watch?v=qZY6y1IVIfw

45. Fusion4D
• Real capture and dynamic reconstruction
• RGBD sensors + incremental reconstruction
Dou, M., Khamis, S., Degtyarev, Y., Davidson, P., Fanello, S. R., Kowdle, A., ... & Izadi, S. (2016). Fusion4d:
Real-time performance capture of challenging scenes. ACM Transactions on Graphics (TOG), 35(4), 1-13.

46. Fusion4D Demo
• https://www.youtube.com/watch?v=rnz0Kt36mOQ

47. Using Machine Learning to Improve Tracking
• Visual inertial navigation (ARKit, ARCore)
• relies on continuous visual tracking, fails when bad lighting, fast motion, repeated textures, etc.
• Can use IMU to reduce the dependence on vision information
• But IMU drifts over time, so Deep learning can robustly estimate relative displacement
• Integrate neural network IMU observations with visual-inertial navigation system
Chen, D., Wang, N., Xu, R., Xie, W., Bao, H., & Zhang, G. RNIN-VIO: Robust Neural Inertial
Navigation Aided Visual-Inertial Odometry in Challenging Scenes. (2021)

48. Method
• Motion Inference module trained on human motion

49. https://www.youtube.com/watch?v=jpOQpS1fR8g

50. AR Cloud Based Tracking
• AR Cloud
• a machine-readable 1:1 scale model of the real world
• processing recognition/tracking data in the cloud
• Can create cloud from input from multiple devices
• Store key visual features in cloud, Stitch features from multiple devices
• Retrieve for tracking/interaction
• AR Cloud Companies
• 6D.ai (Niantic), Vertical.ai, Ubiquity6, etc

51. 6D.ai/Niantic Demo
• https://www.youtube.com/watch?v=AwwU14gllS0

52. PERCEPTION AND NEUROSCIENCE

53. Sensor Enhanced HMDs
Eye tracking, heart rate,
pupillometry, and face camera
HP Omnicept Project Galea
EEG, EMG, EDA, PPG,
EOG, eye gaze, etc.

55. Brain Synchronization

56. Pre-training (Finger Pointing) Session Start

57. Post-Training (Finger Pointing) Session End

58. Brain Synchronization in VR

61. asfd

63. COLLABORATION

64. A wide variety of communication cues used.
Speech
Paralinguistic
Para-verbals
Prosodics
Intonation
Audio
Gaze
Gesture
Face Expression
Body Position
Visual
Object Manipulation
Writing/Drawing
Spatial Relationship
Object Presence
Environmental
Face to Face Communication

65. Face to Face Communication
Audio Cues
Visual Cues
Environmental Cues

66. Remote Conferencing
• da

67. Remote Communication
Audio Cues
Visual Cues
Environmental Cues

68. Connecting at a Distance
• Using AR/VR to connect at a distance
• Restore spatial cues
• Sharing non-verbal cues
• Creating shared spaces

69. Remote Communication

70. MiniMe
Virtual Cues Enhanced Emotion
Brain
Synchronization
Emotion Recognition
Scene Capture
AI

71. • Using AR/VR to share communication cues
• Gaze, gesture, head pose, body position
• Sharing same environment
• Virtual copy of real world
• Collaboration between AR/VR
• VR user appears in AR user’s space
Piumsomboon, T., Dey, A., Ens, B., Lee, G., & Billinghurst, M. (2019). The effects of sharing awareness cues
in collaborative mixed reality. Frontiers in Robotics and AI, 6, 5.
Sharing Virtual Communication Cues (2019)

72. Sharing Virtual Communication Cues
• AR/VR displays
• Gesture input (Leap Motion)
• Room scale tracking
• Conditions
• Baseline, FoV, Head-gaze, Eye-gaze

74. Sharing: Communication Cues (2018)
• What happens when you can’t see your colleague/agent?
Piumsomboon, T., Lee, G. A., Hart, J. D., Ens, B., Lindeman, R. W., Thomas, B. H., & Billinghurst, M.
(2018, April). Mini-me: An adaptive avatar for mixed reality remote collaboration. In Proceedings of the
2018 CHI conference on human factors in computing systems (pp. 1-13).
Collaborating Collaborator out of View

75. Mini-Me Communication Cues in MR
• When lose sight of collaborator a Mini-Me avatar appears
• Miniature avatar in real world
• Mini-Me points to shared objects, show communication cues
• Redirected gaze, gestures

77. SOCIAL AND ETHICAL ISSUES

78. Social Acceptance
• People don’t want to look silly
• Only 12% of 4,600 adults would be willing to wear AR glasses
• 20% of mobile AR browser users experience social issues
• Acceptance more due to Social than Technical issues
• Needs further study (ethnographic, field tests, longitudinal)

79. TAT AugmentedID

82. Ethical Issues
• Persuasive Technology
• Affecting emotions
• Behaviour modification
• Privacy Concerns
• Facial recognition
• Space capture
• Personal data
• Safety Concerns
• Sim sickness, Distraction
• Long term effects
Pase, S. (2012). Ethical considerations in augmented reality applications. In Proceedings of the International Conference on
e-Learning, e-Business, Enterprise Information Systems, and e-Government (EEE) (p. 1). The Steering Committee of The
World Congress in Computer Science, Computer Engineering and Applied Computing (WorldComp).

83. Identification
• Collected set of features from body motion in VR
• Have people perform standard actions
• Able to recognize people with about 40% accuracy (cf. 5% by chance)
• Relative distances between head and hands produced best results
• Head motion alone 30% accurate
Pfeuffer, K., Geiger, M. J., Prange, S., Mecke, L., Buschek, D., & Alt, F. (2019, May). Behavioural biometrics in
vr: Identifying people from body motion and relations in virtual reality. In Proceedings of the 2019 CHI
Conference on Human Factors in Computing Systems (pp. 1-12).

84. Trauma in VR..
• In VR the goal is immersion, presence and sense of self
• What are the consequences of having a traumatic,
aggressive or emotional in-VR experience?

85. CONCLUSIONS

86. Conclusions
• AR is becoming commonly available
• Significant advances over last 50+ years
• In order to achieve Sutherland’s vision, research basics
• Display, Tracking, Interaction
• New AR technologies will enable this to happen
• Display devices, Interaction, Tracking technologies
• There are still significant areas for research
• Social Acceptance, Collaboration, Ethics, Etc.

87. The Metaverse
• Neal Stephenson’s “Snow Crash” (1992)
• The Metaverse is the convergence of:
• 1) virtually enhanced physical reality
• 2) physically persistent virtual space

88. Metaverse Taxonomy
• Four Key Components
• Virtual Worlds
• Augmented Reality
• Mirror Worlds
• Lifelogging
• Metaverse Roadmap
• http://metaverseroadmap.org/

89. Research Opportunities

90. Possible Research Directions
• Lifelogging to VR
• Bringing real world actions into VR, VR to experience lifelogging data
• AR to Lifelogging
• Using AR to view lifelogging data in everyday life, Sharing physiological data
• Mirror Worlds to VR
• VR copy of the real world, Mirroring real world collaboration in VR
• AR to Mirror Worlds
• Visualizing the past in place, Asymmetric collaboration
• And more..

91. Trends in AR Research
Kim, K., Billinghurst, M., Bruder, G., Duh,
H. B. L., & Welch, G. F. (2018). Revisiting
trends in augmented reality research: A
review of the 2nd decade of ISMAR (2008–
2017). IEEE transactions on visualization
and computer graphics, 24(11), 2947-2962.

93. www.empathiccomputing.org
@marknb00
mark.billinghurst@unisa.edu.au