Presentation on trends and future research directions in Augmented Reality. Given by Mark Billinghurst at the Smart Cloud 2015 conference on September 16th, 2015, in Seoul, Korea.
3. Augmented Reality
1.⯠Combines Real andVirtual Images
â˘âŻBoth can be seen at the same time
2.⯠Interactive in real-time
â˘âŻThe virtual content can be interacted with
3.⯠Registered in 3D
â˘âŻVirtual objects appear fixed in space
Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.
4. 50 Years of Progress (1965-2015)
â˘âŻMoving from lab to living room
â˘âŻAR devices available in every pocket
1968: First AR HMD
1980âs: SuperCockpit
1997: Outdoor AR
2005: Mobile AR
18. Natural Gesture (2-5 years)
â˘âŻFreehand gesture input
â˘âŻ Depth sensors for gesture capture
â˘âŻ Move beyond simple pointing
â˘âŻ Rich two handed gestures
â˘âŻEg Microsoft Research Hand Tracker
â˘âŻ 3D hand tracking, 30 fps, single sensor
â˘âŻCommercial Systems
â˘âŻ Meta, MS Hololens, Occulus, Intel, etc
Sharp, T., Keskin, C., Robertson, D., Taylor, J., Shotton, J., Leichter, D. K. C. R. I., ... & Izadi, S.
(2015, April). Accurate, Robust, and Flexible Real-time Hand Tracking. In Proc. CHI (Vol. 8).
19. Multimodal Input (5+ years)
â˘âŻCombine gesture and speech input
â˘âŻ Gesture good for qualitative input
â˘âŻ Speech good for quantitative input
â˘âŻ Support combined commands
â˘âŻ âPut that thereâ + pointing
â˘âŻEg HIT Lab NZ multimodal input
â˘âŻ 3D hand tracking, speech
â˘âŻ Multimodal fusion module
â˘âŻ Complete tasks faster with MMI, less errors
Billinghurst, M., Piumsomboon, T., & Bai, H. (2014). Hands in Space: Gesture Interaction with
Augmented-Reality Interfaces. IEEE computer graphics and applications, (1), 77-80.
20. Intelligent Interfaces (10+ years)
â˘âŻMove to Implicit Input vs. Explicit
â˘âŻ Recognize user behaviour
â˘âŻ Provide adaptive feedback
â˘âŻ Support scaffolded learning
â˘âŻ Move beyond check-lists of actions
â˘âŻEg AR + Intelligent Tutoring
â˘âŻ Constraint based ITS + AR
â˘âŻ PC Assembly (Westerfield (2015)
â˘âŻ 30% faster, 25% better retention
Westerfield, G., Mitrovic, A., & Billinghurst, M. (2015). Intelligent Augmented Reality Training for
Motherboard Assembly. International Journal of Artificial Intelligence in Education, 25(1), 157-172.
22. Evolution of Tracking
â˘âŻPast
â˘âŻ Location based, marker based,
â˘âŻ magnetic/mechanical
â˘âŻPresent
â˘âŻ Image based, hybrid tracking
â˘âŻFuture
â˘âŻ Ubiquitous
â˘âŻ Model based
â˘âŻ Environmental
23. Model Based Tracking (1-3 yrs)
â˘âŻTrack from known 3D model
â˘âŻ Use depth + colour information
â˘âŻ Match input to model template
â˘âŻ Use CAD model of targets
â˘âŻRecent innovations
â˘âŻ Learn models online
â˘âŻ Tracking from cluttered scene
â˘âŻ Track from deformable objects
Hinterstoisser, S., Lepetit, V., Ilic, S., Holzer, S., Bradski, G., Konolige, K., & Navab, N. (2013).
Model based training, detection and pose estimation of texture-less 3D objects in heavily
cluttered scenes. In Computer VisionâACCV 2012 (pp. 548-562). Springer Berlin Heidelberg.
24. Environmental Tracking (3+ yrs)
â˘âŻEnvironment capture
â˘âŻ Use depth sensors to capture scene & track from model
â˘âŻInifinitAM (www.robots.ox.ac.uk/~victor/infinitam/)
â˘âŻ Real time scene capture on mobiles, dense or sparse capture
â˘âŻ Dynamic memory swapping allows large environment capture
â˘âŻ Cross platform, open source library available
25. Wide Area Outdoor Tracking (5+ yrs)
â˘âŻProcess
â˘âŻ Combine panoramaâs into point cloud model (offline)
â˘âŻ Initialize camera tracking from point cloud
â˘âŻ Update pose by aligning camera image to point cloud
â˘âŻ Accurate to 25 cm, 0.5 degree over very wide area
Ventura, J., & Hollerer, T. (2012). Wide-area scene mapping for mobile visual tracking. In Mixed
and Augmented Reality (ISMAR), 2012 IEEE International Symposium on (pp. 3-12). IEEE.
29. Crossing the Chasm - 5-10 years
http://www.gartner.com/newsroom/id/3114217
30. Getting from Here to There
â˘âŻNew markets
â˘âŻ Medical
â˘âŻ Education
â˘âŻ Industry
â˘âŻ Etc
â˘âŻNew applications enabled
â˘âŻ Training
â˘âŻ Collaboration
â˘âŻ Information Presentation
â˘âŻ Etc
32. Example: Social Panoramas
â˘âŻGoogle Glass
â˘âŻCapture live image panorama (compass + camera)
â˘âŻRemote device (tablet)
â˘âŻImmersive viewing, live annotation
Reichherzer, C., Nassani, A., & Billinghurst, M. (2014). Social panoramas using wearable
computers. In Mixed and Augmented Reality (ISMAR), 2014 IEEE International Symposium on
(pp. 303-304). IEEE.
33. Example: AR remote collaboration
â˘âŻLocal user uses AR display
â˘âŻ Move real objects using AR cues
â˘âŻRemote expert on desktop interface
â˘âŻ Place 3D objects with independent view
Tait, M., & Billinghurst, M. The Effect of View Independence in a Collaborative AR System.
Computer Supported Cooperative Work (CSCW), 1-27.
34. Research Needed in Many Areas
â˘âŻSocial Acceptance
â˘âŻ Overcome social problems with AR
â˘âŻCloud Services
â˘âŻ Cloud based storage/processing
â˘âŻUbiquitous AR
â˘âŻ Using AR with Ubicomp/IoT technologies
â˘âŻCollaborative Experiences
â˘âŻ AR teleconferencing
â˘âŻEtc..
35. SocialAcceptance
â˘âŻPeople donât want to look silly
â˘âŻ Only 12% of 4,600 adults would be willing to wear AR glasses
â˘âŻ 20% of mobile AR browser users experience social issues
â˘âŻAcceptance more due to Social than Technical issues
â˘âŻ Needs further study (ethnographic, field tests, longitudinal)
37. Conclusions
â˘âŻAR is becoming commonly available
â˘âŻIn order to achieve significant growth AR needs to
â˘âŻ Expand into new markets
â˘âŻ Move onto new platforms
â˘âŻ Create new types of applications
â˘âŻNew AR technologies will enable this to happen
â˘âŻ Display, Interaction, Tracking technologies
â˘âŻHowever there are still significant areas for research
â˘âŻ Social Acceptance, Cloud Services, Ubiquitous AR, Etc