Application in Augmented and Virtual Reality
•
9 likes•4,177 views
Talk given by Mark Billinghurst to Bajaj Finance Limited in India, on May 9th 2020. The talk describes AR and VR applications, example AR/VR applications in financial services, and potential research directions.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Application in Augmented and Virtual Reality
Similar to Application in Augmented and Virtual Reality (20)
More from Mark Billinghurst
More from Mark Billinghurst (12)
Recently uploaded
Recently uploaded (20)
Application in Augmented and Virtual Reality
- 1. APPLICATIONS IN AUGMENTED AND VIRTUAL REALITY Mark Billinghurst mark.billinghurst@auckland.ac.nz Bajaj Finance Limited, May 9th 2020
- 2. IBM 360 - $10,000 a month to rent in 1965
- 3. The Incredible Disappearing Computer 1960-70’s Room 1970-80’s Desk 1980-90’s Lap 1990-2000’s Hand
- 6. Graphical User Interfaces • Separation between real and digital worlds • WIMP (Windows, Icons, Menus, Pointer) metaphor
- 7. Ubiquitous Computing/IoT • Embed computing and sensing in real world • Smart objects, sensors, etc..
- 8. Virtual Reality • Users immersed in Computer Generated environment • HMD, gloves, 3D graphics, body tracking
- 10. Virtual Reality •Defining Characteristics •Immersion •Interaction •Imagination Grigore Burdea (1993). Virtual Reality Technology
- 11. Augmented Reality • Virtual Images blended with the real world • See-through HMD, handheld display, viewpoint tracking, etc..
- 13. Augmented Reality •Defining Characteristics •Combines Real andVirtual Images •Interactive in real-time •Registered in 3D Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.
- 14. AR vsVR
- 15. Rekimoto, J. and Nagao, K. 1995. The world through the computer: computer augmented interaction with real world environments. Making Interfaces Invisible
- 16. From Reality toVirtual Reality Ubiquitous Computing Augmented Reality Virtual Reality Real World Virtual World
- 17. Milgram’s Mixed Reality Continuum Ubiquitous Computing Augmented Reality Virtual Reality Real World Virtual World Mixed Reality "...anywhere between the extrema of the virtuality continuum." P. Milgram and A. F. Kishino, (1994) A Taxonomy of Mixed Reality Visual Displays
- 19. Mixed Reality in 2020 • Large growing market • $11B in 2017, $19B in 2020 • Many available devices • HMD, phones, tablets, HUDs • Robust developer tools • Vuforia, WebGL, Unity, Wikitude, etc • Large number of applications • > 250K developers, > 100K mobile apps • Strong research/business communities • ISMAR, AWE, AugmentedReality.org, IEEE VR, etc
- 20. Market Growth
- 21. Market Breakdown
- 23. Using Mixed Reality in the Workplace • Blending digital/physical worlds.. • Productivity • Marketing • Training • Collaboration
- 24. Example: Productivity • MR technology for hands busy tasks • Warehouse picking (DHL, SAP) • Smart glasses show pick location • Simple user interface • Hands-free interaction • Improved performance RW VR
- 28. Example: Marketing • Add magic to company products • Snapchat lenses • Face tracking technology • Simple interaction • Animated content • Significant ROI • $750K for 224 million views • c.f. $5 million for SuperBowl ad RW VR
- 30. Example: Training • AR/VR training for difficult/dangerous tasks • Train with real objects, controls • GlobalSim Crane Simulator • Virtual Reality simulation • Train with real controls • Virtual body/hands • Realistic physics RW VR
- 32. Example: Collaboration • Use MR technology for remote collaboration • See through another’s eyes, virtual avatars • Microsoft Remote Assist • HoloLens see-through display • Remote video sharing • AR drawing/annotation • Document sharing • Group calling RW VR
- 34. FINTECH AR AND VR APP
- 35. Introducing AR/VR into the Workplace •Identify key real-world tasks • Involve spatial ability, physical interaction, etc. •Justify Return on Investment • Small technology investment leads to big return •Focus on Human Needs • User centered design •Work with Experts • Combine AR/VR experts with domain experts
- 36. Application Areas in Financial Services • Customer Experience • Customer Support • Agent Training • On-site Agent Support • Data Visualization • Meeting Support • Etc..
- 37. Enhancing Customer Experience – Westpac NZ https://www.youtube.com/watch?v=_dnKD2m_mPE
- 38. Customer Support – Virtual Agent – FidelityLabs https://www.youtube.com/watch?v=FwNsDQLiazU
- 39. Customer Support – Remote Insurance Claims https://www.youtube.com/watch?v=S59Xttnp2RY
- 40. Agent Training – Insurance Agents https://www.youtube.com/watch?v=IlSt4p5XTmY
- 41. On Site Agent Support – Building Assessment https://www.youtube.com/watch?v=a3_AtLFDisE
- 42. Data Visualization - Citibank https://www.youtube.com/watch?v=0NogltmewmQ
- 43. Meeting Support - Spatial https://www.youtube.com/watch?v=PG3tQYlZ6JQ
- 45. Key Enabling Technologies • Combines Real and Virtual Images • Display Technology • Interactive in real-time • Interaction Technologies • Registered in 3D • Tracking Technologies
- 46. • Past • Bulky Head mounted displays • Current • Handheld, lightweight head mounted • Future • Wide FOV see through • Retinal displays • Contact lens Display Technology
- 47. Wide FOV See-Through (3+ years) •Waveguide techniques • Thin, wider FOV • Socially acceptable •Pinlight Displays • LCD panel + point light sources • 110 degree FOV Lumus DK40
- 49. Retinal Displays (5+ years) • Photons scanned into eye • Infinite depth of field • Bright outdoor performance • True 3D stereo with depth modulation • Microvision (1993-) • Head mounted monochrome • MagicLeap (2013-) • Projecting light field into eye
- 50. Contact Lens (10 – 15 + years) • Contact Lens only • Unobtrusive • Significant technical challenges • Babak Parviz (2008) • Mojo Vision • https://www.mojo.vision/ • 14K ppi display • Over $100m investment http://spectrum.ieee.org/biomedical/bionics/augmented-reality-in-a-contact-lens/
- 51. Interaction • Past • Limited interaction • Viewpoint manipulation • Present • Screen based, simple gesture • Tangible interaction • Future • Natural gesture, Multimodal • Intelligent Interfaces, Sensor based
- 52. Natural Gesture (2+ years) • Freehand gesture input • Depth sensors for gesture capture • Rich two handed gestures • E.g. Microsoft Research Hand Tracker • 3D hand tracking, 30 fps, single sensor • Commercial Systems • Hololens2, Occulus, Leap Motion, Intel, etc..
- 54. Multimodal Input (5-10+ years) • Combine gesture and speech input • Gesture good for qualitative input • Speech good for quantitative input • Support combined commands • “Put that there” + pointing • E.g. HIT Lab NZ multimodal input • 3D hand tracking, speech, multimodal fusion • Complete tasks faster with MMI, less errors Billinghurst, M., Piumsomboon, T., & Bai, H. (2014). Hands in Space: Gesture Interaction with Augmented-Reality Interfaces. IEEE computer graphics and applications, (1), 77-80.
- 55. HIT Lab NZ Multimodal Input
- 56. Tracking • Past • Location-based, marker based • magnetic/mechanical • Present • Image based, hybrid tracking • Future • Ubiquitous • Model based • Environmental
- 57. Environmental Tracking (1-3+ yrs) • Environment capture • Use depth sensors to capture scene & track from model • InifinitAM (www.robots.ox.ac.uk/~victor/infinitam/) • Real time scene capture on mobiles, dense or sparse capture • Dynamic memory swapping allows large environment capture
- 59. 6D.ai AR Cloud Tracking https://www.youtube.com/watch?v=AwwU14gllS0
- 60. Wide Area Outdoor Tracking (5 + yrs) • Combine panorama’s into point cloud model (offline) • Initialize camera tracking from point cloud • Update pose by aligning camera image to point cloud • Accurate to 25 cm, 0.5 degree over very wide area Ventura, J., & Hollerer, T. (2012). Wide-area scene mapping for mobile visual tracking. In Mixed and Augmented Reality (ISMAR), 2012 IEEE International Symposium on (pp. 3-12). IEEE.
- 61. Wide Area Outdoor Tracking
- 62. Example – Holoscape Outdoor AR Game https://www.youtube.com/watch?v=Ndjdp9I6tFs
- 63. Other Research Areas • Social Acceptance • Overcome social problems with AR • Cloud Services • Cloud based storage/processing • AR Authoring Tools • Easy content creation for non-experts • Collaborative Experiences • AR teleconferencing • Etc..
- 64. Example:SocialAcceptance • People don’t want to look silly • Only 12% of 4,600 adults would be willing to wear AR glasses • 20% of mobile AR browser users experience social issues • Acceptance more due to Social than Technical issues • Needs further study (ethnographic, field tests, longitudinal)
- 66. Modern Technology Trends 1. Improved Content Capture • Move from sharing faces to sharing places 2. Increased Network Bandwidth • Sharing natural communication cues 3. Implicit Understanding • Recognizing behaviour and emotion
- 69. Empathic Computing Can we develop systems that allow us to share what we are seeing, hearing and feeling with others?
- 70. Empathy Glasses • Combine together eye-tracking, display, face expression • Implicit cues – eye gaze, face expression ++ Pupil Labs Epson BT-200 AffectiveWear
- 71. Remote Collaboration • Eye gaze pointer and remote pointing • Face expression display • Implicit cues for remote collaboration
- 73. Shared Sphere – 360 Video Sharing Shared Live 360 Video Host User Guest User
- 75. Shared Sphere System Features •AR / VR view collaboration • VR user feels that they are in AR users location •View independence • Allow one user to look around independently from another •View awareness cues • Show where the other user is looking •Hand gesture communication cues • Sharing hand gesture cues
- 76. 3D Live Scene Capture • Use cluster of RGBD sensors • Fuse together 3D point cloud
- 78. View Sharing Evolution • Increased immersion • Improved scene understanding • Better collaboration 2D 360 3D
- 79. Technology Trends • Advanced displays • Wide FOV, high resolution • Real time space capture • 3D scanning, stitching, segmentation • Natural gesture interaction • Hand tracking, pose recognition • Robust eye-tracking • Gaze points, focus depth • Emotion sensing/sharing • Physiological sensing, emotion mapping
- 80. Emotion Sensing HMDs are Coming •Looxid • EEG/Emotion sensing • Eye tracking •Emteq • EMG sensing •Neurable • EEG/BCI for VR
- 81. Technology Trends •Advanced displays •Real time space capture •Natural gesture interaction •Robust eye-tracking •Emotion sensing/sharing Empathic Tele-Existence
- 82. Empathic Tele-Existence • Move from Observer to Participant • Explicit to Implicit communication • Experiential collaboration – doing together
- 83. CONCLUSIONS
- 84. Conclusions •AR/VR in the workspace •Breaks barrier between real/digital worlds •Proven technology (display, input, sensing) •Ready for implementation •Future Research •Display, tracking, interaction, user experience •Empathic Computing