Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Fifty Shades of Augmented Reality: Creating Connection Using AR

3,081 views

Published on

Keynote speech by Mark Billinghurst at the Laval Virtual 2017 conference on March 24th 2017. The presentation talks about how Augmented Reality can be used to enhance remote collaboration.

Published in: Technology

Fifty Shades of Augmented Reality: Creating Connection Using AR

  1. 1. FIFTY SHADES OF AUGMENTED REALITY: CREATING CONNECTION USING AR Mark Billinghurst mark.billinghurst@unisa.edu.au March 24th 2017 Laval Virtual
  2. 2. • jg
  3. 3. Ancient Communication Tools
  4. 4. Evolution of Communication Tools
  5. 5. TECHNOLOGY TRENDS
  6. 6. Modern Technology Trends 1. Improved Content Capture • Move from sharing faces to sharing places 2. Increased Network Bandwidth • Sharing natural communication cues 3. Implicit Understanding • Recognizing behaviour and emotion These three trends combined with AR/MR technology can create new types of collaborative experiences
  7. 7. Content Capture Realism Time Photo Film Live Video Panorama 360 Video 3D Space 1850 1900 1940 1990 2000 2010
  8. 8. Capturing Reality • Panoramas (1790s) • Immersive paintings • Photography (1820-30s) • Oldest surviving photo (Niépce, 1826) • Stereo imagery (1830s) • Wheatstone (1832) • Brewster (1851) • Movies (1870s) • Muybridge (1878) • Roundhay Garden Scene (1888)
  9. 9. Sharing Faces to Sharing Spaces • Moving from desktop to handheld cameras
  10. 10. Capturing 360 images Kodak 360 Fly 360 Gear 360 Theta S Nikon LG 360 Pointgrey Ladybug Panono 360 Bublcam
  11. 11. 360 Stereo Capture Samsung Project Beyond Jaunt VR camera https://www.youtube.com/watch?v=FnBIvq8jzds
  12. 12. 3D Space Capture Google Project Tango Occipital Structure Sensor https://www.youtube.com/watch?v=XA7FMoNAK9M
  13. 13. Content Capture Realism Time Photo Film Live Video Panorama 360 Video 3D Space 1850 1900 1940 1990 2000 2010 2D Static Immersive Live Experience
  14. 14. Networking Speeds Log (b/s) Time 100 b/s 10 Kb/s 1 Mb/s 1980 1985 1990 1995 2000 2010 100 Mb/s 2005
  15. 15. Networking Technology • Telephone (1876) • Bell Picture Phone (1955) • B+W live video calls • Mobile Phones (1985) • Desktop Video Conferencing (1990’s) • Netmeeting, Skype https://www.youtube.com/watch?v=OBkcYyenNBM
  16. 16. Mobile Conferencing (2010-) • Video conferencing • Skype, facebook • Live broadcasting • Meerkat, Periscope • Support for 360 video
  17. 17. Network Innovation Universal Connectivity
  18. 18. Networking Speeds Log (b/s) Time 100 b/s 10 Kb/s 1 Mb/s 1980 1985 1990 1995 2000 2010 100 Mb/s 2005 Text Audio Natural Video
  19. 19. Interaction Technology Natural Time Punch Card Keyboard Mouse Speech Gesture Emotion 1950 1960 1980 1990 2000 2010 Thought
  20. 20. Natural Gesture Input • Freehand input • Depth sensors, rich two handed gestures Sharp, T., Keskin, C., Robertson, D., Taylor, J., Shotton, J., Leichter, D. K. C. R. I., ... & Izadi, S. (2015, April). Accurate, Robust, and Flexible Real-time Hand Tracking. In Proc. CHI (Vol. 8). https://www.youtube.com/watch?v=A-xXrMpOHyc
  21. 21. Physiological Sensing Emotiv Empatica
  22. 22. Interaction Technology Natural Time Punch Card Keyboard Mouse Speech Gesture Emotion 1950 1960 1980 1990 2000 2010 Thought Implicit Explicit
  23. 23. MIXED REALITY CONFERENCING
  24. 24. Milgram’s Reality-Virtuality Continuum Mixed Reality Reality - Virtuality (RV) Continuum Real Environment Augmented Reality (AR) Augmented Virtuality (AV) Virtual Environment "...anywhere between the extrema of the virtuality continuum." P. Milgram and A. F. Kishino, Taxonomy of Mixed Reality Visual Displays IEICE Transactions on Information and Systems, E77-D(12), pp. 1321-1329, 1994.
  25. 25. Mixed Reality Conferencing in SciFi
  26. 26. AR Video Conferencing (2001) • Bringing conferencing into real world • Using AR video textures of remote people • Attaching AR video to real objects Billinghurst, M., & Kato, H. (2002). Collaborative augmented reality. Communications of the ACM, 45(7), 64-70.
  27. 27. Demo: AR Video Conferencing
  28. 28. MR Point Cloud Sharing (2016) • Use video-see through HMD • Occulus Rift • Depth camera on HMD • Softkinetic • Create point cloud • Remote hand gestures • User’s space
  29. 29. Demo: Point Cloud Capture
  30. 30. Holoportation (2016) • Augmented Reality + 3D capture + high bandwidth • http://research.microsoft.com/en-us/projects/holoportation/
  31. 31. Holoportation Video https://www.youtube.com/watch?v=7d59O6cfaM0
  32. 32. Social VR • Facebook Social Virtual Reality, AltspaceVR • Bringing Avatars into VR space • Natural social interaction
  33. 33. Demo: Facebook Social VR (2016) https://www.youtube.com/watch?v=yxHwWHHg4Vs
  34. 34. AR/VR Collaboration (2017) • Make 3D copy of real space • AR user in real world, VR user in 3D copy of real space • Hololens for AR user, HTC Vive for VR user • Share virtual body cues (head, hands, gaze information) Real World Virtual World
  35. 35. Virtual Cues • AR/VR displays • Gesture input • Leap Motion • Room scale tracking
  36. 36. Demo: AR/VR Conferencing
  37. 37. EMPATHIC COMPUTING
  38. 38. Natural Collaboration Implicit Understanding Experience Capture
  39. 39. Natural Collaboration Implicit Understanding Experience Capture Empathic Computing
  40. 40. Empathy “Seeing with the Eyes of another, Listening with the Ears of another, and Feeling with the Heart of another..” Alfred Adler
  41. 41. Empathic Computing Systems that allow us to share what we are seeing, hearing and feeling with others..
  42. 42. Example: Google Glass • Camera + Processing + Display + Connectivity • Ego-Vision Collaboration (But with FixedView)
  43. 43. Current Collaboration onWearables • First person remote conferencing/hangouts • Limitations • Single POV, no spatial cues, no annotations, etc
  44. 44. Social Panoramas (ISMAR 2014) • Capture and share social spaces in real time • Supports independent views into Panorama Reichherzer, C., Nassani, A., & Billinghurst, M. (2014, September). [Poster] Social panoramas using wearable computers. In Mixed and Augmented Reality (ISMAR), 2014 IEEE International Symposium on (pp. 303-304). IEEE.
  45. 45. Implementation • Google Glass • Capture live image panorama (compass + camera) • Remote device (tablet) • Immersive viewing, live annotation
  46. 46. User Interfaces Glass View Tablet View
  47. 47. Social Panorama Demo https://www.youtube.com/watch?v=vdC0-UV3hmY&t=1s
  48. 48. JackIn – Live Immersive Video Streaming • Jun Rekimoto – University of Tokyo/Sony CSL
  49. 49. JackIn Hardware • Wide angle cameras – 360 degree video capture • Live video stitching
  50. 50. JackIn Demo https://www.youtube.com/watch?v=Lkuz6LhlBf8
  51. 51. • zxcz
  52. 52. JospehTame – Tokyo Marathon • Live streaming from Tokyo marathon • http://josephta.me/en/tokyo-marathon/
  53. 53. Empathy Glasses (CHI 2016) • Combine together eye-tracking, display, face expression • Impicit cues – eye gaze, face expression ++ Pupil Labs Epson BT-200 AffectiveWear Masai, K., Sugimoto, M., Kunze, K., & Billinghurst, M. (2016, May). Empathy Glasses. In Proceedings of the 34th Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems. ACM.
  54. 54. AffectiveWear – Emotion Glasses • Photo sensors to recognize expression • User calibration • Machine learning • Recognizing 8 face expressions
  55. 55. Empathy Glasses in Use • Eye gaze pointer and remote pointing • Face expression display • In future integrated eye-tracking/display
  56. 56. Empathy Glasses Demo https://www.youtube.com/watch?v=CdgWVDbMwp4
  57. 57. Empathic VR Environments (2017) • Player and Viewer • Viewer slaved to player • Share emotional signals • Heart rate, GSR • Remote affect measuring
  58. 58. Sharing VR Experiences • HTC Vive HMD • Empathic glove • Empatica E4
  59. 59. VR Environments • Butterfly World: calm scene, collect butterflies • Zombie Attack: scary scene, fighting zombies
  60. 60. Demo Video https://www.youtube.com/watch?v=SaiHxps-Ofw
  61. 61. AR and VR for Empathic Computing • VR systems are ideal for trying experiences: • Strong story telling medium • Provide total immersion/3D experience • Easy to change virtual body scale and representation • AR systems are idea for live sharing: • Allow overlay on real world view/can share viewpoints • Support remote annotation/communication • Enhance real world task
  62. 62. CONCLUSIONS
  63. 63. Conclusions •Trend towards Empathic Computing • Understanding, Experiencing, Sharing •AR/VR Enables Empathic Experiences • Changes perspective • Sharing space/experience • Supports annotation/communication •Many directions for future research
  64. 64. www.empathiccomputing.org @marknb00 mark.billinghurst@unisa.edu.au

×