Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Creating Augmented Reality Apps with Web Technology

816 views

Published on

If 2017 was the year of Virtual Reality, 2018 may become the year of Augmented Reality. With Apple's ARKit for iOS and Google's ARCore for Android, we should not forget that actually the open web and Augmented Reality are the perfect couple.

But how do we create AR apps with web technology? In my talk you will learn how to write your own AR app that runs in the browser. You will see how Augmented Reality works, what markers are for, and how to create them. And you'll get an overview about the devices you can use to test your AR projects.

We are just at the beginning of the rise of Augmented Reality. This is why lots of UX design questions are still unanswered. But there are patterns and best practises, and you will learn how to use them to get a great user experience in your AR application.

Published in: Technology
  • Be the first to comment

Creating Augmented Reality Apps with Web Technology

  1. 1. @fischaelameergeildanke.com @SCRIPT18 Augmented Reality with Web Technology
  2. 2. AR
  3. 3. VR AR MR
  4. 4. Virtual Reality
  5. 5. Virtual Reality completely digital
  6. 6. Augmented Reality virtual overlay
  7. 7. Mixed Reality virtual and real world coincide
  8. 8. @fischaelameergeildanke.com @SCRIPT18 A Short History of Augmented Reality
  9. 9. Ivan Sutherland
  10. 10. Ivan Sutherland’s The Sword of Damocles
  11. 11. Ivan Sutherland’s The Sword of Damocles
  12. 12. Sword of Damocles, 1812 Richard Westall
  13. 13. https://www.youtube.com/watch?v=ISJWZpFIAlQ
  14. 14. Tom Caudell
  15. 15. ❝ Augmented Reality (AR) is a variation of Virtual Environments (VE), or Virtual Reality as it is more commonly called. ❞ Ronald Azura, 1997
  16. 16. ❝ VE technologies completely immerse 
 a user inside a synthetic environment. While immersed, the user cannot see the 
 real world around him. ❞ Ronald Azura, 1997
  17. 17. ❝ In contrast, AR allows the user to see the real world, with virtual objects superimposed upon or composited with the real world. Therefore, AR supplements reality, rather than completely replacing it. ❞ Ronald Azura, 1997
  18. 18. The Touring Machine
  19. 19. @fischaelameergeildanke.com @SCRIPT18 Technology and Use Cases of Augmented Reality Today
  20. 20. Innovation Trigger
  21. 21. Innovation Trigger Peak of Inflated Expectations
  22. 22. Innovation Trigger Peak of Inflated Expectations Trough of Disillusionment
  23. 23. Innovation Trigger Peak of Inflated Expectations Trough of Disillusionment Slope of
 Enlightenment
  24. 24. Innovation Trigger Peak of Inflated Expectations Trough of Disillusionment Slope of
 Enlightenment Plateau of Productivity
  25. 25. Innovation Trigger Peak of Inflated Expectations Trough of Disillusionment Slope of
 Enlightenment Plateau of Productivity Machine Learning Blockchain Augmented Reality Virtual Reality
  26. 26. Camera Tracking GPS Screen
  27. 27. PositionRotation
  28. 28. https://www.microsoft.com/de-de/hololens
  29. 29. http://www.metavision.com/
  30. 30. https://www3.lenovo.com/at/de/tango/
  31. 31. https://developer.apple.com/arkit/
  32. 32. https://developers.google.com/ar/
  33. 33. https://play.google.com/store/apps/details?id=com.evogames.stackitar
  34. 34. https://play.google.com/store/apps/details?id=com.roomle.android
  35. 35. https://play.google.com/store/apps/details?id=com.signalgarden.atomvisualizer
  36. 36. https://www.volkswagenag.com/de/group/research/virtual-technologies.html
  37. 37. https://www.linkedin.com/pulse/20140502184728-1282856-guide-to-augmented-reality-in-healthcare/
  38. 38. https://vrodo.de/augmented-reality-mit-hololens-microsoft-zeigt-holoportation/
  39. 39. https://techcrunch.com/2015/07/29/google-translates-app-now-instantly-translates-printed-text-in-27-languages/
  40. 40. https://www.urdesignmag.com/technology/2015/01/26/rideon-augmented-reality-ski-goggles/
  41. 41. @fischaelameergeildanke.com @SCRIPT18 Augmented Reality and the Web?
  42. 42. Write Once, Runs Anywhere
  43. 43. Write Once, Runs Anywhere No App Download Required
  44. 44. Camera & Display
  45. 45. Camera & Display Tracking Sensors
  46. 46. Camera & Display Tracking Sensors World Recognition
  47. 47. WebGL API & WebVR API
  48. 48. WebGL API & WebVR API Media Capture and Streams API
  49. 49. Marker-less AR vs. Marker-based AR
  50. 50. https://aframe.io/
  51. 51. @fischaelameergeildanke.com @SCRIPT18 Markerless Augmented Reality with ARCore and ARKit
  52. 52. ARKit by Apple
  53. 53. ARKit ARCore by Apple by Google
  54. 54. ARKit ARCore by Apple by Google World/Motion Tracking Scene Analyzation Light Estimation
  55. 55. ARKit ARCore by Apple by Google WebAROnARKit WebAROnARCore Two Experimental Browser Apps https://github.com/google-ar/WebARonARKit https://github.com/google-ar/WebARonARCore
  56. 56. ARKit ARCore by Apple by Google three.ar.js three.js for AR
  57. 57. <script src="aframe-master.min.js"></script> <script src="three.ar.js"></script> <script src="aframe-ar.js"></script> <script src="3dio.min.js"></script> <script src="main.js"></script>
  58. 58. <script src="aframe-master.min.js"></script> <script src="three.ar.js"></script> <script src="aframe-ar.js"></script> <script src="3dio.min.js"></script> <script src="main.js"></script>
  59. 59. <script src="aframe-master.min.js"></script> <script src="three.ar.js"></script> <script src="aframe-ar.js"></script> <script src="3dio.min.js"></script> <script src="main.js"></script>
  60. 60. <script src="aframe-master.min.js"></script> <script src="three.ar.js"></script> <script src="aframe-ar.js"></script> <script src="3dio.min.js"></script> <script src="main.js"></script>
  61. 61. <script src="aframe-master.min.js"></script> <script src="three.ar.js"></script> <script src="aframe-ar.js"></script> <script src="3dio.min.js"></script> <script src="main.js"></script>
  62. 62. https://furniture.3d.io/
  63. 63. <script src="aframe-master.min.js"></script> <script src="three.ar.js"></script> <script src="aframe-ar.js"></script> <script src="3dio.min.js"></script> <script src="main.js"></script>
  64. 64. <body> <a-scene ar> <a-entity io3d-furniture="id:123" visible="false"></a-entity> <a-sphere id="intersection" visible="false" radius="0.04" color="yellow"></a-sphere> <a-camera> <a-entity cursor="fuse:false" raycaster ar-raycaster></a-entity> <a-entity position="0 0 -0.1" scale="0.001 0.001 0.001" geometry="primitive: ring; radiusInner: 0.8; radiusOuter: 1" material="color: yellow; shader: flat; transparent:true"> </a-entity> </a-camera> </a-scene> </body>
  65. 65. <body> <a-scene ar> <a-entity io3d-furniture="id:123" visible="false"></a-entity> <a-sphere id="intersection" visible="false" radius="0.04" color="yellow"></a-sphere> <a-camera> <a-entity cursor="fuse:false" raycaster ar-raycaster></a-entity> <a-entity position="0 0 -0.1" scale="0.001 0.001 0.001" geometry="primitive: ring; radiusInner: 0.8; radiusOuter: 1" material="color: yellow; shader: flat; transparent:true"> </a-entity> </a-camera> </a-scene> </body>
  66. 66. let scene = new THREE.Scene(); let camera = new THREE.PerspectiveCamera(75, window.innerWidth / window.innerHeight, 0.1, 10000); let renderer = new THREE.WebGLRenderer(); renderer.setSize(window.innerWidth, window.innerHeight); document.body.appendChild(renderer.domElement); function render() { requestAnimationFrame(render); renderer.render(scene, camera); } render();
  67. 67. <body> <a-scene ar> <a-entity io3d-furniture="id:123" visible="false"></a-entity> <a-sphere id="intersection" visible="false" radius="0.04" color="yellow"></a-sphere> <a-camera> <a-entity cursor="fuse:false" raycaster ar-raycaster></a-entity> <a-entity position="0 0 -0.1" scale="0.001 0.001 0.001" geometry="primitive: ring; radiusInner: 0.8; radiusOuter: 1" material="color: yellow; shader: flat; transparent:true"> </a-entity> </a-camera> </a-scene> </body>
  68. 68. <body> <a-scene ar> <a-entity io3d-furniture="id:123" visible="false"></a-entity> <a-sphere id="intersection" visible="false" radius="0.04" color="yellow"></a-sphere> <a-camera> <a-entity cursor="fuse:false" raycaster ar-raycaster></a-entity> <a-entity position="0 0 -0.1" scale="0.001 0.001 0.001" geometry="primitive: ring; radiusInner: 0.8; radiusOuter: 1" material="color: yellow; shader: flat; transparent:true"> </a-entity> </a-camera> </a-scene> </body>
  69. 69. <body> <a-scene ar> <a-entity io3d-furniture="id:123" visible="false"></a-entity> <a-sphere id="intersection" visible="false" radius="0.04" color="yellow"></a-sphere> <a-camera> <a-entity cursor="fuse:false" raycaster ar-raycaster></a-entity> <a-entity position="0 0 -0.1" scale="0.001 0.001 0.001" geometry="primitive: ring; radiusInner: 0.8; radiusOuter: 1" material="color: yellow; shader: flat; transparent:true"> </a-entity> </a-camera> </a-scene> </body>
  70. 70. <body> <a-scene ar> <a-entity io3d-furniture="id:123" visible="false"></a-entity> <a-sphere id="intersection" visible="false" radius="0.04" color="yellow"></a-sphere> <a-camera> <a-entity cursor="fuse:false" raycaster ar-raycaster></a-entity> <a-entity position="0 0 -0.1" scale="0.001 0.001 0.001" geometry="primitive: ring; radiusInner: 0.8; radiusOuter: 1" material="color: yellow; shader: flat; transparent:true"> </a-entity> </a-camera> </a-scene> </body>
  71. 71. <body> <a-scene ar> <a-entity io3d-furniture="id:123" visible="false"></a-entity> <a-sphere id="intersection" visible="false" radius="0.04" color="yellow"></a-sphere> <a-camera> <a-entity cursor="fuse:false" raycaster ar-raycaster></a-entity> <a-entity position="0 0 -0.1" scale="0.001 0.001 0.001" geometry="primitive: ring; radiusInner: 0.8; radiusOuter: 1" material="color: yellow; shader: flat; transparent:true"> </a-entity> </a-camera> </a-scene> </body>
  72. 72. <body> <a-scene ar> <a-entity io3d-furniture="id:123" visible="false"></a-entity> <a-sphere id="intersection" visible="false" radius="0.04" color="yellow"></a-sphere> <a-camera> <a-entity cursor="fuse:false" raycaster ar-raycaster></a-entity> <a-entity position="0 0 -0.1" scale="0.001 0.001 0.001" geometry="primitive: ring; radiusInner: 0.8; radiusOuter: 1" material="color: yellow; shader: flat; transparent:true"> </a-entity> </a-camera> </a-scene> </body>
  73. 73. <body> <a-scene ar> <a-entity io3d-furniture="id:123" visible="false"></a-entity> <a-sphere id="intersection" visible="false" radius="0.04" color="yellow"></a-sphere> <a-camera> <a-entity cursor="fuse:false" raycaster ar-raycaster></a-entity> <a-entity position="0 0 -0.1" scale="0.001 0.001 0.001" geometry="primitive: ring; radiusInner: 0.8; radiusOuter: 1" material="color: yellow; shader: flat; transparent:true"> </a-entity> </a-camera> </a-scene> </body>
  74. 74. let scene = document.querySelector('a-scene'); let raycaster = document.querySelector('[ar-raycaster]'); let mark = document.querySelector('#intersection'); raycaster.addEventListener('raycaster-intersection', function(e) { mark.setAttribute('position', e.detail.intersections[0].point); mark.setAttribute('visible', true); }); raycaster.addEventListener('raycaster-intersection-cleared', function(e) { mark.setAttribute('visible', false); });
  75. 75. window.addEventListener('click', function () { let furniture = document.querySelector('[io3d-furniture]'); let position = mark.getAttribute('position'); furniture.setAttribute('position', position); furniture.setAttribute('visible', true); });
  76. 76. https://3d.io/
  77. 77. @fischaelameergeildanke.com @SCRIPT18 Marker-based AR with AR.js
  78. 78. https://poly.google.com/view/77wHkzwlpOq
  79. 79. <script src="aframe.min.js"></script> <script src="aframe-ar.js"></script> <script src="annyang.min.js"></script> <script src="aframe-speech-command-component.js"></script>
  80. 80. <script src="aframe.min.js"></script> <script src="aframe-ar.js"></script> <script src="annyang.min.js"></script> <script src="aframe-speech-command-component.js"></script>
  81. 81. <script src="aframe.min.js"></script> <script src="aframe-ar.js"></script> <script src="annyang.min.js"></script> <script src="aframe-speech-command-component.js"></script>
  82. 82. <script src="aframe.min.js"></script> <script src="aframe-ar.js"></script> <script src="annyang.min.js"></script> <script src="aframe-speech-command-component.js"></script>
  83. 83. <a-scene arjs="trackingMethod:best;"> <a-assets>[…]</a-assets> <a-anchor hit-testing-enabled="true">[…]</a-anchor> <a-camera-static/> </a-scene>
  84. 84. <a-assets> <a-asset-item id="sun-obj" src="assets/sun.obj"></a-asset-item> <a-asset-item id="sun-mtl" src="assets/sun.mtl"></a-asset-item> </a-assets>
  85. 85. <a-anchor hit-testing-enabled="true"> <a-box position="0 0.5 0" material="opacity:0;"> <a-entity obj-model="obj:#sun-obj; mtl:#sun-mtl" scale="0.2 0.2 0.2" visible="false"> <a-animation attribute="rotation" to="360 0 0" dur="5000" easing="linear" repeat="indefinite"> </a-animation> </a-entity> </a-box> </a-anchor>
  86. 86. <a-anchor hit-testing-enabled="true"> <a-box position="0 0.5 0" material="opacity:0;"> <a-entity obj-model="obj:#sun-obj; mtl:#sun-mtl" scale="0.2 0.2 0.2" visible="false"> <a-animation attribute="rotation" to="360 0 0" dur="5000" easing="linear" repeat="indefinite"> </a-animation> </a-entity> </a-box> </a-anchor>
  87. 87. <a-anchor hit-testing-enabled="true"> <a-box position="0 0.5 0" material="opacity:0;"> <a-entity obj-model="obj:#sun-obj; mtl:#sun-mtl" scale="0.2 0.2 0.2" visible="false"> <a-animation attribute="rotation" to="360 0 0" dur="5000" easing="linear" repeat="indefinite"> </a-animation> </a-entity> </a-box> </a-anchor>
  88. 88. <a-anchor hit-testing-enabled="true"> <a-box position="0 0.5 0" material="opacity:0;"> <a-entity obj-model="obj:#sun-obj; mtl:#sun-mtl" scale="0.2 0.2 0.2" visible="false"> <a-animation attribute="rotation" to="360 0 0" dur="5000" easing="linear" repeat="indefinite"> </a-animation> </a-entity> </a-box> </a-anchor>
  89. 89. <a-entity id="annyang" annyang-speech-recognition></a-entity> <a-anchor hit-testing-enabled="true"> <a-box position="0 0.5 0" material="opacity:0;"> <a-entity obj-model="obj:#sun-obj; mtl:#sun-mtl" scale="0.2 0.2 0.2" visible="false"> <a-animation attribute="rotation" to="360 0 0" dur="5000" easing="linear" repeat="indefinite"> </a-animation> </a-entity> </a-box> </a-anchor>
  90. 90. <a-entity id="annyang" annyang-speech-recognition></a-entity> <a-anchor hit-testing-enabled="true"> <a-box position="0 0.5 0" material="opacity:0;"> <a-entity obj-model="obj:#sun-obj; mtl:#sun-mtl" scale="0.2 0.2 0.2" visible="false" speech-command__show="command: sun; type: attribute; attribute: visible; value: true;" speech-command__hide="command: hide; type: attribute; attribute: visible; value: false;"> <a-animation attribute="rotation" to="360 0 0" dur="5000" easing="linear" repeat="indefinite"> </a-animation> </a-entity> </a-box> </a-anchor>
  91. 91. https://poly.google.com/view/77wHkzwlpOq
  92. 92. https://jeromeetienne.github.io/AR.js/three.js/examples/arcode.html
  93. 93. @fischaelameergeildanke.com @SCRIPT18 UX Design for Augmented Reality
  94. 94. @fischaelameergeildanke.com @SCRIPT18 No Established Patterns Yet
  95. 95. Interpretability Comfort Usefulness Delight https://medium.com/@beaucronin/the-hierarchy-of-needs-in-virtual-reality-development-4333a4833acc
  96. 96. Interpretability Comfort Safety Usefulness Delight https://medium.com/@beaucronin/the-hierarchy-of-needs-in-virtual-reality-development-4333a4833acc
  97. 97. User Input and Interaction User Interface and Information Interpretability Context Comfort Safety Illusion Usefulness Delight https://medium.com/@beaucronin/the-hierarchy-of-needs-in-virtual-reality-development-4333a4833acc
  98. 98. VoiceGestures Gaze Movement
  99. 99. Respond Immediately User Input and Interaction
  100. 100. Respond Immediately Use Simple Interactions User Input and Interaction
  101. 101. Respond Immediately Use Simple Interactions Don’t Apply 2D patterns User Input and Interaction
  102. 102. User Interface and Information Provide Information in Context
  103. 103. DoDon’t
  104. 104. Provide Information in Context Use Visual Indicators User Interface and Information
  105. 105. https://developer.apple.com/ios/human-interface-guidelines/technologies/augmented-reality/
  106. 106. Provide Information in Context Use Visual Indicators Onboarding is Important User Interface and Information
  107. 107. Walktrough Camera Access Request Scanning Placing Objects Interacting
  108. 108. Walktrough Camera Access Request Scanning Placing Objects Interacting
  109. 109. Walktrough Camera Access Request Scanning Placing Objects Interacting
  110. 110. Walktrough Camera Access Request Scanning Placing Objects Interacting
  111. 111. DoDon’t https://medium.com/inborn-experience/onboarding-in-augmented-reality-mobile-application-6e08e1d338f8
  112. 112. Provide Information in Context Use Visual Indicators Onboarding is Important Error Handling, too User Interface and Information
  113. 113. Creating an Illusion Light and Shadow are Important
  114. 114. https://www.youtube.com/watch?v=ttdPqly4OF8
  115. 115. Light and Shadow are Important Avoid Interruptions Creating an Illusion
  116. 116. Light and Shadow are Important Avoid Interruptions Avoid Clutter Creating an Illusion
  117. 117. Consider the Context Consider a User’s Environment
  118. 118. Consider a User’s Environment There’s No Optimal Setting Consider the Context
  119. 119. Comfort and Safety
  120. 120. Avoid Fatiguing The User Comfort and Safety
  121. 121. Avoid Fatiguing The User Support Short Usage Comfort and Safety
  122. 122. Avoid Fatiguing The User Support Short Usage Avoid Large Movements Comfort and Safety
  123. 123. Avoid Fatiguing The User Support Short Usage Avoid Large Movements Respect Safe Space Comfort and Safety
  124. 124. @fischaelameergeildanke.com @SCRIPT18 The Future of AR in the Web
  125. 125. WebXR as an Extension The Future of AR in the Web
  126. 126. WebXR as an Extension Declarative WebGL The Future of AR in the Web
  127. 127. WebXR as an Extension Declarative WebGL 3D Positions for Elements The Future of AR in the Web
  128. 128. https://www.youtube.com/watch?v=8GXUlijX60E
  129. 129. Get involved! The Future of AR in the Web
  130. 130. @fischaelameergeildanke.com @SCRIPT18 Thank you!
  131. 131. https://github.com/jeromeetienne/AR.js/ https://github.com/google-ar/three.ar.js/ https://research.mozilla.org/mixed-reality/ https://github.com/mozilla/webxr-api https://webxr.io/webar-playground/app/ Libraries https://github.com/mozilla-mobile/webxr-ios https://github.com/google-ar/WebARonARKit https://github.com/google-ar/WebARonARCore Browsers https://3d.io/docs/api/1/aframe-components.html https://3d.io/docs/api/1/furniture.html https://poly.google.com/ https://sketchfab.com/ https://jeromeetienne.github.io/AR.js/three.js/examples/arcode.html Assets

×