Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Building AR and VR Experiences for Web Apps with JavaScript

54 views

Published on

It is increasingly important to understand how AR and VR technologies are changing what is possible to do in modern web applications. There are a lot of tools and technologies that you can choose from to tie all of the pieces together to start implementing AR and VR features, but not all of them are JavaScript or web friendly. In this talk Hasan will get into AR and VR development from the perspective of a web app developer who is competent with modern JavaScript and web development tools. He will also introduce a way to share what you build on the growing Oculus platform, and why Oculus is a great entry point for VR.

Objective
Learn how to approach your first feature or project involving AR or VR in your applications that run in browsers and mobile devices, all using JavaScript

Target Audience
Web application developers interested in building AR and VR driven features in their web applications

Assumed Audience Knowledge
JavaScript, web technology

Five Things Audience Members Will Learn
WebGL basics
Intro to React360
Working with Three.js
What types of use cases to apply AR and VR technology to
Building for Oculus

Published in: Internet
  • Be the first to comment

  • Be the first to like this

Building AR and VR Experiences for Web Apps with JavaScript

  1. 1. AR/VR in JavaScript Apps FITC Web Unleashed 2018
  2. 2. The Rise of Extended Reality (AR/VR) • AR (Augmented Reality) – Interaction with physical realty augmented with computer generated input and output • VR (Virtual Reality) – Replace physical reality with a computer generated one • Hardware costs plummeting (Oculus Go launched F8 2018, $199) • Software for building AR and VR experiences getting better every day
  3. 3. Introduction – Hasan Ahmad • Principal Consultant at DEV6 • Developer Circles from Facebook (Toronto) • https://dev6.com • https://www.facebook.com/groups /DeveloperCirclesToronto/ • https://twitter.com/has_ibr_ahm
  4. 4. Industries that are embracing XR • Gaming • Media • Mobile Apps • Streaming • Education • Industrial • Retail
  5. 5. AR/VR tech-stack for web devs • Little bit of math & physics background • Smartphone or VR headset • Three.js • WebVR API / Web XR API • A-Frame • React 360 • AR.js • 3D Content
  6. 6. Three.js • https://threejs.org/ • WebGL graphics library • Scene – What to display • Camera – What to view • Renderer – How to display • Geometry – Objects, textures, etc
  7. 7. $ npm install three ... import { Scene } from 'three’; const scene = new Scene(); OR <script src="https://fastcdn.org/three.js/73/three.min.js"> </script>
  8. 8. var scene = new THREE.Scene(); var camera = new THREE.PerspectiveCamera(75, window.innerWidth / window.innerHeight, 0.1, 1000); var renderer = new THREE.WebGLRenderer(); renderer.setSize(window.innerWidth, window.innerHeight); document.body.appendChild(renderer.domElement); var geometry = new THREE.BoxGeometry(1, 1, 1); var material = new THREE.MeshBasicMaterial({color: 0x00ff00}); var cube = new THREE.Mesh(geometry, material); scene.add(cube); camera.position.z = 5;
  9. 9. WebVR • Low-level API to gather info about the VR display + capabilities • Eye parameters, the data to render the scene for each eye • Field of view data • Position of the VR display (and velocity and acceleration)
  10. 10. WebXR (unstable!) • Evolution of WebVR spec • Much faster than WebVR API • Better architecture to support both VR and AR, multiple view types • Desktop • VR Headset • Smartphone • Magic Window • Touchscreen, Mouse, Gamepad, Controllers
  11. 11. The future of the web is immersive (Google I/O ‘18) https://www.youtube.com/watch?v=1t1gBVykneA
  12. 12. WebXR-Polyfill • Best way to make sure code written to XR spec (proposal) will actually work https://github.com/immersive-web/webxr-polyfill <script src='https://cdn.jsdelivr.net/npm/webxr- polyfill@latest/build/webxr-polyfill.js’></script> $ npm install --save webxr-polyfill
  13. 13. function initXR() { xrButton = new XRDeviceButton({ onRequestSession: onRequestSession, onEndSession: onEndSession }); document.querySelector('header').appendChild(xrButton.domElement); if (navigator.xr) { navigator.xr.requestDevice().then((device) => { device.supportsSession({immersive: true}).then(() => { xrButton.setDevice(device); }); }); } } function onRequestSession(device) { device.requestSession({immersive: true}).then(onSessionStarted); }
  14. 14. function onSessionStarted(session) { xrButton.setSession(session); session.addEventListener('end', onSessionEnded); gl = createWebGLContext({compatibleXRDevice: session.device}); renderer = new Renderer(gl); scene.setRenderer(renderer); session.baseLayer = new XRWebGLLayer(session, gl); session.requestFrameOfReference('eye-level').then((frameOfRef) => { xrFrameOfRef = frameOfRef; session.requestAnimationFrame(onXRFrame); }); }
  15. 15. function onXRFrame(t, frame) { let session = frame.session; scene.startFrame(); session.requestAnimationFrame(onXRFrame); let pose = frame.getDevicePose(xrFrameOfRef); if (pose) { gl.bindFramebuffer(gl.FRAMEBUFFER, session.baseLayer.framebuffer); gl.clear(gl.COLOR_BUFFER_BIT | gl.DEPTH_BUFFER_BIT); for (let view of frame.views) { let viewport = session.baseLayer.getViewport(view); gl.viewport(viewport.x, viewport.y, viewport.width, viewport.height); scene.draw(view.projectionMatrix, pose.getViewMatrix(view)); } } else {} scene.endFrame(); }
  16. 16. A-Frame • Web framework originally from Mozilla for rendering AR and VR in web pages • Declarative syntax • 3D scene graph with markup language
  17. 17. <head> <script src="https://aframe.io/releases/0.8.0/aframe.min.js"></script> </head> $ npm install aframe ... require(‘aframe’);
  18. 18. <body> <a-scene> <a-box position="-1 0.5 -3" rotation="0 45 0" color="#4CC3D9"></a-box> <a-sphere position="0 1.25 -5" radius="1.25" color="#EF2D5E"></a-sphere> <a-cylinder position="1 0.75 -3" radius="0.5" height="1.5" color="#FFC65D"></a-cylinder> <a-plane position="0 0 -4" rotation="-90 0 0" width="4" height="4" color="#7BC8A4"></a-plane> <a-sky color="#ECECEC"></a-sky> </a-scene> </body>
  19. 19. Loading 3D Models <a-assets> <a-asset-item id="cityModel" src="https://cdn.aframe.io/test-models/models/virtualcity/VC.gltf"> </a-asset-item> </a-assets>
  20. 20. https://aframe.io/ https://aframe.io/aframe-school/#/
  21. 21. $ npm install -g react-360-cli $ react-360 init Hello360 $ cd Hello360 $ npm start React 360
  22. 22. React 360 • You can use React to build VR web UIs • Render React Native components in 3D
  23. 23. import { AppRegistry, StyleSheet, Text, View } from'react-360; export default class Hello360 extends React.Component { render() { return ( <View style={styles.panel}> <View style={styles.greetingBox}> <Text style={styles.greeting}> Welcome to React 360 </Text> </View> </View> ); } };
  24. 24. React 360 • Similar in Architecture to React Native • Uses Web Workers to avoid single- threaded computation limitation, which could impact performance, break immersion
  25. 25. React 360 • Can also load 3D models, using Entity (multiple formats) // to reference a GLTF2 model <Entity source={{gltf2: asset('myModel.gltf')}} /> // to reference an untextured OBJ model <Entity source={{obj: asset('myModel.obj')}} /> // to reference an OBJ with matching MTL file <Entity source={{obj: asset('myModel.obj'), mtl: asset('myModel.mtl')}} />
  26. 26. Augmented Reality • Similar problems solved in VR • Must be able to identify real- world geometry • Capable of marker-based AR at 60fps, even on budget smartphones https://aframe.io/blog/arjs/
  27. 27. Building AR with A-Frame (AR.js) <script src="https://jeromeetienne.github.io/AR.js/aframe/build/aframe-ar.js"> </script> <a-scene embedded arjs> <a-marker-camera preset='hiro'></a-marker-camera>
  28. 28. Hiro Marker
  29. 29. Building AR with A-Frame <body style='margin : 0px; overflow: hidden;’> <a-scene embedded arjs> <!-- create your content here. just a box for now --> <a-box position='0 0.5 0' material='opacity: 0.5;'></a-box> <!-- define a camera which will move according to the marker position --> <a-marker-camera preset='hiro'></a-marker-camera> </a-scene> </body>
  30. 30. Wikitude SDK • Paid SDK, for implements sophisticated AR algorithms, available as a plugin for Native or Cordova projects • Free trial available (for experiments and education) • Instant Tracking, SLAM, SMART • Built on top of ARCore and ARKit • https://www.wikitude.com/augmented-reality-instant-tracking/
  31. 31. 3D Assets • https://sketchfab.com/ • https://poly.google.com/ • Create or buy 3D content to build amazing AR and VR experiences.
  32. 32. Continued Study • https://www.khanacademy.org/math/linear-algebra • https://medium.com/@necsoft/three-js-101-hello-world-part-1- 443207b1ebe1 • https://developer.mozilla.org/en- US/docs/Web/API/WebGL_API#Guides • https://github.com/mozilla/aframe-xr • https://aframe.io/blog/arjs/
  33. 33. Summary • Extended Reality is of increasing interest to many industries • Web tech can get us quite far, even with today’s experimental APIs • There are a number of entry points into this tech stack, pick the right level of abstraction for you
  34. 34. Thank You!

×