Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

WebXR: A New Dimension For The Web Writing Virtual and Augmented Reality Apps With Web Technology

136 views

Published on

Writing 3D applications for the browser is possible since some years due to the WebGL API. But since Virtual Reality and Augmented Reality are getting more popular, web developers and designers now have a serious reason to get to know the 3D web, its APIs and design patterns.

Published in: Technology
  • Be the first to comment

WebXR: A New Dimension For The Web Writing Virtual and Augmented Reality Apps With Web Technology

  1. 1. @fischaelameergeildanke.com #JSCamp Writing WebXR Apps 
 with Web Technology
  2. 2. React 360
  3. 3. WebXR Device API React 360
  4. 4. UX Design for WebXR WebXR Device API React 360
  5. 5. WebXR
  6. 6. WebXR WebVR
  7. 7. WebXR WebVR WebAR
  8. 8. WebXR WebVR WebAR
  9. 9. Created by Laura Hernández from the Noun Project
  10. 10. Created by Laura Hernández from the Noun Project
  11. 11. Completely DigitalVirtual Reality
  12. 12. Virtual Overlay Augmented Reality:
  13. 13. Virtual and Real World Coincide Mixed Reality
  14. 14. WebXR WebVR WebAR
  15. 15. WebXR Device API Detect AR/VR Devices !11
  16. 16. WebXR Device API Detect AR/VR Devices !11 Get Device’s Capabilities
  17. 17. WebXR Device API Detect AR/VR Devices !11 Get Device’s Capabilities Get Device’s Orientation/Position
  18. 18. WebXR Device API Detect AR/VR Devices !11 Get Device’s Capabilities Get Device’s Orientation/Position Display Images With A Fitting Frame Rate
  19. 19. WebVR API
  20. 20. WebVR API WebXR Device API
  21. 21. WebXR Device API Supports Augmented Reality !13
  22. 22. WebXR Device API Supports Augmented Reality !13 Clean, Consistent, Predictable
  23. 23. WebXR Device API Supports Augmented Reality !13 Clean, Consistent, Predictable Better Browser Optimizations
  24. 24. WebXR Device API Supports Augmented Reality !13 Clean, Consistent, Predictable Better Browser Optimizations Unified Input Model
  25. 25. Lifetime of a WebXR App Request XR Device
  26. 26. Lifetime of a WebXR App Request XR Device Reveal XR Functionality
  27. 27. Lifetime of a WebXR App Request XR Device Reveal XR Functionality Request XR Session
  28. 28. Lifetime of a WebXR App Request XR Device Reveal XR Functionality Request XR Session Run Render Loop
  29. 29. Request a XR Devices navigator.xr.requestDevice().then(device => { if (device) { handleXRAvailable(device); } }).catch(error => console.error('Unable to request an XR device: ', error); });
  30. 30. Request a XR Devices navigator.xr.requestDevice().then(device => { if (device) { handleXRAvailable(device); } }).catch(error => console.error('Unable to request an XR device: ', error); });
  31. 31. Check XR Session Support let xrDevice = null; function handleXRAvailable(device) { xrDevice = device; xrDevice.supportsSession({exclusive: true}).then(() => { addWebXRButtonToPage(); }).catch((error) => { console.log('Session not supported: ' + error); }); }
  32. 32. Check XR Session Support let xrDevice = null; function handleXRAvailable(device) { xrDevice = device; xrDevice.supportsSession({exclusive: true}).then(() => { addWebXRButtonToPage(); }).catch((error) => { console.log('Session not supported: ' + error); }); }
  33. 33. Request a XR Session function beginXRSession() { let canvas = document.createElement('canvas'); let context = canvas.getContext('xrpresent'); document.body.appendChild(canvas); xrDevice.requestSession({exclusive: true, outputContext: context}) .then(onSessionStarted) .catch((error) => {console.log('requestSession failed: ' + error);}); }
  34. 34. Start a XR Session let xrSession = null; let xrFrameOfReference = null; function onSessionStarted(session) { xrSession = session; xrSession.requestFrameOfReference('eyeLevel') .then((frameOfReference) => {xrFrameOfReference = frameOfReference;}) .then(setupWebGLLayer) .then(() => {xrSession.requestAnimationFrame(onRenderFrame);}); }
  35. 35. Start a XR Session let xrSession = null; let xrFrameOfReference = null; function onSessionStarted(session) { xrSession = session; xrSession.requestFrameOfReference('eyeLevel') .then((frameOfReference) => {xrFrameOfReference = frameOfReference;}) .then(setupWebGLLayer) .then(() => {xrSession.requestAnimationFrame(onRenderFrame);}); }
  36. 36. Start a XR Session let xrSession = null; let xrFrameOfReference = null; function onSessionStarted(session) { xrSession = session; xrSession.requestFrameOfReference('eyeLevel') .then((frameOfReference) => {xrFrameOfReference = frameOfReference;}) .then(setupWebGLLayer) .then(() => {xrSession.requestAnimationFrame(onRenderFrame);}); }
  37. 37. Setup an XRLayer let glCanvas = document.createElement('canvas'); let glContext = glCanvas.getContext('webgl'); function setupWebGLLayer() { return glContext.setCompatibleXRDevice(xrDevice).then(() => { xrSession.baseLayer = new XRWebGLLayer(xrSession, glContext); }); }
  38. 38. Setup an XRLayer let glCanvas = document.createElement('canvas'); let glContext = glCanvas.getContext('webgl'); function setupWebGLLayer() { return glContext.setCompatibleXRDevice(xrDevice).then(() => { xrSession.baseLayer = new XRWebGLLayer(xrSession, glContext); }); }
  39. 39. Start a XR Session let xrSession = null; let xrFrameOfReference = null; function onSessionStarted(session) { xrSession = session; xrSession.requestFrameOfReference('eyeLevel') .then((frameOfReference) => {xrFrameOfReference = frameOfReference;}) .then(setupWebGLLayer) .then(() => {xrSession.requestAnimationFrame(onRenderFrame);}); }
  40. 40. Start the Render Loop function onRenderFrame(timestamp, xrFrame) { let pose = xrFrame.getDevicePose(xrFrameOfReference); if (pose) { for (let view of xrFrame.views) { // Draw something } } // Input device code xrSession.requestAnimationFrame(onRenderFrame); }
  41. 41. Start the Render Loop function onRenderFrame(timestamp, xrFrame) { let pose = xrFrame.getDevicePose(xrFrameOfReference); if (pose) { for (let view of xrFrame.views) { // Draw something } } // Input device code xrSession.requestAnimationFrame(onRenderFrame); }
  42. 42. Start the Render Loop function onRenderFrame(timestamp, xrFrame) { let pose = xrFrame.getDevicePose(xrFrameOfReference); if (pose) { for (let view of xrFrame.views) { // Draw something } } // Input device code xrSession.requestAnimationFrame(onRenderFrame); }
  43. 43. Start the Render Loop function onRenderFrame(timestamp, xrFrame) { let pose = xrFrame.getDevicePose(xrFrameOfReference); if (pose) { for (let view of xrFrame.views) { // Draw something } } // Input device code xrSession.requestAnimationFrame(onRenderFrame); }
  44. 44. Exit the WebXR Session function endXRSession() { if (xrSession) {xrSession.end().then(onSessionEnd);} } function onSessionEnd() { xrSession = null; window.requestAnimationFrame(onDrawFrame); }
  45. 45. Exit the WebXR Session function endXRSession() { if (xrSession) {xrSession.end().then(onSessionEnd);} } function onSessionEnd() { xrSession = null; window.requestAnimationFrame(onDrawFrame); }
  46. 46. Fallback: Magic Window let mwCanvas = document.createElement('canvas'); let mwContext = mwCanvas.getContext('xrpresent'); document.body.appendChild(mwCanvas); function beginMagicWindowXRSession() { xrDevice.requestSession({exclusive: false, outputContext: mwContext}) .then(OnSessionStarted) .catch((error) => {console.log('requestSession failed: ' + error);}); }
  47. 47. Fallback: Magic Window let mwCanvas = document.createElement('canvas'); let mwContext = mwCanvas.getContext('xrpresent'); document.body.appendChild(mwCanvas); function beginMagicWindowXRSession() { xrDevice.requestSession({exclusive: false, outputContext: mwContext}) .then(OnSessionStarted) .catch((error) => {console.log('requestSession failed: ' + error);}); }
  48. 48. On Page Load: Magic Window
  49. 49. On Page Load: Magic Window exclusive sessions are supported
  50. 50. On Page Load: Magic Window exclusive sessions are supported Render a "Start VR" Button
  51. 51. 6DOF VR Headset Created by Hans Gerhard Meier, Bence Bezeredy, Laura Hernández, Anil, Sachin Modgekar, Ben Davis from the Noun Project
  52. 52. 6DOF VR Headset AR-ready Smartphone Created by Hans Gerhard Meier, Bence Bezeredy, Laura Hernández, Anil, Sachin Modgekar, Ben Davis from the Noun Project
  53. 53. 6DOF VR Headset AR-ready Smartphone 3DOF VR Headset Created by Hans Gerhard Meier, Bence Bezeredy, Laura Hernández, Anil, Sachin Modgekar, Ben Davis from the Noun Project
  54. 54. Progressive Enhancement 6DOF VR Headset AR-ready Smartphone 3DOF VR Headset Created by Hans Gerhard Meier, Bence Bezeredy, Laura Hernández, Anil, Sachin Modgekar, Ben Davis from the Noun Project
  55. 55. Progressive Enhancement 6DOF VR Headset AR-ready Smartphone 3DOF VR Headset WebXR Polyfill Created by Hans Gerhard Meier, Bence Bezeredy, Laura Hernández, Anil, Sachin Modgekar, Ben Davis from the Noun Project
  56. 56. Progressive Enhancement 6DOF VR Headset AR-ready Smartphone 3DOF VR Headset Magic Window WebXR Polyfill Created by Hans Gerhard Meier, Bence Bezeredy, Laura Hernández, Anil, Sachin Modgekar, Ben Davis from the Noun Project
  57. 57. Progressive Enhancement 6DOF VR Headset AR-ready Smartphone 3DOF VR Headset Magic Window Gyroscope WebXR Polyfill Created by Hans Gerhard Meier, Bence Bezeredy, Laura Hernández, Anil, Sachin Modgekar, Ben Davis from the Noun Project
  58. 58. Progressive Enhancement 6DOF VR Headset AR-ready Smartphone 3DOF VR Headset Magic Window Gyroscope WebXR Polyfill Static Image Created by Hans Gerhard Meier, Bence Bezeredy, Laura Hernández, Anil, Sachin Modgekar, Ben Davis from the Noun Project
  59. 59. The Future: Augmented Reality AR Session
  60. 60. The Future: Augmented Reality AR Session Hit Test
  61. 61. The Future: Augmented Reality AR Session Hit Test AR Anchors
  62. 62. React 360 https://facebook.github.io/react-360/
  63. 63. React 360 https://aframe.io/ A-Frame
  64. 64. React 360 Full-Sphere Gallery
  65. 65. $ npm install -g react-360-cli React 360 Full-Sphere Gallery
  66. 66. $ npm install -g react-360-cli $ react-360 init HelloJSCampBarcelona React 360 Full-Sphere Gallery
  67. 67. $ npm install -g react-360-cli $ react-360 init HelloJSCampBarcelona $ cd HelloJSCampBarcelona React 360 Full-Sphere Gallery
  68. 68. $ npm install -g react-360-cli $ react-360 init HelloJSCampBarcelona $ cd HelloJSCampBarcelona $ npm start React 360 Full-Sphere Gallery
  69. 69. $ npm install -g react-360-cli $ react-360 init HelloJSCampBarcelona $ cd HelloJSCampBarcelona $ npm start React 360 Full-Sphere Gallery http://localhost:8081/index.html
  70. 70. React 360 Application
  71. 71. React 360 Application React 360 Runtime
  72. 72. React 360 Application React 360 Runtime Renders 3D Objects Keeps A High Frame Rate Web Worker Environment
  73. 73. . ├── __tests__ ├── client.js ├── index.html ├── index.js ├── node_modules ├── package.json ├── rn-cli.config.js └── static_assets React 360 Full-Sphere Gallery
  74. 74. . ├── __tests__ ├── client.js ├── index.html ├── index.js ├── node_modules ├── package.json ├── rn-cli.config.js └── static_assets React 360 Full-Sphere Gallery
  75. 75. . ├── __tests__ ├── client.js ├── index.html ├── index.js ├── node_modules ├── package.json ├── rn-cli.config.js └── static_assets React 360 Full-Sphere Gallery
  76. 76. . ├── __tests__ ├── client.js ├── index.html ├── index.js ├── node_modules ├── package.json ├── rn-cli.config.js └── static_assets React 360 Full-Sphere Gallery
  77. 77. <body> <div id="container"></div> <script src="./client.bundle?platform=vr"></script> <script> React360.init( 'index.bundle?platform=vr&dev=true', document.getElementById('container') ); </script> </body> index.html
  78. 78. <body> <div id="container"></div> <script src="./client.bundle?platform=vr"></script> <script> React360.init( 'index.bundle?platform=vr&dev=true', document.getElementById('container') ); </script> </body> index.html
  79. 79. <body> <div id="container"></div> <script src="./client.bundle?platform=vr"></script> <script> React360.init( 'index.bundle?platform=vr&dev=true', document.getElementById('container') ); </script> </body> index.html
  80. 80. client.js import { ReactInstance } from 'react-360-web' function init(bundle, parent, options = {}) { const r360 = new ReactInstance(bundle, parent, 
 { fullScreen: true, …options }) r360.renderToSurface( r360.createRoot('HelloJSCampBarcelona', {}), r360.getDefaultSurface() ) r360.compositor.setBackground(r360.getAssetURL(‘360_world.jpg')) } window.React360 = { init }
  81. 81. client.js import { ReactInstance } from 'react-360-web' function init(bundle, parent, options = {}) { const r360 = new ReactInstance(bundle, parent, 
 { fullScreen: true, …options }) r360.renderToSurface( r360.createRoot('HelloJSCampBarcelona', {}), r360.getDefaultSurface() ) r360.compositor.setBackground(r360.getAssetURL(‘360_world.jpg')) } window.React360 = { init }
  82. 82. client.js import { ReactInstance } from 'react-360-web' function init(bundle, parent, options = {}) { const r360 = new ReactInstance(bundle, parent, 
 { fullScreen: true, …options }) r360.renderToSurface( r360.createRoot('HelloJSCampBarcelona', {}), r360.getDefaultSurface() ) r360.compositor.setBackground(r360.getAssetURL(‘360_world.jpg')) } window.React360 = { init }
  83. 83. client.js import { ReactInstance } from 'react-360-web' function init(bundle, parent, options = {}) { const r360 = new ReactInstance(bundle, parent, 
 { fullScreen: true, …options }) r360.renderToSurface( r360.createRoot('HelloJSCampBarcelona', {}), r360.getDefaultSurface() ) r360.compositor.setBackground(r360.getAssetURL(‘360_world.jpg')) } window.React360 = { init }
  84. 84. client.js import { ReactInstance } from 'react-360-web' function init(bundle, parent, options = {}) { const r360 = new ReactInstance(bundle, parent, 
 { fullScreen: true, …options }) r360.renderToSurface( r360.createRoot('HelloJSCampBarcelona', {}), r360.getDefaultSurface() ) r360.compositor.setBackground(r360.getAssetURL(‘360_world.jpg')) } window.React360 = { init }
  85. 85. index.js export default class HelloJSCampBarcelona extends React.Component { render() { return ( <View style={styles.panel}> <View style={styles.greetingBox}> <Text style={styles.greeting}>Welcome to React 360</Text> </View> </View> ) } } AppRegistry.registerComponent('HelloJSCampBarcelona', () => HelloChainReactPortland)
  86. 86. client.js … r360.renderToSurface( r360.createRoot('HelloJSCampBarcelona', { photos: [ { uri: './static_assets/1.jpg', title: 'Boat', format: '2D' }, { uri: './static_assets/2.jpg', title: 'Beach', format: '2D' }, { uri: './static_assets/3.jpg', title: 'OnsieJS', format: '2D' }, ], }), r360.getDefaultSurface(), ) …
  87. 87. index.js <View style={styles.wrapper}> <Background uri={current.uri} format={current.format} /> <View style={styles.controls}> <VrButton onClick={this._prevPhoto} style={styles.button}> <Text style={styles.buttonText}>{'<'}</Text> </VrButton> <View><Text style={styles.title}>{current.title}</Text></View> <VrButton onClick={this._nextPhoto} style={styles.button}> <Text style={styles.buttonText}>{'>'}</Text> </VrButton> </View> </View>
  88. 88. index.js <View style={styles.wrapper}> <Background uri={current.uri} format={current.format} /> <View style={styles.controls}> <VrButton onClick={this._prevPhoto} style={styles.button}> <Text style={styles.buttonText}>{'<'}</Text> </VrButton> <View><Text style={styles.title}>{current.title}</Text></View> <VrButton onClick={this._nextPhoto} style={styles.button}> <Text style={styles.buttonText}>{'>'}</Text> </VrButton> </View> </View>
  89. 89. index.js class Background extends React.Component { constructor(props) { super() Environment.setBackgroundImage(props.uri, {format: props.format}) } componentWillReceiveProps(nxtPrps) { if (nxtPrps.uri !== this.props.uri || nxtPrps.format !== this.props.format) { Environment.setBackgroundImage(nxtPrps.uri, {format: nxtPrps.format}) } } render() { return null } }
  90. 90. index.js state = { index: 0 } _prevPhoto = () => { let next = this.state.index - 1; if (next < 0) { next += this.props.photos.length } this.setState({ index: next }) } _nextPhoto = () => { this.setState({ index: this.state.index + 1 }) }
  91. 91. index.js render() { const current = this.props.photos[ this.state.index % this.props.photos.length ] return ( <View style={styles.wrapper}> <Background uri={current.uri} format={current.format} /> … ) }
  92. 92. UX Design for WebXR Apps
  93. 93. Do Not Apply 2D Patterns
  94. 94. No Established Patterns Yet
  95. 95. UX Design Best-Practises Add Feedback, Respond Immediately
  96. 96. Add Feedback
  97. 97. Add Feedback
  98. 98. Add Feedback
  99. 99. Add Feedback
  100. 100. It was the pioneer days; people had to make their own interrogation rooms. Out of cornmeal. These endless days are finally ending in a blaze. When I say, 'I love you,' it's not because I want you or because I can't have you. It's my estimation that every man ever got a statue made of him was one kind of sommbitch or another. Oh my god you will never believe what happened at school today. From beneath you, it devours. I am never gonna see a merman, ever. It was supposed to confuse him, but it just made him peppy. It was like the Heimlich, with stripes! How did your brain even learn human speech? I'm just so curious. Apocalypse, we've all been there; the same old trips, why should we care? Frankly, it's ludicrous to have these interlocking bodies and not...interlock. I just don't see why everyone's always picking on Marie-Antoinette. You're the one freaky thing in my freaky world that still makes sense to me. You are talking crazy-person talk.
  101. 101. It was the pioneer days; people had to make their own interrogation rooms. Out of cornmeal. These endless days are finally ending in a blaze. When I say, 'I love you,' it's not because I want you or because I can't have you. It's my estimation that every man ever got a statue made of him was one kind of sommbitch or another. Oh my god you will never believe what happened at school today. From beneath you, it devours. I am never gonna see a merman, ever. It was supposed to confuse him, but it just made him peppy. It was like the Heimlich, with stripes! How did your brain even learn human speech? I'm just so curious. Apocalypse, we've all been there; the same old trips, why should we care? Frankly, it's ludicrous to have these interlocking bodies and not...interlock. I just don't see why everyone's always picking on Marie-Antoinette. You're the one freaky thing in my freaky world that still makes sense to me. You are talking crazy-person talk.
  102. 102. It was the pioneer days; people had to make their own interrogation rooms. Out of cornmeal. These endless days are finally ending in a blaze. When I say, 'I love you,' it's not because I want you or because I can't have you. It's my estimation that every man ever got a statue made of him was one kind of sommbitch or another. Oh my god you will never believe what happened at school today. From beneath you, it devours. I am never gonna see a merman, ever. It was supposed to confuse him, but it just made him peppy. It was like the Heimlich, with stripes! How did your brain even learn human speech? I'm just so curious. Apocalypse, we've all been there; the same old trips, why should we care? Frankly, it's ludicrous to have these interlocking bodies and not...interlock. I just don't see why everyone's always picking on Marie-Antoinette. You're the one freaky thing in my freaky world that still makes sense to me. You are talking crazy-person talk.
  103. 103. UX Design Best-Practises Add Feedback, Respond Immediately Guide Users with Gaze Cues
  104. 104. Add Gaze Cues
  105. 105. Add Gaze Cues
  106. 106. Add Gaze Cues
  107. 107. Add Gaze Cues
  108. 108. UX Design Best-Practises Add Feedback, Respond Immediately Guide Users with Gaze Cues Provide Information in Context
  109. 109. DoDon’t
  110. 110. Interpretability Usefulness Delight Beau Cronin https://medium.com/@beaucronin/the-hierarchy-of-needs-in-virtual-reality-development-4333a4833acc Comfort
  111. 111. Presence Interpretability Usefulness Delight Beau Cronin https://medium.com/@beaucronin/the-hierarchy-of-needs-in-virtual-reality-development-4333a4833acc Comfort
  112. 112. Presence Interpretability Usefulness Delight Beau Cronin Comfort & Safety https://medium.com/@beaucronin/the-hierarchy-of-needs-in-virtual-reality-development-4333a4833acc
  113. 113. Comfort and Safety in AR There is No Optimal AR Environment
  114. 114. Comfort and Safety in AR There is No Optimal AR Environment Consider a User’s Movement
  115. 115. https://giphy.com/gifs/hyperrpg-vr-ouch-3oKIPAKenYskqXzYnC
  116. 116. https://giphy.com/gifs/hyperrpg-vr-ouch-3oKIPAKenYskqXzYnC
  117. 117. Comfort and Safety in AR There is No Optimal AR Environment Consider a User’s Movement Avoid Fatiguing your Users
  118. 118. Comfort and Safety in VR Do Not Trigger Phobias
  119. 119. Comfort and Safety in VR Do Not Trigger Phobias Do Not Move Things Fast Towards the Camera
  120. 120. Comfort and Safety in VR Do Not Trigger Phobias Do Not Move Things Fast Towards the Camera Respect a User’s Safe Space
  121. 121. https://www.reddit.com/r/VRFail/comments/4s7nc1/friend_loses_his_vrginity_and_then_some_crappy/
  122. 122. https://www.reddit.com/r/VRFail/comments/4s7nc1/friend_loses_his_vrginity_and_then_some_crappy/
  123. 123. https://www.techradar.com/
  124. 124. https://www.reddit.com/r/VRFail/comments/4p9zgj/pool_shot/
  125. 125. https://www.reddit.com/r/VRFail/comments/4p9zgj/pool_shot/
  126. 126. Prevent Simulation Sickness Do Not Move the Horizon or the Camera
  127. 127. Prevent Simulation Sickness Do Not Move the Horizon or the Camera Do Not Use Acceleration
  128. 128. https://web.colby.edu/cogblog/2016/05/09/2451/
  129. 129. https://web.colby.edu/cogblog/2016/05/09/2451/
  130. 130. Prevent Simulation Sickness Do Not Move the Horizon or the Camera Do Not Use Acceleration Avoid Flicker and Blur
  131. 131. Prevent Simulation Sickness Do Not Move the Horizon or the Camera Do Not Use Acceleration Avoid Flicker and Blur Add a Stable Focus Point
  132. 132. @fischaelameergeildanke.com #JSCamp Respect Your Users! Test Your Product on a Diverse Audience.
  133. 133. @fischaelameergeildanke.com #JSCamp Get Involved! https://github.com/immersive-web/webxr https://github.com/facebook/react-360 https://github.com/mozilla/aframe-xr

×