JavaScript for VR with notes
•Download as PPTX, PDF•
0 likes•531 views
JS for VR draft - older deck, but notes section contains details from talk
Recommended
More Related Content
What's hot
What's hot (20)
Recently uploaded
Recently uploaded (20)
JavaScript for VR with notes
- 2. @misslivirose Virtual & Augmented Reality Evangelist Microsoft
- 3. What is VR? The Matrix for Today’s Developer
- 4. Head Mounted Displays An Overview
- 6. The VR App Ecosystem
- 7. JavaScript ♥ VR Why build VR for the web
- 8. Under the Hood The WebVR Boilerplate Template by Boris Smus
- 9. VRManager.JS
- 10. VREffect.JS
- 11. Example Projects
- 13. SceneVR Interactive worlds in the browser
- 15. Your Turn! How to begin building your own VR Web Applications
- 16. • Download FF Nightly/Chromium WebVR • Examine the WebVR Boilerplate code • Experiment with Three.js • Put it all together! http://aka.ms/jsvr
- 17. Q & A
Editor's Notes
- Hi everyone! My name is Liv Erickson and I am a virtual and augmented reality evangelist at Microsoft and enthusiast 24/7. I studied computer science with a focus on user experience engineering at Virginia Tech back in the US, and after reading Ready Player One and watching The Matrix a bunch of times, I moved to the San Francisco Bay Area and decided that it was time to take the red pill and see exactly what the virtual reality rabbit hole had to offer. I began developing for virtual reality back in early 2014, when Google announced their Cardboard headset – I think it was a matter of days between trying out the Cardboard for the first time and ordering my Oculus developer kit and deciding to go all in with building for VR. I was previously working on a product group at Microsoft that wasn’t related to virtual reality at all, so I spent most of my free time teaching myself Unity and JavaScript for WebVR, and, of course – playing around in my headsets.
- Introduction to Virtual and Augmented Reality Just about everyone knows what The Matrix is, even if they aren’t a developer and don’t consider themselves technical. For many, though, that’s where their understanding of virtual reality ends. The reality of VR is not quite as dreary – right now, immersive computing is undergoing a resurgence in popularity and, at least if you ask me – it’s probably going to stick around. Last year, Facebook bought Oculus VR, a company working on creating affordable head mounted virtual reality displays (or, HMDs) for consumer use, for over 2 billion dollars. Analysts are now estimating that virtual and augmented reality will be an industry worth 150 billion by 2020. Today, hardware companies are rallying to create their own virtual reality and augmented reality headsets ranging from self-contained, mobile powered HMDs to wired devices running off of high end graphics cards and entire room tracking solutions. Do we have the equivalent of the Matrix yet in the VR industry? No, and we’re not really even close – but between you and me, that’s probably a good thing. This isn’t the first time that virtual reality has been “the next big thing”, but it is the first time that it’s hit consumer pricing and the developer pool has never been stronger for creating content. One third of the world’s population has a computer in their pocket with them – from a saturation standpoint, the market is finally there. What then, exactly, do we have? We have the hardware to create tiny displays that we wear on our heads to view content that covers our field of view. Need a new monitor? No problem – just spin up another window in your VR desktop. We have head mounted displays selling for as little as $20, powered by the smart phones that we already use daily. Several companies have announced their own offerings in the higher-end HMD space, devices that connect to your desktop and deliver immensely immersive and realistic environments right to our desks. What is AR and VR? How will it impact computing as a platform? Isn’t this all a bunch of hype? What is different from the last time that “VR” was the “Next Big Thing” in computing?
- Right now, there are two predominant paths for head mounted display technologies, with many companies working on various differentiating feature sets within the two categories. Both types of HMDs share similar features – casing, lens; but are powered separately – one uses your desktop and treats the HMD as a separate display, and one uses your smartphone as both the display and the processing unity. For this talk, I’ll refer to mobile VR headsets as VR headsets that are powered by a smartphone, which replaces the screen and processing units for rendering 3D applications. Some examples of these headsets include the Oculus-powered Samsung Gear VR, which works with the Galaxy Note 4 and S6 phones, and the low-cost Google Cardboard, which is exactly what it sounds like. These are different from desktop powered and standalone HMDs, such as the Oculus Rift or HTC Vive, which are tethered to a desktop computer and generally consist of more components than the mobile powered HMDs – most notably, the addition of a separate screen within the headset itself. These devices make use of the processing power of the computer that they are connected to, and can utilize GPU acceleration for higher quality experiences. The tradeoff for this, though, is that the user’s natural motion is limited due to being physically connected to the machine, but we’re starting to see advancements in standalone devices that are self contained and are more powerful than what you can get with a smartphone device. Before this wave of virtual reality computing technology, virtual reality devices were impractical for consumers – an immersion rig might cost upwards of $40,000 just ten years ago, meaning that it wasn’t making it’s way into the home at the time. In 2013, Oculus VR created a consumer-priced virtual reality headset prototype, the Oculus Rift, for under $400 – the cost of VR headsets has dropped to 1% of its previous cost. Since Oculus made its first announcements, others have joined in the VR race with their own hardware. There is a range of performance with the devices, with focus mainly centering around field of view, display resolution and pixel density, weight, and comfort. Some of the devices contain additional “perks” – the FOVE headset provides eye-tracking as an input mechanism; the Gear VR has a touch pad integrated into the outside of the device – and innovation is continuing to grow.
- Right now, the virtual and augmented reality application ecosystem is in the position of outpacing the hardware – almost all desktop caliber virtual reality headsets are still in developer preview, but mobile-based applications are filling app stores as lower-end head mounted displays powered by mobile phones are saturating the market. Right now, the bulk of development being done is around first person POV games or experiences, and are being developed on an “app by app” basis. This is due to several factors: right now, the majority of virtual reality devices require higher than average graphics processing power; entertainment is a huge market for early virtual reality; Unity and Unreal game engines were early supporters of VR with easy to use plugins for 3D stereoscopic rendering. The power of virtual reality is hitting mainstream though, even with the offset in hardware availability, and is being adopted into other industries with exceptional results. The New York times is utilizing VR technology to augment breaking news stories, architecture firms are beginning to use mobile VR to visualize in-progress buildings, and medical researchers have begun testing immersive environments for treating emotional disorders – all of these are just the tip of a new paradigm for application development and experience. Let’s look at a few numbers - Google Play has its own category for virtual and augmented reality applications under its Cardboard name, and the ongoing Oculus Mobile Game Jam has 503 submissions of applications and experiences for mobile VR. Both iOS and Windows Phone have their own VR titles, and you can buy a mobile powered headset for under $20. WearVR, a site that aggregates virtual reality applications, is approaching 1000 experiences available to try today.
- JavaScript as a language has helped completely redefine what we expect from computing experiences as more and more of the digital world takes to the cloud and consumers utilize browsers as their portal for the digital age. It’s time to bring that into virtual reality now – JavaScript and the existing libraries already in place, the utility of WebGL, and the vast developer talent pool means that it’s never been a better time to create content for 3D, immersive experiences. Today’s prevalence of developer resources for the web around JavaScript enable you to take your current experience and knowledge of the web to revolutionize how VR applications are built once to run anywhere, regardless of the headset fragmentation that is already being seen. Right now, VR software is being primarily built with game engines such as Unity or Unreal – which is great for creating immersive media-based content, such as first person games, but just as we don’t require heavy development tools to build our web applications today, there is a growing opportunity to not just build out the virtual reality web ecosystem, but to help define it. JavaScript’s role in virtual reality on the web is gaining traction primarily with WebVR, an under-development API that Mozilla and Google have partnered in to bring browser-level VR device detection for both desktop and mobile HMDs. WebVR offloads the bulk of the work with detecting and utilizing display and input devices for immersive computing to display content to a user with a VR-enabled device, and websites utilizing this API to render content in a VR-friendly way can make use of a JavaScript VR Rendering library (there are several) to provide a clean way to switch between non-VR and VR-enabled browsers. Right now, when the WebVR API finds a supported VR device, the browser will detect and enumerate the device, which can be one of two types of VR devices that allow for stereoscopic, positional rendering: an HMDVRDevice, which represents the optical display components of the VR headset, or a PositionSensorVRDevice, which is a device that doesn’t have any optical components. Because of the hardware differences between various VR devices, the API also provides read-only interfaces for permanent device features, such as those that represent field of view, and those that are temporary, such as the device’s orientation at a given moment; these are both passed through the browser in a way that allows renderers to interact with the device without needing to dig into the details of specific devices at runtime. Currently, Firefox Nightly and specific builds of Chromium support WebVR, and boilerplate code for renderers are available for use with common 3D JavaScript libraries with the most common being Three.JS. Websites that are built to support WebVR using one of these rendering libraries are able to display 3D content within their browser window, and the browser itself will handle whether or not there are supported VR devices detected. The rendering libraries will generally check whether or not there is a supported device enumerated by the browser’s detection, and will render the canvas accordingly – stereoscopically if a headset is present, and with a single camera defined in the application if not.
- Setting up a WebVR project is organizationally the same as any other JavaScript application. If you’re already familiar with WebGL and 3D graphic development, it’s almost as simple as dropping in a library that renders the WebGL camera stereoscopically on top of your exisiting WebGL canvases. There are several boilerplate libraries that can be used to create VR-enabled websites, which contain several components that make it easier to interface with the WebVR API. Generally, these libraries make it so that you can use one or two lines of code to render your application in VR mode, and have components that handle the checks for whether a VR device is present, if your website is in “VR” mode, the switching between VR- and standard viewing modes, and cover a lot of the specific graphic rendering distortions based on the exact device that is connected. If you’re writing all of this code from scratch, you will need a manager component that takes care of querying the browser to see if there are devices connected, what type of devices they are, and decide whether or not your site should be rendered in VR. Your application will also need a way to query the browser for the display resolution field of view, and orientation, so that you can adjust how your canvas is displayed within each eye, and offset appropriately based on the distortion of the VR device, which eye you’re accounting for, and update that every time the camera moves. So, I stick to the boilerplate! I have to admit that I am majorly impressed and thankful for the folks writing the rendering libraries, because they help allow developers to focus more on creating content than worrying about what goes on under the hood with the API. Realistic, first person, 3D stereoscopic rendering isn’t the easiest, but these libraries take that out of the equation, so that all we have to do is use a few of their files to create rich, immersive experiences in VR, regardless of the device type we’re viewing it on. On top of the code that handles interfacing with the devices, we get into the fun stuff, where we get to start creating worlds. We heard a really great talk [[there’s an upcoming talk]] about Three.JS, a popular 3D graphics library for JavaScript, which can be used with a VR Renderer and manager to build a 3-dimensional computing experience that you can get up and running with just a few functions. The actual environment that you build for is no different than existing 3D WebGL canvases, with some exceptions around best practices for VR environments, that utilize a camera created from the VR device that orients the viewer within the scene itself based on how the browser is reporting the coordinates and positioning. Instead of seeing just one part of the canvas on a webpage, its as though the viewer is standing directly in the scene itself.
- The VR Manager handles: Whether or not VR devices are available Which types of VR devices are present Entering and exiting full screen mode Creating a button to switch into VR mode Watch for & attempt to avoid sleep mode / screen lock for mobile
- The VREffect handles: Rendering the canvas stereoscopically Applying the scaling/clipping as needed based on the field of view and display There are a number of interesting considerations for viewing something in VR with regards to the camera effects, which is why having rendering libraries available and handling the majority of cases for developers is exceptionally useful. With VR displays, the screen resolution, field of view, distance from the viewer, amount of lens distortion, and camera placement within the scene all play a role in how comfortable and realistic the viewer’s experience is. The VR display effect in this example creates two cameras that are offset within the scene, and uses the WebVR API to query the devices for the resolution and field of view of each eye. This is then called within the scene being displayed in the browser, such as a Three.JS scene, instead of the default WebGL renderer.
- RiftSketch is a great example of an application that utilizes WebVR, Three.js, and WebGL – it provides the user with a real-time programming environment that can be used to create objects in the same scene that the user is inside of. The application utilizes the same structure as the VR Boilerplate template that we just covered, but instead of simply displaying a predefined WebGL scene in VR, it enables the user to modify his or her environment through the text editor using Three JS code. As the user types and creates new objects, they appear – as seen with the tree above. The developer has several videos that show this in action, which I highly recommend checking out on YouTube. Brian, in this particular example, is using a Leap Motion input controller to pass through the image of his keyboard as he codes so that he can look down and see what he’s typing – covering VR input devices would be an entirely separate presentation, but this is just one way that developers are bringing various parts of the physical world into the virtual one, and in this case, it’s extremely helpful for developers to see their input as they are building their scene.
- In addition to simply creating WebVR experiences with JavaScript, JS can also be used as a powerful way to develop Virtual Reality tools. Vizor VR is an excellent example of a way that JavaScript is helping define the way VR content is created. Written almost entirely in JavaScript, Vizor VR is a visual programming environment that can be self-hosted or accessed through the browser at create.vizor.io. With Vizor, various components of your VR scene are represented with a drill-down type hierarchy of graphs that are connected to create immersive environments and behaviors. The editor contains a list of objects and commands that can be used to build a VR-enabled scene, which can then be placed into a WebGL canvas using a generated JSON file from the graph created.