Recommended
Recommended
More Related Content
Similar to Building WebXR Apps in Practice: A Guide
Similar to Building WebXR Apps in Practice: A Guide (20)
More from AugmentedWorldExpo
More from AugmentedWorldExpo (20)
Recently uploaded
Recently uploaded (20)
Building WebXR Apps in Practice: A Guide
- 1. Building WebXR Applications in Practice Casper Fabricius, Co-Founder & CTO, Cimmerse
- 15. Introducing Eric Prince as Cimmerse Co-Founder & CEO Casper, let’s build a connected persistent volumetric spatial 3D interface.
- 16. React Redux A-Frame Three.js WebVR Ruby Rails GraphQL
- 23. www.muuce.com
- 26. • Solves problems of distance, angle and obstruction creating a truly useable tool for the real world. • Browser scan triggers a recognition automatically. • Only 20% of image needs to occupy screen for recognition to occur Image and video recognition tool
- 31. Windows Android Origin trial Daydream “Cardboard” behind origin trial Gear VR Oculus Go not yet supported Virtual reality Windows Mixed Reality only
- 32. iOS Android Soon, behind origin trial Augmented reality WebARonARKit WebARonARCore Windows Meta 2 Hololens
Editor's Notes
- Let’s talk about the future of user interfaces. People sometimes talk about “the final interface”. The interface that’s so good that it won’t get any better.
- Is this the final interface?
- Is this?
- How about this?
- This is a great interface
- And this is THE interface - at least for you and me
- But I’m getting ahead of myself. My name is Casper Fabricius, and I’m the CTO and co-founder of Cimmerse. I’ve done full stack web development for 18 years, VR & AR development for 3 years and I’m from Copenhagen, Denmark. We are known for being the world’s happiest country with the world’s highest tax rates. I can’t really explain how that works.
- I realised a few years ago that the Internet is going from 2D ...
- … to 3D, to being all around us.
- It’s with that knowledge and excitement that I co-founded Cimmerse 2 years ago, and I’ll tell you a bit more about that later, but first: How did I get there.
- This may have been last time I was seen out in the sunshine, away from the computer.
- Fast forward to 2015, where the Oculus Rift DK2 came out, and I literally became obsessed with virtual reality. I quit my job and started freelancing.
- Coming from 15 years of web development, a new library called A-Frame quickly caught my attention. A-Frame is a javascript library that builds on top of Three.js to make super easy to build 3D and virtual reality experiences.
- As an early paid gig, I built a retrospective experience for the social VR app Altspace VR. While Altspace itself is a Unity app, they allow developers to add WebVR experiences built with Three.js to it. And since A-Frame uses Three.js, I was able to pull together an interactive multi-user experience using very familiar javascript libraries.
- I continued to build A-Frame solutions for customers for while, but then I met a guy who shared my vision of an Internet in three dimensions and we started our company, Cimmerse, together.
- We initially focused on bringing art into virtual reality in super high quality, using photogrammetry and, of course, WebVR. The frontend stack with A-Frame and React at the core was still my first choice, and I added my favorite backend stack: Ruby on Rails.
- Using React and Redux to generate interactive 3D experiences is not without its challenges though. We did a project for a customer that required a lot of transitions and animations, and those of you familiar with React and Redux will know that it basically rips elements in and out of the DOM when the state changes, which is less than optimal if you want something to nicely animate in and out.
- A-Frame works very nicely for static VR & AR experiences. You load all your assets up front as the page loads, then you jump into VR or AR, and all is good. However, loading additional assets once you are in the experience is tricky because javascript runs on a single thread in the browser. Particularly in VR, this can lead to many dropped frames creating an unpleasant experience for the user. Some of the work can be offloaded to a WebWorker, but the actual rendering still have to occur on the main thread.
- Back to my story. What happened next was that Apple released ARKit. And shortly after Google released ARCore. And we knew we were onto something big if we could add web-based AR support to our platform.
- So first we used Google’s Chrome build with AR support to get some basics working. But while three.ar.js from Google works well - also on iOS I should say - it only works with special browser builds that are not available through the app stores.
- So we were quite excited when Mozilla released their take on a WebXR polyfill and soon after put an AR Browser in the App Store for iOS. Finally something we could take to customers! Mozilla’s AR browser is called WebXR Viewer and is meant for experimental purposes only. Since any startup is essentially an experiment, we thought that was a perfect fit.
- So we refactored our AR solution to use the WebXR polyfill and work with the WebXR Viewer. Still using the same basic stack, but adding in Mozilla’s aframe-xr library to facilitate easy use from our A-Frame based solution.
- One of our customers, Muuce; an ecommerce website selling framed poster art, already had our solution with 3D and VR support integrated. Now, we were able to also enable AR support on their site, and you can go to muuuce.com yourself right now if you want to see this in action.
- One of the challenging things about worked with web based AR, is that you only have access to a limited subset of the features ARKit and ARCore provide. One example is that you don’t know the state of the plane detection directly. So you don’t directly know if the user needs to scan the room some more, if there are too few visual features, too bad light conditions and so on. But there’s a work around for that: We start hit testing - that is; trying to detect a plane - as soon as the web application has loaded. When we get the first hit, we know that AR is initialized. It’s crude, but it works.
- A great advantage of web based AR and VR is that you have all the power of the browser available. And you can easily integrate an experience with any existing solutions out there. We thought about how we could bring our AR experiences out of the websites and into the real world.
- So we teamed up with Geenee, which is a very powerful visual recognition engine. It’s super useful together with AR because it can recognize images and videos from challenging angles and distances, with no need to walk up close or click a capture button. Geenee is breaking down the barriers between the real world and the augmented world, and our favorite thing about it is that they support fully web based image recognition running straight out of the browser. That’s a perfect match for our web based AR solution. Let me show you how we recognize an outdoor ad for a movie, and then trigger an AR experience – all from the phone’s browser.
- Since this is a tech talk, I’ll also tell you about the challenges of such an integration. Apple has decided that you can only access the camera directly from the browser in Mobile Safari. Any other browser on iOS, be it Chrome, Firefox or the WebXR Viewer, is not allowed to do that. That’s why we have to start out in Safari for the image recognition part, then switch over to the WebXR Viewer for the AR part. Luckily, that’s all going to change as soon as web based AR becomes available in the main browsers, something I believe Blair will enlighten us on in this talk coming right after this. So why do AR & VR need to be web based? I may hurt someone’s feelings when I say this ...
- … but apps aren’t the solution to everything. It takes trust and great interest in a brand to go through the hassle of installing their app, and after having installed an app, users rarely return to it. In particular when shopping online, consumers don’t want to have to install an app just see the product, but having AR instantly available with the click of a button may be the deciding factor converting them to a sale.
- Instant apps for Android just got support for ARCore, so that’s an interesting middle ground we are certainly looking at. But apps still have the fundamental problem of Google and Apple being in control of what we can out out there and when. For Apple in particular, initial app reviews can take weeks to complete, and sometimes it feels like your app gets rejected simply because the reviewer is having a bad day.
- So where are we in practice? What browsers and devices support web based VR and AR today? It always dangerous to do this kind of overviews, because things change literally all the time and there’s bound to be plenty of people here that knows more than me. So bear with me. It’s looking pretty good for WebVR. A lot of good people at Google, Mozilla and Microsoft have worked hard to push things out. (Firefox: Also the upcoming Realities browser)
- For augmented reality, support is also improving. Mozilla putting WebXR Viewer in the App Store was a game changer for us, and Google’s recent announcement that AR support is coming in Canary soon is also fantastic. I’m happy that web based AR is also available for Meta 2 and Hololens, but with both of these devices still being only available as dev kits, it’s not really something we can build a business around at this point.
- Hopefully, you’re as excited as me about where all of this is going. Augmented reality is no longer just for expensive entreprise headsets, but is actually reaching the consumer. Mobile phones are the first step in wide adoption, but as we clearly see here at AWE, much more immersive AR tech is on it’s way to the masses. As creators of VR and AR experiences, we need to reach the consumers where they are today, and where they will be in the future. And that’s in the browser. Get in touch if you want to talk. Thank you.