The UX of Mixed Reality
•
3 likes•636 views
My session from http://uxaustralia.com.au August 2017 It feels like Virtual Reality is everywhere you look this year. For a technology that is over 55 years in the making, it seems like it’s taken a long time to become an “overnight success”. What is really driving this buzz and is it deserving of the hype? The context will be set as to why a perfect storm of Mixed Reality (including Augmented and Virtual Reality), Machine Learning and Artificial Intelligence are set to drive the next computing paradigm, much like mobile has done for the last 15-20 years, and the PC before that. What are the key components to these technologies that you will start using to solve design problems? How can you implement them in ways that create a frictionless, seamless experience for people across multiple devices (not just AR and VR goggles)? And what are the real world constraints that you need to keep in mind?
Recommended
More Related Content
What's hot
What's hot (20)
Similar to The UX of Mixed Reality
Similar to The UX of Mixed Reality (20)
Recently uploaded
Recently uploaded (20)
The UX of Mixed Reality
- 1. A L E X Y O U N G @ alexmyoung The UX of Mixed Reality
- 5. What we’re covering WHAT IS MR/XR/AR/VR o History – know what’s come before o Making sense of the language & labels o Different technologies & approaches THE UX LAYER o The first use experience & roadblocks o Making sense of a spatial environment o Guiding principles o The on-boarding process
- 7. What are VR & AR? “It takes 20 years to become an overnight success”EDDIE CANTOR
- 8. A short history… The link simulator otherwise we’ll be here all day 1929
- 9. Antonin Artaud French poet, playwright, actor. Coined the term “Virtual Reality” 1938
- 11. 1960 1966 TELESPHERE MASK SWORD OF DAMOCLES
- 14. 2010 Kinect AND FROM THERE WE KNOW WHAT HAPPENS FOR VR…
- 15. So where did AR come from? 1990 Tom Caudell
- 16. 1994Milgram’s Mixed Reality Continuum Image source: https://cdn2.hubspot.net/hubfs/479277/blog-files/Reality_Continuum.png
- 17. 1994Milgram’s Mixed Reality Continuum Demo running entirely in a web browser using awe.media https://www.youtube.com/watch?v=fhJSfqBYjKE
- 18. Definition of AR Augmented Reality allows the user to see the real world, with virtual objects superimposed upon or composited with the real world. Augmented Reality: RON AZUMA (1997) 1. Combines real and virtual 2. Interactive in real time 3. Registered in 3-D
- 19. AND NOW TO XR…
- 20. 360º WHAT DO WE MEAN BY 360º OR PANOSPHERE? TYPE OF “360º” – PANORAMA, ECQUIRECTANGULAR, SPHERICAL STATIC VS INTERACTIVE USE CASES – TELEPORTING, VIRTUAL TOUR-GUIDING, EDUCATION & TRAINING CAN BE LINKED TO PHYSICAL LOCATIONS
- 21. VR HOW IS VR DIFFERENT TO 360º? DISCOVERY VS. LED THROUGH CHALLENGES MORE COMPLEX STORYTELLING, CREATION GENERALLY MORE EXPENSIVE THAN 360º CONTENT
- 22. The multi-flavoured landscape of AR LOCATION o Relative o Geo or “Fixed” o Sensor COMPUTER VISION o Recognition / Visual Search o Tracking o SLAM
- 23. Content is relative to the viewer rather than a physical location Location-based AR RELATIVE
- 24. HOW DOES IT WORK? HOW ACCURATE IS IT? Location-based AR FIXED or GEO
- 25. WHAT IS IT? WHY WOULD YOU USE IT? Location-based AR SENSOR-BASED
- 27. 2D planes Spaces – SLAM (e.g. ARKit) DEPTH SENSING Computer vision TRACKING
- 28. Touch Controllers Sensors – motion / gesture / depth Physical input
- 30. SLIDE:Modes of mixed reality PUBLIC PERSONAL PRIVATE INTIMATE
- 31. PUBLIC WHOLE BODY IN FOV MORE SPECTATORS MR MODES LIMITED DISTRIBUTION
- 32. PERSONAL MR MODES WHOLE BODY / UPPER TORSO IN FOV FEWER SPECTATORS WIDER DISTRIBUTION
- 33. PRIVATE MR MODES ONLY EXTREMETIES OR HEAD IN FOV 1-2 PARTICIPANTS WIDE DISTRIBUTION & LOCATIONS
- 34. INTIMATE MR MODES ONLY EXTREMETIES IN FOV MOSTLY 1 PARTICIPANT * STILL LIMITED DISTRIBUTION ALTHOUGH GROWING * Except for shared VR and AR experiences
- 35. HMDs market size
- 36. Samsung 21% Sony 19% HTC 8%Oculus 4% Others 47% HMDs shipped Jan-Mar this year Source: IDC Worldwide Quarterly Augmented and Virtual Reality Headset Tracker report Note: “Others” is predominantly Daydream and cardboard devices.
- 37. 37 iOS is slipping markedly The smartphone market – UNITS SHIPPED
- 38. How many people can use AR & VR right now? Installed web browsers 26 MILLION (END 2017) Includes both the mobile and HMD users above 2 BILLION 3 BILLION Mobile devices HMD only
- 39. Emerging Technologies Hype Cycle GARTNER, 2016 http://blogs.gartner.com/smarterwithgartner/files/2016/08/Emerging-Technology-Hype-Cycle-for- 2016_Infographic_revise2.jpg
- 40. Is this all a passing fad? Don’t be fooled by the S-curve PAUL SAFFO http://mcgeesmusings.net/2005/12/22/paul-saffo-on-rules-for-forecasting/
- 41. Obstacles to use UX and content largest obstacles Guiding principles from a UX perspective (i.e. inclusiveness, first use, onboarding) Developing vs. viewing across different platforms including web Other issues with single platform/siloed approach
- 42. Guiding principles Planning – discovery, storyboarding, prototyping, user testing, etc. You’re introducing a spatial environment that relates to the real world around people, or if not completely then introduces real world elements and interactions. Make interaction patterns consistent and strive to make it feel familiar and non-threatening (unless that is your intent!) Inclusiveness If you’re developing for a single platform, don’t aim for fragmented silos. Think about how your audience on other platforms or unsupported devices can also take part or be involved in the experience. Smartphones & tablets will be the dominant form factor for at least the next 5+ years. HMDs are expensive. Example from education Minimise friction Are you solving a real problem, or just using the technology because you can? You need to make the complex simple.
- 43. “People don’t care about the technology or the label. They want magic, that just works!” MOST IMPORTANT PIECE OF ADVICE… SOURCE - ME
- 44. UX & CONTENT OBSTACLES Still largely in exploration phase Adapt and re-purpose content. You don’t always have to start from scratch. Creative content that supports the narrative. How do you want to guide people through the experience? If you set up rules and dictate flows, then make sure your audience understands what you need them to do. VR and AR is not just the domain of 3D modeling. Traditional forms of digital media can be used really effectively. How can it be integrated with existing web- based content (more on this later…) https://sketchfab.com/trends/q2-2017
- 45. How do you view VR & AR? Have a suitable HMD v 1 2Have a valid account with the HMD vendor's app store 3 Complete the HMD vendor's first use 4 v Search for & find the app in their app store 5 Download/install the app through their app store 6Often requires system or app updates also v 7 Select/play the downloaded app ! MOBILE ONLY APP Have a valid account with the Mobile OS vendor's app store v 1 2 Find the app in their app store3 Download/install the app through their app store 4 v Often requires system or app updates also 5 Download/install the app through their app store 6 Select/play the downloaded app v 7 ! View promo for app on their website WEB APP 1 Click on a web link to open it in your browser on any device HMD ONLY APP
- 46. How do you create VR & AR? MOBILE ONLY APPs WEB APPSHMD ONLY APPS Hire or be a developer v 1 2 Download & get to know (& deal with the various issues/quirks) the developer SDK for the target platform 3 Develop your app/s 4 v Test it on the target HMD vendor platform/s 5 Convert media and 3D assets to the app/s format 6 Sign & submit your app/s to the app store/s & wait for approval/s v 7 Publish & distribute links to one or more app stores ! Download & get to know (& deal with the various issues/quirks) the developer SDK for the target mobile OS v 1 2 Develop your app/s3 Test it on the target HMD vendor platform/s 4 v Convert media and 3D assets to the app/s format 5 Sign & submit your app/s to the app store/s & wait for approval/s 6 Publish & distribute links to one or more app stores v 7 Hire or be a developer 8 Create promotional web content to drive people to the app/s ! 8 Create promotional web content to drive people to the app/s Get an account with awe.media *blatant plug v 1 2 Upload any types of common media you need (photos, videos, 3D & more) 3 Manipulate these objects and add rich interactions to create your VR & AR experiences 4 Open your browser on any type of device v 5 Share a single link that points directly to your web-based VR/AR experience
- 47. WHAT MADE IT SO SUCCESSFUL? Content seeding Latent brand fan base Social across demographics THE LONG PLAY Community and partnerships THE MECHANICS Simple locative AR The Pokemon GO effect
- 48. Constraints in MR environments On-site experience Have you actually been to the locations? Physical obstacles Factors impacting location accuracy Line of sight Interior of buildings Gathering location data Location source not allowed by the user Have you considered what the experience will be like for people accessing off-site? Friction when moving between modes (demo: https://www.youtube.com/watch?v=FQ6DvK8Wb_k) LOCATION SPECIFIC
- 49. Constraints in MR environments Image / object quality Image print quality Light source Overlaid file sizes / object complexity RECOGNITION & TRACKING SPECIFIC
- 50. Our torch to bear is to make the increasingly more fragmented world of devices & technologies easy to adopt and use.
- 51. A L E X Y O U N G The UX of Mixed Reality
Editor's Notes
- - A bit about our last 10 years (super short) - awe platform
- https://www.youtube.com/watch?v=JnLk2_Xbr90
- Add to this abstract groupings of the agenda and tasks.
- https://cdn2.hubspot.net/hubfs/479277/blog-files/Reality_Continuum.png https://www.researchgate.net/profile/Coskun_Bayrak/publication/47508440/figure/fig1/AS:202501564440578@1425291543360/Milgram-et-al-1-defines-the-distinctive-worlds-on-virtual-reality.png
- Robman to add
- - Strip down to grey-scale - Look for feature definition, so basically points where there are clear contrast changes. These create unique patterns that can be used to identify the image. - Examples that don’t work so well… - Pastel colours – wall graphics example – basically becomes one hue of grey - Australia’s obsession with clean, white lines and bloody subway tiles! White bathrooms and kitchen real estate example, 1 million images.
- 2D planes – fiducial markers and NFT Once a trigger is identified, then objects can be positioned over or at a relative distance from the image. As long as the source trigger is in view the objects will “stick” to it, so you can change the orientation and the related objects will move with it. If you hide any part of the trigger, the related objects will disappear. SLAM Simultaneous localisation and mapping. Looks for features and builds a global map, so when they are out of sight it stills knows it’s there, unlike normal computer vision, where once say the trigger is out of sight it’s gone and any objects that were being tracked are no longer displayed. Portal example of ARKit - Depth sensing So normal cameras use RGB and depth cameras use other sensors to sense the space. So if you think of our eyes as a “normal” camera and, not that it uses sound, but a depth sensing camera more like a bat’s sonar. You can create 3D objects from everyday things, do scene reconstruction so in your AR view you could bounce digital balls off real walls and furniture for example and even be able to side digital objects behind real world objects.
- We’re seeing a massive fragmentation in this area. - Although there are “standards” they are not being fully adhered to and there are lot of new “default” standards being pushed into the market. - Even using standards or “standard” approaches to integrating cross-platform support for controllers has issues and needs adaptation across platforms and for different devices.
- WEARABLE SKIN: https://twitter.com/MixedReality11/status/881846276328968192
- The table represents shipments of Augmented Reality and Virtual Reality headsets only. Any additional software, services, or compute devices (e.g. PC, game console, smartphone, etc.) are excluded. Shipments of simplistic headsets that do not have any technology built in are excluded. Examples of such headsets include Google Cardboard and other Cardboard-like devices.
- Mary Meekers slide (and how does this impact on Apple’s ARKit market? Likely transition pathway
- http://blogs.gartner.com/smarterwithgartner/files/2016/08/Emerging-Technology-Hype-Cycle-for-2016_Infographic_revise2.jpg
- http://blogs.gartner.com/smarterwithgartner/files/2016/08/Emerging-Technology-Hype-Cycle-for-2016_Infographic_revise2.jpg - VR resurgence, physical hardware you can sell. AR still in left segment, but ARKit will likely change people’s perception of that.
- - Include VR/360 which part of your scene do you want to load In viewers fov initially? A part of the background or which way points north for example.
- Include VR/360 which part of your scene do you want to load In viewers fov initially? A part of the background or which way points north for example. Inclusiveness example: Can’t expect everyone to have immersive headsets/displays - the general expectation is that everyone will be walking around with these or have access to them, but in reality that’s not the case. They are expensive and we’re finding more often than not, even out of the league of all but very cashed up corporate cost centres. BUT, everyone has one of these… and this will be their primary tool for at least the next 8-10 years - adoption takes a lot longer than anyone predicts for new technologies. So make experiences inclusive… another reason for web-based - how expensive is it to create standalone experiences across multiple platforms and devices types.
- Include VR/360 which part of your scene do you want to load In viewers fov initially? A part of the background or which way points north for example. Inclusiveness example: Can’t expect everyone to have immersive headsets/displays - the general expectation is that everyone will be walking around with these or have access to them, but in reality that’s not the case. They are expensive and we’re finding more often than not, even out of the league of all but very cashed up corporate cost centres. BUT, everyone has one of these… and this will be their primary tool for at least the next 8-10 years - adoption takes a lot longer than anyone predicts for new technologies. So make experiences inclusive… another reason for web-based - how expensive is it to create standalone experiences across multiple platforms and devices types.