Successfully reported this slideshow.

Building the Matrix: Your First VR App (SVCC 2016)

2

Share

Upcoming SlideShare
อ้างอิง
อ้างอิง
Loading in …3
×
1 of 59
1 of 59

Building the Matrix: Your First VR App (SVCC 2016)

2

Share

Download to read offline

The slides from my talk, Building The Matrix: Your First VR App at Silicon Valley Code Camp, Oct. 2016. Development, design, and sample projects for virtual reality applications.

The slides from my talk, Building The Matrix: Your First VR App at Silicon Valley Code Camp, Oct. 2016. Development, design, and sample projects for virtual reality applications.

More Related Content

Related Books

Free with a 14 day trial from Scribd

See all

Related Audiobooks

Free with a 14 day trial from Scribd

See all

Building the Matrix: Your First VR App (SVCC 2016)

  1. 1. BUILDING THE MATRIX: YOUR FIRST VR APP @MISSLIVIROSE 1
  2. 2. ABOUT ME Livi Erickson Livi.Erickson@Microsoft.com  Virtual & Augmented Reality Developer / Evangelist  VR Blog: The Matrix is my Office  @MissLiviRose on Twitter 2
  3. 3. BUILDING VIRTUAL REALITY APPLICATIONS  Why Build for Virtual Reality?  Understanding the VR Device Ecosystem  Breakdown of a VR application  Building a VR app: Development Options  Design Considerations & Concerns for VR today  Q & A 3
  4. 4. WHY BUILD FOR VR? 4
  5. 5. 5 It’s Really, Really Fun.
  6. 6. 6 We’re Used to 3D Interactions
  7. 7. 7 The Freedom to Redefine Possibilities
  8. 8.  Storytelling is significantly more immersive in an environment that surrounds the user  Virtual experiences improve empathy in the viewer  Visualizations of complex data  New and improved ways of interacting with the technical ecosystem that we’ve been building for decades
  9. 9. VR DEVICE ECOSYSTEM DESKTOP // MOBILE 9
  10. 10. PLATFORM GROWTH  2012:  Oculus Developer Kit 1  2014:  Oculus Developer Kit 2  Cardboard  GearVR Innovator Edition 1  2015:  GearVR Innovator Edition 2  Gear VR  HTC Vive Developer Kit 1  OSVR  2016:  Oculus Rift  HoloLens Developer Kit  HTC Vive Pre  HTC Vive  PlayStation VR  FOVE Developer Kit  Gear VR
  11. 11. TODAY’S TECHNOLOGY Virtual Reality Fully immersive experience that replaces your physical world Example: Oculus Rift Augmented Reality Overlays digital information about the physical world around you Example: Google Glass Mixed Reality Combines virtual objects and the physical world Example: Microsoft HoloLens
  12. 12. HEAD MOUNTED DISPLAYS Head Mounted Displays (HMDs) can be desktop- powered devices or mobile-powered devices  Desktop: Separate display device self-containing screen, content provided externally from a separate PC  Mobile: Content provided and displayed by a smart phone inserted into the casing  Standalone: Entirely self-contained headsets 12
  13. 13. DESKTOP VIRTUAL REALITY Powered from external computers Hardware includes separate screens Treated as an external display at the hardware level Software runs on the desktop and is passed to the display Examples: Oculus Rift, HTC Vive, FOVE 13
  14. 14. MOBILE VIRTUAL REALITY  Headsets are containers that support stereoscopic rendering for mobile phones  Applications are typical smartphone apps  The display is the phone screen  Usually, almost all computational aspects are done on the phone rather than the headset  Examples: GearVR, Homido, FreeFly, Cardboard 14
  15. 15. INPUT - CONTROLLERS  Traditional controllers don’t solve the “I can’t see my hands” issues with VR  Navigation is less natural for movement with joysticks due to the sense of being in one place  Limited number of inputs / actions  Some apps may rely on specific types of input controllers and have compatibility problems 15
  16. 16. INPUT - WEARABLES  Devices that are lower profile and can be worn to control various inputs without specific controller input  Example: Nod Ring  Example: Myo Bracelet  Example: Emotiv Headset 16
  17. 17. HAPTICS  Components to a VR system that provide tactile feedback based on what is going on in the virtual environment  Add to realism of a VR system by allowing the user to physically touch and experience things going on in virtual reality  Example: GloveOne (pictured) has multiple feedback points that mimic the sense of touch based on collisions in a VR game 17
  18. 18. MORE OPTIONS  Gestures via cameras: example: Leap Motion Controller  Omnidirectional treadmills  Voice commands 18
  19. 19. OVERVIEW OF A VR APPLICATION 19
  20. 20. DESIGNING & PLANNING  Consider the platforms that you’re interested in targeting  Understand the input and design considerations for the given platform as you plan out your experience  Outline key scenes and emotions that you’re looking to capture in your app  Choose a development option that aligns with your platform of choice 20
  21. 21. 21 Render Loop Hardware / Input Update Game Logic Lifecycle of a VR App
  22. 22. Game Logic • Defines the behavior of your application components • Models and textures are defined for your scene • Scripts control behavior of the scene objects • Generally doesn’t rely on the status of the hardware, but will likely contain checks for changes Render Loop Hardware / Input Update Game Logic 22
  23. 23. Hardware / Input • Check on the orientation of the device to calculate how the camera should render objects in the scene • Evaluate if there are actions triggering different behaviors in your scene • Update scene variables or components based on input checks Render Loop Hardware / Input Update Game Logic 23
  24. 24. Render Loop • Evaluate the scene appearance based on the orientation of the headset • Apply effects based on changing scene (shadows, lighting, etc.) dynamically to match the user’s expectation • Draw to the headset / display • Example: If player has moved under a tree, update the shadows cast by the player accordingly Playback Required Refresh Rate Film / TV 24 – 48 FPS Console Gaming 30 FPS PC 50-60 Hz VR 90-120 Hz Render Loop Hardware / Input Update Game Logic 24
  25. 25. DESIGNING FOR VR 25
  26. 26. INTERACTION & DISCOVERY  There are no standards yet in VR for expected behaviors and interfaces  Capturing and guiding a user through a process is a new challenge for the medium  Agency of a player is critical  The amount of control or interaction a user can have in an environment will vary dependent on experience and platform 26
  27. 27. PRESENCE IN VR  Consider how your user will want to interact with the environment  Environment scale will impact how realistic a scene feels to a player  Non-autonomous character actions will break the sense of self 27
  28. 28. Eyes: We’re Moving! Body: No, we aren’t! Body: We’re moving! Eyes: No, we aren’t! 28
  29. 29. DISPLAY PERFORMANCE & CONSIDERATIONS  Low Framerates = Motion Sickness  “Tearing” around the edges of the display  Shadowing (duplication) of scene objects at high framerates  Motion blur 29
  30. 30. TOOLS OF THE TRADE UNITY, UNREAL ENGINE, WEBVR (OH MY!) 30
  31. 31. DEVELOPMENT OPTIONS  Native development with Platform SDKs  Game Engines:  Unity  Unreal  CryEngine  Web:  WebGL VR support through WebVR  Three.JS  A-Frame  Vizor 31
  32. 32. EXAMPLE NATIVE TOOLS: OCULUS SDK  Add stereoscopic, distortion, and update effects within the graphics pipeline of your application  Render to a texture specifically output to the Oculus device as well as to a non-stereoscopic screen view  Great for custom game engines adding in VR support with the Rift  More customization and control granted  Versions available for desktop (Rift) and mobile (GearVR) 32 Source: developer.oculus.com
  33. 33. UNITY  Game Engine from Unity Technologies  Visual editor for environment building & scripting  Languages: C#, UnityScript (a JS derivative), Boo 33
  34. 34. 34
  35. 35. UNITY: VR MODE  Added in 5.1 (current is 5.4)  Requires runtime for headsets installed  Supports a number of devices built-in, many others have packaged support add-ins  Implementing it: 1. Edit -> Project Settings -> Player 2. Under “Other Settings” check the ‘Virtual Reality Supported’ box 3. Platform dependent on device type 35
  36. 36. UNREAL ENGINE  Game Engine from Epic Games  Visual editor for environment building & scripting  Language: C++ 36
  37. 37. 37
  38. 38. BENEFITS OF GAME ENGINES  Handles many of the hardware-level interactions with devices  Templates for different app types are available  Implements physics and lighting engines  Robust communities for asset creation  Additional services for common application needs (networking, multiplayer, awards)  Central tooling for cross-platform development 38
  39. 39. DOWNSIDES OF GAME ENGINES  Less granular control over application behaviors  Reliant on implementation of SDKs and devices  Black box code for engine functionality 39
  40. 40. WEB VR  Web applications with VR support  Front-end Rendering:  WebGL  Three.JS, Babylon.JS  Desktop & mobile support with one codebase  www.webvr.info 40
  41. 41. BENEFITS OF WEBVR  No installation of application is needed – run in the browser on any device  Automatically support mobile and desktop devices  Utilize the current set of tools and libraries for JavaScript  Easily switch between VR and non-VR mode  Offload interfacing with hardware to the browser 41
  42. 42. DOWNSIDES OF WEBVR  Performance hits for non-native graphics  Non-standard support for some browsers  Less insight and control into bugs with hardware  Relies on browser detection of devices 42
  43. 43. PLANNING YOUR EXPERIENCE 43
  44. 44. THINK ABOUT THE FOLLOWING CONSIDERATIONS:  3D Development  Scripting  Animation & 3D Modeling  Audio  Testing 44
  45. 45. PLANNING A VR APPLICATION  Evaluate how the medium lends itself to your game or application  Minimize the amount of locomotion  Choose one platform to target at a time  Read up on best practices, but don't lock things in  Prepare to iterate - a lot! 45
  46. 46. IMPLEMENTATION OF A VR APP - MY STRATEGY  Build out the basics in your environment so that you have a rough understanding of what your application components will do  Script the behaviors for gameplay elements & the rest of the application components  Finalize the environment with lighting, textures, effects 46
  47. 47. TESTING YOUR VR APP  One person QA is not quite enough!  Variations in height, gender, life experiences, comfort with technology, etc. will all impact your user to a much greater extent in VR  Performance is hugely important to prevent motion sickness 47
  48. 48. SAMPLE PROJECTS SOURCE CODE & TUTORIALS AVAILABLE! 48
  49. 49. SPACE PHOTOS IN SPACE 49 Space Photography Viewer Built using WebVR https://github.com/misslivirose/webvrspace
  50. 50. BUBBLE BLASTER 50 A mini-game for the HTC Vive Built with Unity and the SteamVR Toolkit https://github.com/misslivirose/vive_bubbleblaster
  51. 51. STAR WARS VR TEXT EFFECT 51https://github.com/misslivirose/SWScrollText Sample Experience built with World Canvas for VR Text Made in Unity
  52. 52. VISUAL DATA 52https://github.com/misslivirose/visualdata WebVR App for Visualizing Excel Charts ASP.NET with Three.JS WebVR Site
  53. 53. FINAL CONSIDERATIONS 53
  54. 54. TODAY'S CHALLENGES  Input devices are non-standardized across platforms  Constant evolution in tools  Market Size 54
  55. 55. RESOURCES  Blog: http://livierickson.com/blog for short-form VR development tutorials, reviews, and news  GitHub: http://github.com/misslivirose for all the code in my VR experiments (including Kitten VR!)  ARVR Academy Introductory course materials: http://github.com/arvracademy/intro-academy/  Just A/VR Show: http://justavrshow.com/  Learning Unity: http://livi.link/learnunity 55
  56. 56. Andrew Moll Technical Evangelist Doris Chen Technical Evangelist John Alioto Principal Technical Evangelist Tobiah Zarlez Technical Evangelist Daniel Egan Technical Evangelist Livi Erickson Technical Evangelist Tim Reilly Technical Evangelist Jeremy Foster Technical Evangelist Tierney Wixted Community Evangelist Annie Bubinski Community Evangelist Shaina Houston Director of Community Evangelism Yesenia Alvarez Community Evangelist Joe Shirey Director of Technical Evangelism Beth Massi Program Manager Jennelle Crothers Technical Evangelist Robin Shahan Content Developer Jessica Deen Technical Evangelist Justin Chizer Microsoft Student Partner MEET THE TEAM
  57. 57. Paul Hacker Visual Studio and Dev Tech Mathias Brandewinder Visual Studio and Dev Tech Deborah Kurata Visual Studio and Dev Tech Jeremy Clark Visual Studio and Dev Tech David McCarter Visual Studio and Dev Tech Ward Bel Windows Development Vishal Saxena Microsoft Azure Craig Berntson Visual Studio and Dev Tech Chander Dhall Visual Studio and Dev Tech Steve Evans Visual Studio and Dev Tech MEET OUR MVP/RDS
  58. 58. • Complete the survey for a chance to win Beats by Dre Headphones • Microsoft Azure Essentials book giveaway for survey completion while supplies last Survey  aka.ms/SVCC • Connect with our speakers Visit our booth • Fill out a session card to enter a raffle for a Ricoh Theta 360 camera Submit Your Session Cards MAKE THE MOST OF YOUR EXPERIENCE
  59. 59. 59 @misslivirose

Editor's Notes

  • Why do we care about virtual and augmented reality as a platform?

    We live in a three dimensional world and computing in its current form doesn’t translate well into 3D experiences
    Humans rely on their spatial awareness for so many things in the real world, and it’s time to make computing represent that as well
    When we create fully immersive experiences, we can have profound impact on emotional response

    Talk briefly about VR as a tool for:
    Distraction (pain therapy)
    Emotional well-being (phobia treatment)
    Empathy

    As well as:
    How memories in VR form like real-world memories
  • Why do we care about virtual and augmented reality as a platform?

    We live in a three dimensional world and computing in its current form doesn’t translate well into 3D experiences
    Humans rely on their spatial awareness for so many things in the real world, and it’s time to make computing represent that as well
    When we create fully immersive experiences, we can have profound impact on emotional response

    Talk briefly about VR as a tool for:
    Distraction (pain therapy)
    Emotional well-being (phobia treatment)
    Empathy

    As well as:
    How memories in VR form like real-world memories
  • Why do we care about virtual and augmented reality as a platform?

    We live in a three dimensional world and computing in its current form doesn’t translate well into 3D experiences
    Humans rely on their spatial awareness for so many things in the real world, and it’s time to make computing represent that as well
    When we create fully immersive experiences, we can have profound impact on emotional response

    Talk briefly about VR as a tool for:
    Distraction (pain therapy)
    Emotional well-being (phobia treatment)
    Empathy

    As well as:
    How memories in VR form like real-world memories
  • Also talk about how dedicated two-handed controller tracking
  • Ask the audience how many of them are game developers. Spend a little more time talking about the principles of game development cycles if most of them are unfamiliar. If the general body has a game development background, it will be okay to go through this part quickly.
  • Understand the limitations of the platforms that you’re using, especially around input
  • Let’s talk about specific things to consider when designing and developing for VR
  • Emphasize the importance of presence here:

    In making people feel comfortable
    In their actions being directly correlated to a VR action
    Not to force action on the user

    Describe the proprioceptive system
  • Talk about how the options differ and why some may be more beneficial to others:

    Native: Great for fine control, custom engine support, existing games, interesting control over effects and mono/stereoscopic
    Game Engines: Awesome for not reinventing the wheel, provide a solid implementation of existing game components (e.g. lighting, physics, etc.)
    Web: Lower performance, but great for experiments that utilize existing APIs and pull in data from the web
  • Include that this is C++
  • Discuss how WebVR enables developers to use the same code base to run on mobile and desktop
    Explain the specialized browsers required and how the browser handles hardware updates and transforms
  • ×