The Challenges of!
Creating VR Applications!
in Unity
Sébastien Kuntz!
Sebastien.Kuntz@imin-vr.com!
CEO
Sébastien Kuntz
!

•

Founder & President!

• 12 years in VR!
!

•

- French railways!
!

•
• Simplify VR!
• Creation!
• Deployment!
• So YOU can spend more time creating
applications
Plan
• Unity!
• Challenges!
• Presence!
• Device management!
• Display management!
• Cluster management!
• Interactions!
•...
Plan
Unity!
• Challenges!
• Presence!
• Device management!
• Display management!
• Cluster management!
• Interactions!
• D...
• Platform for creating interactive 3d applications!
• Widely used!
• 2 million registered developers!
• Certified by the U...
• High-quality rendering!
• Deferred renderer!
• Occlusion culling!
• Realtime shadows!
• Ambient occlusion!
• Light mappi...
Unity 3D
Features
• Programming!
• C#, Javascript!

3D formats !

• Profiler!
• Physics!
• NVidia PhysX!
• Cloth!
• Soft/ri...
Editor

• Simple to use!
• Create your own tools inside Unity
Asset store
• Buy / sell assets!
• 3D models!
• Textures!
• Shaders!
• Scripts!
• Editor plugins
Desktop
Plan
• Unity!

Challenges!
• Presence!
• Device management!
• Display management!
• Cluster management!
• Interactions!
• ...
Plan
• Unity!
• Challenges!

Presence!
• Device management!
• Display management!
• Cluster management!
• Interactions!
• ...
Virtual reality
Presence in a virtual world

(c) eMagin

(c) Barco
Experience of presence
is in your brain !
Cognitive presence
Perceptive presence
To create and maintain presence,
you need to ...
• Build hardware
• Build software
• Build user experience
Plan
• Unity!
• Challenges!
• Presence!

Device management!
• Display management!
• Cluster management!
• Interactions!
• ...
Device management

Physical device!
!

Driver

Application
Device management
Tracker 1!

Driver Tracker 1
Application

Tracker 2!
!

Driver Tracker 2
Plan
• Unity!
• Challenges!
• Presence!
• Device management!

Display management!
• Cluster management!
• Interactions!
• ...
Correct perspective

(c) Johnny Lee
Correct perspective
Stereoscopy
Warping & Blending
Graphics cards
limitations

• Number of outputs!
• Processing power
multiple
graphics cards
• Pro!
• More outputs!
• More processing power!
• Con!
• 3D engines are not optimized for multi-pi...
Cluster
• Pro!
• More outputs!
• More processing power!
• Con!
• 3D engines don’t support clustering
Cluster

• No seams across displays require 3 layers of sync!
• 3D objects synchro (Scenelock)!
• New images display (Swap...
Plan
• Unity!
• Challenges!
• Presence!
• Device management!
• Display management!
• Cluster management!

Interactions!
• ...
Interactions
• Many different 3D user interactions!
• Navigation!
• Selection!
• Manipulation!
• Draw!
• System control!
•...
Interactions are
dependent on hardware
Menus
Haptics
(force feedback)
Plan
• Unity!
• Challenges!
• Presence!
• Device management!
• Display management!
• Cluster management!
• Interactions!

...
Deployment

• Modify application to!
• Change drivers!
• Change displays!
• Change interactions
Plan
• Unity!
• Challenges!
• Presence!
• Device management!
• Display management!
• Cluster management!
• Interactions!
•...
Rendering speed
Plan
• Unity!

Challenges!
• Presence!
• Device management!
• Display management!
• Cluster management!
• Interactions!
• ...
Application
Plan
• Unity!
• Challenges!
• Presence!
• Device management!
• Display management!
• Cluster management!
• Interactions!
•...
To create and maintain presence,
you, the author, need to ...
• Build hardware
• Build software
• Build user experience
“VR is mainly a software problem”!
Jaron Lanier
What is MiddleVR ?
• VR plugin
!

• Functionalities
○Manage input devices
•3D trackers, keyboard, mouse, joystick
!

○Mana...
Desktop
MiddleVR tutorial

http://www.youtube.com/watch?v=EEKH-1EilXs
Device management
Device management
Supported Devices
• Via VRPN :
• A.R.T
• Vicon
• Optitrack
• Wiimote
!
• Polhemus Fastrak Tracker, 3Space Tracker,
Patriot...
First integration of
Haption’s haptic devices

http://www.youtube.com/watch?v=lnoE_VclnOI
Stereoscopy

• Active stereoscopy
(OpenGL Quad-Buffer)!
• Which Unity can’t
natively handle!
• Passive stereoscopy
Cluster & Multi-pipe

• Scenelock!
• Swaplock!
• Genlock!
• Multi-pipe (soon)
Supported displays
•
•
•
•
•
•
•
•
•

Virtually any HMD
VR-Wall, Powerwall
Workbench, Holobench
HoloStage
CAVEs
3D TVs
zSp...
Deployment

• Abstractions!
• Devices!
• 3D Nodes (User)!
• Interactions (soon)
Device management
Driver Tracker 1
Application

Driver Tracker 2
Device abstraction
Driver Tracker 1

Virtual
Device
Driver Tracker 2

Application
Device management
Driver

Replay

Simulation

Virtual
Device

Application
User abstraction

Virtual Device 1

Virtual Device 2

Inverse Kinematics

Hand
(3D Node)

Application
Interactions
• MiddleVR includes high-level interactions!
• Navigation!
• Selection!
• Manipulation!
• … More to come!
• F...
Interactions
abstraction?

Grab-air
Steering

Navigation

Point
Raycasting
Hand

Selection

Go-go
Homer
WIM
Hand

Manipula...
Menus
• Ability to create menus and GUIs in HTML5 (soon)
• Unity!

Speed

• Many options for optimizations!
• Occlusion culling!
• Static batching!
• Lightmap baking!
• etc.!
• 32...
Compositing
Stereoscopy
Compositing
Image decomposition
VR in the Cloud
• Temporary increase in power!
• Combine with compositing
Perceptive illusions

• Playing with limitations of human perception!
• Sensitivity of senses!
• Selectivity of attention!...
Perceptive illusions
Redirected walking

(c) Franck Steinicke - Wurzburg
University
Perceptive illusions
Change blindness
Change blindness :

(c) Evan Suma - USC/ICT
Conclusion
!
!

Thank you !
Questions ?
!

Sebastien.Kuntz@imin-vr.com
SAE AR/VR - The challenges of creating a VR application with Unity
SAE AR/VR - The challenges of creating a VR application with Unity
SAE AR/VR - The challenges of creating a VR application with Unity
SAE AR/VR - The challenges of creating a VR application with Unity
SAE AR/VR - The challenges of creating a VR application with Unity
SAE AR/VR - The challenges of creating a VR application with Unity
SAE AR/VR - The challenges of creating a VR application with Unity
Upcoming SlideShare
Loading in...5
×

SAE AR/VR - The challenges of creating a VR application with Unity

1,212

Published on

Unity3D is a great authoring tool to create 3D applications.
Regarding immersive virtual reality, its capabilities are limited.
We will discuss the challenges and solutions to create great VR applications in Unity!

Published in: Technology, Art & Photos
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,212
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
180
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

SAE AR/VR - The challenges of creating a VR application with Unity

  1. 1. The Challenges of! Creating VR Applications! in Unity Sébastien Kuntz! Sebastien.Kuntz@imin-vr.com! CEO
  2. 2. Sébastien Kuntz ! • Founder & President! • 12 years in VR! ! • - French railways! ! •
  3. 3. • Simplify VR! • Creation! • Deployment! • So YOU can spend more time creating applications
  4. 4. Plan • Unity! • Challenges! • Presence! • Device management! • Display management! • Cluster management! • Interactions! • Deployment! • Rendering speed! • MiddleVR for Unity! • Conclusion
  5. 5. Plan Unity! • Challenges! • Presence! • Device management! • Display management! • Cluster management! • Interactions! • Deployment! • Speed! • MiddleVR for Unity! • Conclusion
  6. 6. • Platform for creating interactive 3d applications! • Widely used! • 2 million registered developers! • Certified by the US Army and Air Force! • Clients : Bigpoint, Cartoon Network, CocaCola, Disney, Electronic Arts, LEGO, Microsoft, NASA, Nickelodeon, Ubisoft, US Army, Warner Bros.
  7. 7. • High-quality rendering! • Deferred renderer! • Occlusion culling! • Realtime shadows! • Ambient occlusion! • Light mapping
  8. 8. Unity 3D Features • Programming! • C#, Javascript! 3D formats ! • Profiler! • Physics! • NVidia PhysX! • Cloth! • Soft/rigid bodies! • Ragdolls! • Cars FBX, Collada, 3DS, OBJ…! Audio engine! Networking! Terrain
  9. 9. Editor • Simple to use! • Create your own tools inside Unity
  10. 10. Asset store • Buy / sell assets! • 3D models! • Textures! • Shaders! • Scripts! • Editor plugins
  11. 11. Desktop
  12. 12. Plan • Unity! Challenges! • Presence! • Device management! • Display management! • Cluster management! • Interactions! • Deployment! • Speed! • MiddleVR for Unity! • Conclusion
  13. 13. Plan • Unity! • Challenges! Presence! • Device management! • Display management! • Cluster management! • Interactions! • Deployment! • Speed! • MiddleVR for Unity! • Conclusion
  14. 14. Virtual reality Presence in a virtual world (c) eMagin (c) Barco
  15. 15. Experience of presence is in your brain !
  16. 16. Cognitive presence Perceptive presence
  17. 17. To create and maintain presence, you need to ... • Build hardware • Build software • Build user experience
  18. 18. Plan • Unity! • Challenges! • Presence! Device management! • Display management! • Cluster management! • Interactions! • Deployment! • Speed! • MiddleVR for Unity! • Conclusion
  19. 19. Device management Physical device! ! Driver Application
  20. 20. Device management Tracker 1! Driver Tracker 1 Application Tracker 2! ! Driver Tracker 2
  21. 21. Plan • Unity! • Challenges! • Presence! • Device management! Display management! • Cluster management! • Interactions! • Deployment! • Speed! • MiddleVR for Unity! • Conclusion
  22. 22. Correct perspective (c) Johnny Lee
  23. 23. Correct perspective
  24. 24. Stereoscopy
  25. 25. Warping & Blending
  26. 26. Graphics cards limitations • Number of outputs! • Processing power
  27. 27. multiple graphics cards • Pro! • More outputs! • More processing power! • Con! • 3D engines are not optimized for multi-pipe
  28. 28. Cluster • Pro! • More outputs! • More processing power! • Con! • 3D engines don’t support clustering
  29. 29. Cluster • No seams across displays require 3 layers of sync! • 3D objects synchro (Scenelock)! • New images display (Swaplock)! • Left/Right eyes (Genlock)
  30. 30. Plan • Unity! • Challenges! • Presence! • Device management! • Display management! • Cluster management! Interactions! • Deployment! • Speed! • MiddleVR for Unity! • Conclusion
  31. 31. Interactions • Many different 3D user interactions! • Navigation! • Selection! • Manipulation! • Draw! • System control! • etc.
  32. 32. Interactions are dependent on hardware
  33. 33. Menus
  34. 34. Haptics (force feedback)
  35. 35. Plan • Unity! • Challenges! • Presence! • Device management! • Display management! • Cluster management! • Interactions! Deployment! • Speed! • MiddleVR for Unity! • Conclusion
  36. 36. Deployment • Modify application to! • Change drivers! • Change displays! • Change interactions
  37. 37. Plan • Unity! • Challenges! • Presence! • Device management! • Display management! • Cluster management! • Interactions! • Deployment! Rendering speed! • MiddleVR for Unity! • Conclusion
  38. 38. Rendering speed
  39. 39. Plan • Unity! Challenges! • Presence! • Device management! • Display management! • Cluster management! • Interactions! • Deployment! • Rendering Speed! • MiddleVR for Unity! • Conclusion
  40. 40. Application
  41. 41. Plan • Unity! • Challenges! • Presence! • Device management! • Display management! • Cluster management! • Interactions! • Deployment! • Speed! MiddleVR for Unity! • Conclusion
  42. 42. To create and maintain presence, you, the author, need to ... • Build hardware • Build software • Build user experience
  43. 43. “VR is mainly a software problem”! Jaron Lanier
  44. 44. What is MiddleVR ? • VR plugin ! • Functionalities ○Manage input devices •3D trackers, keyboard, mouse, joystick ! ○Manage display •Viewports, cameras, stereoscopy, cluster
  45. 45. Desktop
  46. 46. MiddleVR tutorial http://www.youtube.com/watch?v=EEKH-1EilXs
  47. 47. Device management
  48. 48. Device management
  49. 49. Supported Devices • Via VRPN : • A.R.T • Vicon • Optitrack • Wiimote ! • Polhemus Fastrak Tracker, 3Space Tracker, Patriot Trackers, • • • • • • • • Ascension Flock of Bird Intersense IS600, IS900 3dTech HiBall-3000 Wide Area Tracker, Advanced Realtime Tracking Gmbh DTrack Client, World Viz Precision Position Tracker PPT 1.2, Natural Point Optitrack Rigid Body Toolkit, Phasespace optical tracking OWL, ... • Native drivers • Oculus Rift • Leap Motion • Kinect (Microsoft SDK) • PNI SpacePoint Fusion • GameTrak • Razer Hydra • Motion Analysis (beta)
  50. 50. First integration of Haption’s haptic devices http://www.youtube.com/watch?v=lnoE_VclnOI
  51. 51. Stereoscopy • Active stereoscopy (OpenGL Quad-Buffer)! • Which Unity can’t natively handle! • Passive stereoscopy
  52. 52. Cluster & Multi-pipe • Scenelock! • Swaplock! • Genlock! • Multi-pipe (soon)
  53. 53. Supported displays • • • • • • • • • Virtually any HMD VR-Wall, Powerwall Workbench, Holobench HoloStage CAVEs 3D TVs zSpace Igloo Vision …
  54. 54. Deployment • Abstractions! • Devices! • 3D Nodes (User)! • Interactions (soon)
  55. 55. Device management Driver Tracker 1 Application Driver Tracker 2
  56. 56. Device abstraction Driver Tracker 1 Virtual Device Driver Tracker 2 Application
  57. 57. Device management Driver Replay Simulation Virtual Device Application
  58. 58. User abstraction Virtual Device 1 Virtual Device 2 Inverse Kinematics Hand (3D Node) Application
  59. 59. Interactions • MiddleVR includes high-level interactions! • Navigation! • Selection! • Manipulation! • … More to come! • Future: Interactions abstractions! • Dynamically change interactions based on hardware
  60. 60. Interactions abstraction? Grab-air Steering Navigation Point Raycasting Hand Selection Go-go Homer WIM Hand Manipulation Application
  61. 61. Menus • Ability to create menus and GUIs in HTML5 (soon)
  62. 62. • Unity! Speed • Many options for optimizations! • Occlusion culling! • Static batching! • Lightmap baking! • etc.! • 32-bit editor only! • But player is 32-bit & 64-bit !! • Subdivide parts in editor! • Compositing! • Cloud
  63. 63. Compositing Stereoscopy
  64. 64. Compositing Image decomposition
  65. 65. VR in the Cloud • Temporary increase in power! • Combine with compositing
  66. 66. Perceptive illusions • Playing with limitations of human perception! • Sensitivity of senses! • Selectivity of attention! • Useful to lower cost/complexity of VR systems
  67. 67. Perceptive illusions Redirected walking (c) Franck Steinicke - Wurzburg University
  68. 68. Perceptive illusions Change blindness Change blindness : (c) Evan Suma - USC/ICT
  69. 69. Conclusion ! ! Thank you ! Questions ? ! Sebastien.Kuntz@imin-vr.com
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×