SEARIS 2014 Keynote - MiddleVR - Philosophy and architecture


Published on

MiddleVR is a generic VR plugin.
SEARIS is the "Software engineering and architecture for realtime interactive systems" workshop for IEEEVR.

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

SEARIS 2014 Keynote - MiddleVR - Philosophy and architecture

  1. 1. Sébastien ‘Cb’ Kuntz @Cb_VRGeek Philosophy and architecture! SEARIS 2014 SEARIS: Software engineering and architecture in interactive systems, an IEEE VR workshop.
  2. 2. When I was in engineering school, we had software architecture classes and projects. Without any experience, this is a very abstract topic When I was an intern at Discreet (Autodesk, 3DS Max, Flame, Inferno, movies sfx), was very impressed with the “architect”, and wanted to be like him ! Turns out it takes a lot of experience to be a good architect, so share my experience architecting MiddleVR and show that the philosophy that you choose, consciously or not, actually drives your architecture
  3. 3. Plan • Introduction! • Origin of MiddleVR! • Goals! • Demo! • Philosophy & Architecture! • Conclusion
  4. 4. (French railways) “Creating Virtual Reality Games:! The Fundamentals” Founder & President Sébastien ‘Cb’ Kuntz Also board member of the national French VR association AFRV ( and founded the VR Geeks association (
  5. 5. And it seems I have been interested in head-mounted displays for longer that I remember.
  6. 6. Zone hunter (Virtuality-1994) I fell in love with VR on a field trip in London when I was a teenager. The experience was far from perfect: the HMD was bulky, one display was blurry, but being able look around and shoot where I wanted incredible. I quickly decided I wanted to make a career in VR.
  7. 7. FRENCH RAILWAYS - SNCF • Polhemus magnetic tracker, treadmill, 5DT Glove • C++ / OpenSceneGraph /VRPN / OpenAL Fiacre: train the operators at maintenance of equipments on the railway tracks. I was in charge of all the VR engine and application development ! (sorry I can’t put SNCF videos online)
  8. 8. FRENCH RAILWAYS - SNCF Simurat: check wagons before they leave on the tracks
  9. 9. VIRTOOLS After 4 years I was hired by Virtools as their VR lead engineer. Virtools was a great tool to create 3D applications. Simple to learn thanks to the building blocks programming scheme, and many non-programmers, like designers and even farmers, created great applications.
  10. 10. VIRTOOLSVR PACK I was in charge of adding VR functionality's to Virtools engine, coding, writing documentation, support customers
  11. 11. I often say that we are at the prehistory of VR. Last week was the Big bang of VR. I could test Oculus DK2, Sony Morpheus and Valve’s HMD. The future is coming quickly.
  12. 12. • VR tools to simplify VR
  13. 13. Virtual reality Presence in a virtual world Interesting because: - natural interactions: no interface to learn. if we want to look under a table, no button to press, just move your body. - natural reactions & emotions : if I throw a ball at you, you will have the reflex to grab it. if you are afraid of heights, you will have this fear in VR.
  14. 14. Ridiculous If it’s real VR, you probably look ridiculous !
  15. 15. • Build hardware To create and maintain presence, you need to ... • Build software • Build user experience Most people forget about taking time to build the user experience. Test with users !
  16. 16. The challenges of creating a VR software We already have great hardware, but we don’t know how to make the most of it.
  17. 17. “VR is mainly a software problem”! !Jaron Lanier! ! Software is like a fuel for your car. Without any fuel, you will not go very far, even if you have a Porsche.
  18. 18. Device management Physical device! ! Driver Application An application needs to get the data from a physical device through a driver.
  19. 19. Device management Tracker 1! Driver Tracker 1 Application Tracker 2! ! Driver Tracker 2 The problem is that each driver gives access to similar data in a different way so you need to have specific part of programs for similar drivers.
  20. 20. code for one driver if you switch to another device, another driver:
  21. 21. This is a completely different code which gives exactly the same information but for another device.
  22. 22. And there are a lot of devices out there.
  23. 23. Correct perspective (c) Johnny Lee ! 3D applications assume that the user is in front of the screen. If you move in front of the screen, it looks like a 2d painting ! If the application knows where the user is, the screen becomes a window on the virtual world (perceived depth without stereoscopy)
  24. 24. Correct perspective In general 3D engines allow you to do that but you have to do all the maths by yourself Necessary for every big VR wall, anytime stereoscopy, HMDs, Caves... !
  25. 25. Stereoscopy Even after you have computed your stereo images, you have different ways to display them. It can be active stereo, passive stereo, autostereo etc.
  26. 26. Warping & Blending Then if you have a curved screen, your perspective will have to be distorted.
  27. 27. Warping & Blending Warping is about getting a correct perspective for screens that are not planar (curved) Blending is about managing the overlapping regions between several projectors.
  28. 28. Graphics cards limitations • Number of outputs! • Processing power
  29. 29. multiple graphics cards • Pro! • More outputs! • More processing power! • Con! • 3D engines are not optimized for multi-pipe
  30. 30. Cluster • Pro! • More outputs! • More processing power! • Con! • 3D engines don’t support clustering
  31. 31. • No seams across displays require 3 layers of sync! • 3D objects synchro (Scenelock)! • New images display (Swaplock)! • Left/Right eyes (Genlock) Cluster
  32. 32. Interactions • Many different 3D user interactions! • Navigation! • Selection! • Manipulation! • Draw! • System control! • etc.
  33. 33. Interactions are dependent on hardware For example navigation will be different for different VR systems
  34. 34. Menus Menus are also a big issue in VR.
  35. 35. Haptics (force feedback) If you need haptics, you will have to have a physics engine that run at 1000hz and interface with specific hardware.
  36. 36. Deployment • Modify application to! • Change drivers! • Change displays! • Change interactions
  37. 37. So to create a complete VR application, you have to manage all of this !
  38. 38. • Platform for creating interactive 3d applications! • Widely used! • 2.5 million registered developers
  39. 39. • High-quality rendering! • Deferred renderer! • Occlusion culling! • Realtime shadows! • Ambient occlusion! • Light mapping
  40. 40. Unity 3D Features • Programming! • C#, Javascript! • Profiler! • Physics! • NVidia PhysX! • Cloth! • Soft/rigid bodies! • Ragdolls! • Cars 3D formats ! FBX, Collada, 3DS, OBJ…! Audio engine! Networking! Terrain
  41. 41. Editor • Simple to use! • Create your own tools inside Unity
  42. 42. Asset store • Buy / sell assets! • 3D models! • Textures! • Shaders! • Scripts! • Editor plugins
  43. 43. Desktop Unity is a great tool but it doesn’t natively support VR systems easily.
  44. 44. • Build hardware To create and maintain presence, you, the author, need to ... • Build software • Build user experience Our goal is to minimize the time you spend on building the software so that you can focus on building the user experience.
  45. 45. In my mind, VR is not that difficult. Doesn’t have to be. A part is that the hardware is hard to use, but a bigger part is that the software is hard to use mostly by design.
  46. 46. In 2010 sent an e-mail to Roland Blach asking for all the SEARIS papers and studied them attentively Software Engineering and Architectures for Realtime Interactive Systems Working Group
  47. 47. Existing VR software • VRPN! • OpenTracker! • VRJuggler! • Virtools! • Worldviz! • …
  48. 48. VRPN In VRPN, you first have to edit a configuration file, understandable by computer scientists only. Then you have to understand how it works: callbacks, terminology is also a bit obscur (analog..) VRPN is only solving the devices management, and as we have seen, there are a lot of other issues when creating a VR app.
  49. 49. VR Juggler VR Juggler needs to understand a lot of concept: proxy device? derive of off OpenGL::App ? My main problem is that you have to bend your application to their design
  50. 50. VIRTOOLSVR PACK VRPack’s configuration file is even more complex than VRPN. Even working at Virtools, it took me quite a while to understand how it works ! Switching configuration files is also complicated.
  51. 51. Worldviz Vizard • Based on OpenSceneGraph! • Python Vizard is a really nice environment but still a bit too complex to use.
  52. 52. Worldviz Vizard For example this is how you would configure trackers and a Cave.
  53. 53. The need for better VR software tools • Hardware getting cheaper! • But engineers are not getting cheaper! • Need to be more efficient and spend less time creating apps! • Democratization! • VR developers are not all engineers anymore
  54. 54. Goals • Simplify VR! • Simplify creation of VR applications! • Simplify deployment of VR applications! • Create new VR tools! • Adapting existing 3D applications
  55. 55. What is ? • Generic VR plugin (C++ SDK) ! • Functionalities ○Manage input devices •3D trackers, keyboard, mouse, joystick ○Manage display •Viewports, cameras, stereoscopy, cluster ○High-level building-blocks ○Interactions ○Menu CaveLib - VRJuggler
  56. 56. Architecture MiddleVR (C++) Drivers OpenGL DirectX Network C# Wrapper GUI Unity Lua Wrapper Python Wrapper
  57. 57. GUI to easily configure any VR system quickly.
  58. 58. Tutorial:
  59. 59. Once you have exported your application, you can simply add it to the list of Simulations in MiddleVR’s gui. Then you can pick any configuration and simply click Run and your application will be dynamically reconfigured for this configuration
  60. 60.
  61. 61. MiddleVR Features
  62. 62. Supported Devices • Via VRPN : • A.R.T • Vicon • Optitrack • Wiimote ! • Polhemus Fastrak Tracker, 3Space Tracker, Patriot Trackers, • Ascension Flock of Bird • Intersense IS600, IS900 • 3dTech HiBall-3000 Wide Area Tracker, • Advanced Realtime Tracking Gmbh DTrack Client, • World Viz Precision Position Tracker PPT 1.2, • Natural Point Optitrack Rigid Body Toolkit, • Phasespace optical tracking OWL, • ... • Native drivers • Oculus Rift • Leap Motion • Kinect (Microsoft SDK) • PNI SpacePoint Fusion • GameTrak • Razer Hydra • SpaceMouse • Motion Analysis (beta)
  63. 63. Haptics (force feedback) Haption IPSI
  64. 64. VR Camera • MiddleVR computes all the cameras’ parameters automatically! • Correct perspective! • Scale 1
  65. 65. Stereoscopy • Active stereoscopy (OpenGL Quad-Buffer)! • Which Unity can’t natively handle! • Passive stereoscopy
  66. 66. Viewports management Viewports configuration
  67. 67. Simply configure your system Configure what is happening in the real life. Seeing things happening in real life changes a lot. Can easily check - axes swap, offsets - screens are correctly placed - if tracking is covering the area ! Allows you to experiment.
  68. 68. Navigations We provide standard navigations: - classic joystick navigation - elastic navigation - grab the world
  69. 69. Homography A nice new feature is the homography. This allows you to grab corners of your viewports and have them match the physical corners of your cave.
  70. 70. Cluster & Multi-pipe • Scenelock! • Swaplock! • Genlock! • Multi-pipe (soon) Regarding clustering, one unity player is running on each computer. SceneLock: synchronise the state of all input devices and position/orientation of some specific objects. SwapLock: implemented our own mechanism Genlock: done in HW
  71. 71. Supported displays • Virtually any HMD • VR-Wall, Powerwall • Workbench, Holobench • HoloStage • CAVEs • 3D TVs • zSpace • Igloo Vision • …
  72. 72. Goals • Simplify creation of VR applications! • Simplicity of adding VR into application! • Simplicity of API! • High-level tools! • Interactions: Selection, manipulation, navigation, menus, GUIs
  73. 73. Example of device management
  74. 74. Driver 1
  75. 75. Driver 2
  76. 76. Device management Replaced manual edition of cfg file by GUI Simply choose the driver from the list, don’t need to rewrite it.
  77. 77. Looks like english ! No need to understand what a sink is (opentracker), or forced to set callbacks.
  78. 78. MiddleVR Here is the minimum code you have to write to use MiddleVR. You don’t have to modify the structure of your application to integrate MiddleVR !
  79. 79. Class reference Most VR systems out there can be described with these highlighted simple, atomic, concepts
  80. 80. The value of Simplicity
  81. 81. The value of simplicity “Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away”! Antoine de St Exupery Simplicity is a huge principle for me. We don’t hesitate to remove code/feature that are covering way too much. EVEN IF ALREADY WRITTEN !! We’re a Startup, we don’t have resources, so we have to be simple No resources is a strength Lots of resources can actually be a curse
  82. 82. The value of simplicity • For each feature you add! • Test! • Document! • Support / maintain! • Each feature costs !! Each feature you add, you have to support it for the entire life of the product. Simplicity is harder to achieve, so cost more at first, but in the long run is beneficial
  83. 83. The value of simplicity • Huge value in simplicity !! • User side! • Faster to learn, less to remember! • Developer side! • Faster to code! • Easier maintain! • Less bugs! • Business side! • Need less coders! • More interesting features! • Easy download / setup > Quickly validate on user’s hardware !! • Easier support
  84. 84. API in simple english • Long discussions for naming of functions, classes, variables! • Consistent api! • SetNumButtons, SetButtonsNum, SetNbButtons > SetButtonsNb! • French not good at english! • Analog ? Digital ? Proxy ?! • Use natural terms depending on device! • Joystick->GetAxisValue()! • Joystick->IsButtonPressed()! • Keyboard->IsKeyPressed()! • Why “Register” ? Why not “Add” ?! • Why so complex, want to sound very clever ?! • AddDevice, AddCamera, …! • API in degrees, because more intuitive / simple to work with
  85. 85. Simplified API • Simple but complete API! • Like legos! • Rotate! • Node3D->Rotate( Quat )! • Node3D->Rotate( Euler )! • Node3D->Rotate( Axis, Angle)! • Quat( quat ), Quat( Euler ), Quat( Axis, Angle) Simplified API: only one way of doing things: less to learn, less to remember Everything you need is in the Quat class, so it’s reusable in any part of the code
  86. 86. The value of simplicity “Simplicity is the ultimate sophistication”! DaVinci Fight hard to create / maintain simplicity. Engineers come and implemented what was in their mind, anticipate cases that would rarely happen. We do code reviews. Often rewrite often 3 times before acceptable.
  87. 87. The value of simplicity • Design user interface for the 80%! • Don’t ruin the experience for the 20%! • Simple things should be simple! • Complex things can be complex! • Put in an “advanced menu”! • Don’t let beauty of design ruin user experience! • Like real life architects who want their design to be beautiful but totally unusable on a day to day! • Example of Lego! • Simple but complete Tell horror network socket story :) ! Current tools designed for geeks/engineers (examples vrpn/virtools vr)
  88. 88. Optimizations • Engineers have a strong tendency to over-optimise! • Complexifies code! • Code must be above all readable! • To avoid bugs! • To maintain maintenability! • If you want to optimise! • First profile your code! • Optimize what is *really* taking time! • not what you *think* takes time
  89. 89. Empathy “The ability to understand and share the feelings of another.”! ! Test with users • if user don’t understand, not add more doc,! • simplify !!
  90. 90. Hardware SDK are mostly too complicated ! Even Oculus !!!!! Less time to implement haptics !!!
  91. 91. Drivers • Why not just a few simple functions with sensible defaults ?! • Init()! • Update()! • GetValue()! • Destroy()! • And add more functions for more advanced HW manufacturers don’t know how to write an API
  92. 92. Documentation • User Guide! • Class references Of course the ideal tool would not need documentation, I honestly like to write documentation. It helps spot errors, inconsistencies. If it’s hard to document, the feature must probably must be simplified instead of writing a more sophisticated doc !!
  93. 93. Goals • Simplify deployment of VR applications! • GUI to configure VR systems! • Dynamic reconfiguration of application! • Ship with standard configurations! • Abstractions
  94. 94. Deployment • Abstractions! • Devices! • 3D Nodes (User)! • Interactions (soon)
  95. 95. Device management Driver Tracker 1 Application Driver Tracker 2 ** The problem is that each driver gives access to similar data in a different way so you need to have specific part of programs for similar drivers.
  96. 96. Device abstraction Virtual Device Application Driver Tracker 1 Driver Tracker 2 A virtual device only stores data. For tracker, position+orientation Drivers write data in the virtual device. Allow to change driver without application (PORTABILITY) --- Replay allows you to playback a recorded working session. It enables quality insurance because you can give the same input to your application and check that the output is always the same.
  97. 97. Device management Driver Virtual Device Application Simulation Replay Replay allows you to playback a recorded working session. It enables quality insurance because you can give the same input to your application and check that the output is always the same.
  98. 98. Display • Cameras & viewport created dynamically
  99. 99. User • Human-centric rather than techno-centric Rather than accessing devices, we could simply ask the position of the user’s head, or hand ! This allows to write scenarios like : What does the user see ? Is the hand close to ... ?
  100. 100. Virtual Device 1 Hand (3D Node) Application Virtual Device 2 Inverse Kinematics User abstraction
  101. 101. Interactions • MiddleVR includes high-level interactions! • Navigation! • Selection! • Manipulation! • … More to come! • Future: Interactions abstractions! • Dynamically change interactions based on hardware
  102. 102. Interactions abstraction? Grab-air Navigation Application Point Steering Selection Manipulation Raycasting Go-go Hand Homer Hand WIM
  103. 103. • How to create portable interactions ?! • Navigations can be portable! • Except collisions! • Generalise what is generalisable! • + Custom part for engine dependent! • Collisions! • Highlights! • Gizmos Interactions
  104. 104. Menus • Ability to create menus and GUIs in HTML5 (soon)
  105. 105. Create immersive HTML5 GUI HTML 5 gui in VR
  106. 106. C++ Reflexion • Also for configuration file! • Cluster synchronisation! • Accessing drivers specific information Property is simple couple of getter/setter. In header write naturally Set/GetParent ! One line to add property in source code.
  107. 107. Working with multiple 3D engines • Different coordinate systems! • vrCoordinateSystem( Front, Up, Right )! • +/- X,Y,Z! • OpenGL:! • Front= -Z! • Up = +Y! • Right = +X! • DirectX:! • Front=+Z! • Up=+Y! • Right=+X
  108. 108. Working with multiple 3D engines • C++! • Compiling! • STL, Boost! • Strings (Wide strings)! • Link issues! • Different compilers (VC2005/2008/2010/2012)! • Different languages! • C wrapper! • Lua wrapper
  109. 109. ! • Simplicity! • Creation! • Deployment! •Adaptability! •Hardware! • Software
  110. 110. Our future • Games/apps development:
  111. 111. Conclusion Talked about the philosophy and architecture You see how deeply the philosophy impacts the architecture Simplicity at its core which has a huge value for everybody, but has to fight hard for it.
  112. 112. ! Thank you ! Questions ? ! @Cb_VRGeek @imin_vr