More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (17)
Similar to Sneak Peek: Designing Tracked Objects for Steam VR
Similar to Sneak Peek: Designing Tracked Objects for Steam VR (20)
Recently uploaded
Recently uploaded (20)
Sneak Peek: Designing Tracked Objects for Steam VR
- 1. 1 | S X S W 2 0 1 7 DESIGNING TRACKED DOUG BRUEY | MICHAEL CIUFFO OBJECTS FOR
- 2. 2 | S X S W 2 0 1 7 ■ Product development for the world’s leading companies ■ 150+ multi-disciplinary engineers ■ Cambridge Consultants • Global parent company • 500+ Engineers and innovators SYNAPSE SEATTLE SAN FRANCISCO ORLANDO BOSTON NEW DELHI CAMBRIDGE HONG KONG SINGAPORE TOKYO
- 3. 3 | S X S W 2 0 1 7 Valve is an entertainment software and technology company founded in 1996. In addition to creating several of the world's most award-winning games, Valve is also a developer of leading-edge technologies including the Source® game engine and Steam®, the premier online gaming platform.
- 4. OCT 2014 Synapse started working with Valve on their vr system HISTORY 4 | S X S W 2 0 1 7 JUNE 2015 Developer release of the base stations APRIL 2016 HTC Vive released AUG 2016 Valve announces royalty free licensing of SteamVR™ tracking SEPT 2016 SteamVR™ tracking reference design released 2017 Develop firmware and assist partners integrating SteamVR™ tracking into their products SEPT 2016 - TODAY SteamVR™ tracking training courses at synapse
- 5. • What is SteamVR™ Tracking? • The problem it solves in VR and AR applications • How it works to track the pose of an object • How to make objects that track well using the technology • How to get started using the technology • Answer as many questions and possible WE’LL COVER 5 | S X S W 2 0 1 7
- 6. YAW ORIENTATION 6 | S X S W 2 0 1 7 ROLL PITCH POSE TRACKING
- 7. Z-Axis z 7 | S X S W 2 0 1 7 X-Axis Y-Axis x POINT (x,y,z) y POSITION POSE TRACKING
- 8. STATE OF THE ART 8 | S X S W 2 0 1 7 IMUs (Inertial Measurement Units) ■ Gyroscopes and accelerometers track orientation External Cameras (Pose Tracking) Internal Cameras (Inside-out Pose Tracking) ■ Popular in AR ■ Depth Perception & Room Mapping SteamVR™ Tracking (Pose Tracking)
- 9. STEAMVR™ TRACKING 9 | S X S W 2 0 1 7 ■ Room scale ■ Security/privacy ■ Low latency & bandwidth ■ Absolute position ■ Eliminates challenges of camera-based tracking ■ Track to submillimeter accuracy
- 10. STEAMVR™ TRACKING SYSTEM OVERVIEW 1 0 | S X S W 2 0 1 7
- 11. BASE STATION 1 1 | S X S W 2 0 1 7
- 12. TRACKED OBJECTS 1 2 | S X S W 2 0 1 7
- 13. ■ Reduces shadows ■ Requires synchronization MULTIPLE BASE STATIONS 1 3 | S X S W 2 0 1 7
- 14. REFERENCE SIGNALING 1 4 | S X S W 2 0 1 7
- 15. ■ Optical receivers detect the reference signals from the base station ■ Photodiode converts IR light to current ■ Transimpedance amplifier converts current to voltage ■ Envelope detector removes the modulation frequency ■ FPGA connects to all sensors to timestamp the arrival of reference signals OPTICAL RECEIVERS 1 5 | S X S W 2 0 1 7
- 16. TRIANGULATION 1 6 | S X S W 2 0 1 7
- 17. ■ The angle to all sensors creates a solution set for the position of the object ■ SteamVR™ matches the measured angles to the known object geometry ■ There must only be one solution to the problem! SOLVING THE SYSTEM 1 7 | S X S W 2 0 1 7
- 18. SOLVING THE SYSTEM 1 8 | S X S W 2 0 1 7
- 19. DEMO 1 9 | S X S W 2 0 1 7 ONE TO FOUR SENSORS
- 20. ■ Reference signal jitter • Motor jitter • Timestamp quantization • Laser modulation ■ Minimized by design • Base station design • Object design ■ Sources of error place requirements on object design SOURCES OF ERROR 2 0 | S X S W 2 0 1 7
- 21. ■ Baseline increases the time between laser hits ■ Sources of error are angular ■ The limit of the system is a minimum detectable angle ■ More baseline accommodates the same angle at a greater distance ■ Need baseline to overcome translation error CONSEQUENCES OF ERROR | TRANSLATION ERROR 2 1 | S X S W 2 0 1 7
- 22. ■ Baseline increases the time between laser hits ■ Sources of error are angular ■ The limit of the system is a minimum detectable angle ■ More baseline accommodates the same angle at a greater distance ■ Need baseline to overcome translation error OVERCOMING TRANSLATION ERROR | SUFFICIENT BASELINE 2 2 | S X S W 2 0 1 7
- 23. ■ Rotation orthogonal to a plane yields significant displacement ■ Rotation in the plane yields much smaller displacement per degree rotation ■ Error dominates the small change in distance CONSEQUENCES OF ERROR | ROTATION ERROR 2 3 | S X S W 2 0 1 7
- 24. ■ Baseline amplifies the effect of rotation ■ Pose tracking detects • Roll • Pitch • Yaw ■ Detecting all three means amplifying all three ■ Need baseline in X, Y and Z axes to overcome rotation error OVERCOMING ROTATION ERROR | BASELINE IN 3 AXES 2 4 | S X S W 2 0 1 7
- 25. ■ Four visible sensors (minimum) ■ Maximize the distance between sensors ■ Maximize baseline in three axes TRACKING PERFORMANCE DRIVERS 2 5 | S X S W 2 0 1 7 d x z y2 3 1 4
- 26. SENSOR FIELD OF VIEW 2 6 | S X S W 2 0 1 7
- 27. DEMO 2 7 | S X S W 2 0 1 7 VIEWING ANGLE
- 28. ■ Objects that track well have geometries designed for optimal sensor placement! ■ Simulation lets us verify performance early in the process. SENSOR PLACEMENT CRITERIA 2 8 | S X S W 2 0 1 7
- 29. X, Y, Z coordinates, facing direction 2 9 | S X S W 2 0 1 7 ±60° from normal Shadows cast by the sensor object Shadows cast by nearby objects
- 30. 3 0 | S X S W 2 0 1 7
- 31. DEMO 3 1 | S X S W 2 0 1 7 HMD DESIGNER VIEWER
- 32. NUMBER OF VISIBLE SENSORS 3 2 | S X S W 2 0 1 7
- 33. INITIAL POSE POSSIBLE 3 3 | S X S W 2 0 1 7
- 34. TRANSLATION ERROR 3 4 | S X S W 2 0 1 7
- 35. ROTATION ERROR 3 5 | S X S W 2 0 1 7
- 36. REFINEMENTS | NUMBER OF VISIBLE SENSORS 3 6 | S X S W 2 0 1 7
- 37. REFINEMENTS | POSE ROTATION ERROR 3 7 | S X S W 2 0 1 7
- 38. REFINEMENTS | POSE TRANSLATION ERROR 3 8 | S X S W 2 0 1 7
- 39. DEMO 3 9 | S X S W 2 0 1 7 HMD DESIGN REFINEMENTS
- 40. ■ Consumer products need to look and feel great ■ Products also need to perform, especially VR products INDUSTRIAL DESIGN CHALLENGES MINIATURIZATION 4 0 | S X S W 2 0 1 7 FLAT SURFACES RIGHT ANGLES LOW PROFILE CURVED SURFACES Translation error? Translation and rotation error?Sensor viewing angle?Rotation error? Sensor covering?
- 41. MECHANICAL DESIGN CHALLENGES 4 1 | S X S W 2 0 1 7 ■ Facets on shapes improve performance • How many slides in the mold? ■ Sensors facing in all directions • Multiple parts to facilitate ejection
- 42. ■ Sensor interconnect ■ FPC design challenge ELECTRICAL DESIGN CHALLENGES 4 2 | S X S W 2 0 1 7
- 43. RECOMMENDATIONS 4 3 | S X S W 2 0 1 7 ■ Collaborate between engineering and industrial design early in the process ■ Teach industrial designers and product visionaries about sensor placement ■ Use the constraints of sensor placement as a seed for creating unique, compelling designs ■ Reduce risk early • Use the simulation tools in the HDK to validate design choices • Prototype shapes using rapid prototyping techniques and evaluation hardware • See your object track in SteamVR™ before investing in tooling
- 44. ■ Laser based pose tracking system ■ Solves the problem in VR and AR applications ■ Tracks the pose of an object through…. ■ Design objects that track well using the technology • Four visible sensors (minimum) • Maximize the distance between sensors • Maximize baseline in three axes ■ Use the software to simulate & refine design ■ Collaboration between ID, ME, EE, & SW is key to successful design SUMMARY | STEAMVR™ TRACKING 4 4 | S X S W 2 0 1 7
- 45. NEXT STEPS partner.steamgames.com/vrtracking STEAMVR™ TRACKING LICENSE 4 5 | S X S W 2 0 1 7 STEAMVR™ TRACKING TRAINING synapse.com/steamvr HARDWARE DEVELOPMENT KIT triadsemi.com/product/steamvr-tracking-hdk
- 46. ■ VR Accessories ■ AR Products ■ Tracking Systems WHAT ARE YOU GOING TO MAKE? 4 6 | S X S W 2 0 1 7
- 47. The information contained herein is confidential and proprietary and may not be reproduced or distributed without the consent of Synapse Product Development QUESTIONS? 4 7 | S X S W 2 0 1 7
Editor's Notes
- Doug and Ciuffo Electrical Engineers at Synapse Involved in SteamVR development since May 2015 Teach the SteamVR Tracking course at Synapse Fun facts? Ciuffo - Vive early adopter, Cloudlands VR Minigolf expert Doug - Still trying to master Joust
- Synapse locations in Seattle (HQ), San Francisco, Orlando, and Hong Kong 150+ Engineers and innovators Our parent company and close partners in product development; Cambridge Consultants has locations in Boston, Cambridge, New Delhi, Singapore, and Tokyo 500+ Engineers and innovators
- Valve was a software company and Synapse helped them enter the hardware space. In addition to creating several of the world's most award-winning games, Valve is also a developer of leading-edge technologies including the Source® game engine and Steam®, the premier online gaming platform.
- Valve/Synapse, SteamVR Tracking, the HTC Vive, and you (the licensee)... Outline the story Synapse started working with Valve on their VR system in Oct, 2014 Collaborated on base stations leading up to the developer release in June, 2015 HTC was already integrating the tech into the HTC Vive which was released in April, 2016 Valve announced royalty free licensing of SteamVR Tracking in Aug, 2016 Developed the SteamVR Tracking reference design released in Sept, 2016 Organized and hosted SteamVR Tracking training courses Sept, 2016 - Today And actually, the material we are presenting today is a compressed version of the introduction day of that 2 ½ day course Continue to develop firmware and assist partners integrating SteamVR tracking into their products
- Provide a technical overview of SteamVR Tracking What is SteamVR Tracking? The problem it solves in VR and AR applications How it works to track the pose of an object How to make objects that track well using the technology How to get started using the technology Answer as many questions and possible
- Pose Tracking, What is it? Orientation (Roll, Pitch, and Yaw) Lets you look around
- Position (X, Y, and Z) Lets you move your head to the side, crouch down and look under a table, and walk around a room
- VR and AR systems employ a variety of sensors and algorithms to achieve pose tracking IMUs (Inertial Measurement Units), gyroscopes and accelerometers track orientation Cannot track location. Stuck in a seat. Rollercoaster looking down example. Samsung GearVR, or Google Daydream Internal Cameras (‘inside-out’ pose tracking), popular in AR Not good enough for VR. Too much lag. Not accurate enough. Microsoft HoloLens External Cameras (pose tracking) Cameras aren’t as accurate. Limited field of view. PlayStation VR or Oculus Rift SteamVR™ Tracking (pose tracking) Sub-mm tracking space Large tracking space. HTC Vive
- Room scale- That’s actually our VR set up in our Seattle office. Julie is able to move around the large room Anecdote - People sometimes don’t immediately realize they can walk around and interact with objects, but they can. Security/privacy Cameras have extra information not necessary for VR. Low latency & Low bandwidth Less information to capture makes it easy to process more quickly and easier to move around the room. Absolute position Unlike IMU solutions, positions are locked to 3D space. Eliminates depth of field and resolution challenges of camera-based tracking Resolution Depth of field Track the pose of an object in a 5m x 5m x 5m volume with submillimeter accuracy.
- What is the latency? “the time it takes sound from my voice to hit audience” Explain Frames/second MENTION WE’RE PASSING OUT BASE STATIONS
- Sync Blinker (IR LEDs) X and Y (IR Lasers/Motors) Synchronization Wired Optical
- SteamVR Tracking Optical sensors- Dimples on the Vive Tracking core electronics
- Two base stations reduce shadows Requires synchronization Synchronization Wired 60 Hz clock from master Balanced wired connection Original method Optical Sync blink Detected by optical receiver Better user experience Maybe mention multi-camera systems like motion capture systems (common misconception) Mention that base station we hand out has no optical sync
- Ciuffo slide. Grab a flashlight and the laser beam splitter. Sync blinks and laser hits are timestamped by the tracked object The difference between the sync blink and laser hit gives the angle End result: Object is somewhere along a ray.
- IR light is modulated…ambient light Optical receivers detect the reference signals from the base station Photodiode converts IR light to current Transimpedance amplifier converts current to voltage Envelope detector remove the modulation frequency FPGA connects to all sensors to timestamp the arrival of reference signals
- The SteamVR software is doing this math in the background
- Relative locations of sensors are fixed on the device. The angle from the base station to each sensor is known SteamVR™ matches the measured angles to the known object geometry This example has some ambiguity. Will result in bad performance.
- Adding another sensor here makes only one position/orientation the correct one.
- Blown up Single sensor X-Y is good No Z No rotation Two sensors More rotation info (left to right) Ambiguity of distance and rotation Must have another sensors Three sensors In line No more information than 2 sensors First rule of sensor placement “avoid collinear sensors!” In plane Now can tell between rotation and translation Still don’t know which way we’re rotating Four sensors In plane No better than 3 sensors. Still don’t know which way we’re rotating Out of plane Finally, have absolute pose AFTER DEMO: Rearrange object for small baseline example.
- 0.8 seconds over the course of a day. Toggle Demo: Noise
- Translation Error As distance increases Tangential velocity increases Time between sensors decreases Error begins to dominate Limits the maximum radius from the base station How could we reduce this error?
- Baseline increases the time between laser hits Sources of error are angular The limit of the system is a minimum detectable angle More baseline accommodates the same angle at a greater distance Need baseline to overcome translation error Toggle Demo: Sufficient Baseline
- Rotation Error Rotation orthogonal to a plane yields significant displacement Rotation in the plane yields much smaller displacement per degree rotation Error dominates the small change in distance How could we reduce this error?
- Baseline amplifies the effect of rotation Pose tracking detects Roll Pitch Yaw Detecting all three means amplifying all three Need baseline in X, Y and Z axes to overcome rotation error Demo: Make sure to rotate normal to plane before rotating in plane
- Viewing Angle with Large object Allude to need for sensors to be at angles: Tracking doesn’t work from behind. Tracking doesn’t really work from the side either Need to hand off to other sensors. Mention self-occlusion Doug spitballs while toggle over to HTC Vive controller Talk about the wings- probably added to create that sufficient baseline in a 3rd axes Toggle back to slideshow Launch HMD-Designer viewer project of X-frame.
- Objects that track well have geometries designed for optimal sensor placement! Simulation lets us verify performance early in the process.
- Sensor positions and normals Position = X, Y, Z coordinates Normal = facing direction Sensor field of view ±60° from normal Self occlusion Shadows cast by the sensor object Obstacle occlusion Shadows cast by nearby objects Model hands, heads, handles, accessories, etc.
- Number or Visible Sensors Rotation Error Translation Error Initial Pose Possible? - Explain
- Simple demo object: # of visible sensors Rotation error Translation error Initial pose possible Bad HMD design Show sensors & new shape & complexity (don’t go into each for the simulation outputs, Doug goes through those next)
- Initial indicator of placement quality
- Object boots or not from a given pose
- MENTION WE’RE PASSING OUT REFERENCE HARDWARE
- Refined Object: Mention it is the actual reference hardware, and that we’re passing them out
- Consumer products need to look and feel great Products also need to perform, especially VR products Appealing design features: Miniaturization: What about translation error? Flat surfaces: What about rotation error? Right angles: What about sensor viewing angle? Low profile: What about translation and rotation error? Curved surfaces: What about sensor covering?
- Facets on shapes improve performance How many slides in the mold? Sensors facing in all directions Multiple parts to facilitate ejection
- Sensor interconnect 20 - 32 sensors Distributed over the surface of an object Four wires to each sensor All sensors at different angles FPC design challenge Circuit size Panel efficiency Interconnect density MENTION: I see a lot of nervous faces in the room. We do this all the time, and it’s not that difficult if you start early knowing the challenges..
- Collaborate between engineering and industrial design early in the process Teach industrial designers and product visionaries about sensor placement Use the constraints of sensor placement as a seed for creating unique, compelling designs Reduce risk early Use the simulation tools in the HDK to validate design choices Prototype shapes using rapid prototyping techniques and evaluation hardware See your object track in SteamVR before investing in tooling
- Laser based pose tracking system Solves the problem in VR and AR applications Tracks the pose of an object through…. Design objects that track well using the technology Four visible sensors (minimum) Maximize the distance between sensors Maximize baseline in three axes Use the software to simulate & refine design Collaboration between ID, ME, EE, & SW is key to successful design
- Become a SteamVR Tracking Licensee Just a couple weeks ago, Valve made the HDK(hardware development kit) available, so now you can access the …. Also, We’ll still be offering classes throughout the year at Synapse, a 2.5 day class that extends and covers Generating and simulating sensor placement Writing object config files You’d walk away with the reference hardware (get the dev kit) Testing and calibrating objects Integrating objects with SteamVR Troubleshooting tracking problems Hands on experience Synapse is helping half a dozen companies customize hardware for their needs
- Different tracking systems out there, there’s no one system that is right for every scenario If you are looking for room scale, submillimeter, highly precise pose tracking system for the most immersive experience possible, that’s where Steam VR shines HTC has done it, LG just announced they’re doing it, there’s a hundred people who have come who our class who have done it And valve has supplied the software So you can do it too Automotive, cinema, aviation, AR