Download to read offline
More Related Content
Similar to Ai lecture about VR technology discuss.pptx
Ai lecture about VR technology discuss.pptx
- 1. LECTURE 3: VR TECHNOLOGY MuhammadAsim Khan
- 2. • Presence • Perceptionand VR • Human Perception • Sight, hearing, touch, smell, taste • VR Technology • Visual display Recap – Lecture 2
- 3. Presence . . “The subjectiveexperience of being in one place or environment even when physically situated in another” Witmer, B. G., & Singer, M. J. (1998). Measuring presence in virtual environments: A presence questionnaire. Presence: Teleoperators and virtual environments, 7(3), 225-240.
- 4. How do WePerceive Reality? • We understand the world through our senses: • Sight, Hearing, Touch, Taste, Smell (and others..) • Two basic processes: • Sensation – Gathering information • Perception – Interpreting information
- 5.
- 6. Creating the Illusionof Reality • Fooling human perception by using technology to generate artificial sensations • Computer generated sights, sounds, smell, etc
- 7. Reality vs. Virtual Reality •In a VR system there are input and output devices between human perception and action
- 8. Using Technology toStimulate Senses • Simulate output • E.g. simulate real scene • Map output to devices • Graphics to HMD • Use devices to stimulate the senses • HMD stimulates eyes Example: Visual Simulation Visual Simulation 3D Graphics HMD Vision System Brain Human-Machine Interface
- 9. Creating an Immersive Experience •HeadMounted Display •Immerse the eyes •Projection/Large Screen •Immerse the head/body •Future Technologies •Neural implants •Contact lens displays, etc
- 10. HMD Basic Principles • Usedisplay with optics to create illusion of virtual screen
- 11. Key Properties of HMDs •Lens • Focal length, Field of View • Occularity, Interpupillary distance • Eye relief, Eye box • Display • Resolution, contrast • Power, brightness • Refresh rate • Ergonomics • Size, weight • Wearability
- 12.
- 13.
- 14. Tracking in VR Head Tracking HandTracking • Need for Tracking • User turns their head and the VR graphics scene changes • User wants to walking through a virtual scene • User reaches out and grab a virtual object • The user wants to use a real prop in VR • All of these require technology to track the user or object • Continuously provide information about position and orientation
- 15. • Degree ofFreedom = independent movement about an axis • 3 DoF Orientation = roll, pitch, yaw (rotation about x, y, or z axis) • 3 DoF Translation = movement along x,y,z axis • Different requirements • User turns their head in VR -> needs 3 DoF orientation tracker • Moving in VR -> needs a 6 DoF tracker (r,p,y) and (x, y, z) Degrees of Freedom
- 16. Tracking and Renderingin VR Tracking fits into the graphics pipeline for VR
- 17. Tracking Technologies Active(device sends out signal) • Mechanical, Magnetic, Ultrasonic • GPS, Wifi, cell location Passive (device senses world) • Inertial sensors (compass, accelerometer, gyro) • Computer Vision • Marker based, Natural feature tracking Hybrid Tracking • Combined sensors (eg Vision + Inertial)
- 18.
- 19. Mechanical Tracker (Active) •Idea: mechanicalarms with joint sensors Microscribe Sutherland • ++: high accuracy, haptic feedback •-- : cumbersome, expensive
- 20. Magnetic Tracker (Active) • Idea:difference between a magnetic transmitter and a receiver • ++: 6DOF, robust • -- : wired, sensible to metal, noisy, expensive • -- : error increases with distance Flock of Birds (Ascension)
- 21. Example: Razer Hydra • Developedby Sixense • Magnetic source + 2 wired controllers • Short range (1-2 m) • Precision of 1mm and 1o • $600 USD
- 22.
- 23. Inertial Tracker (Passive) • Idea:measuring linear and angular orientation rates (accelerometer/gyroscope) • ++: no transmitter, cheap, small, high frequency, wireless • -- : drift, hysteris only 3DOF IS300 (Intersense) Wii Remote
- 24. Optical Tracker (Passive) • Idea:Image Processing and Computer Vision • Specialized • Infrared, Retro-Reflective, Stereoscopic • Monocular Based Vision Tracking ART Hi-Ball
- 25.
- 26. Example: Vive Lighthouse Tracking •Outside-in tracking system • 2 base stations • Each with 2 laser scanners, LED array • Headworn/handheld sensors • 37 photo-sensors in HMD, 17 in hand • Additional IMU sensors (500 Hz) • Performance • Tracking server fuses sensor samples • Sampling rate 250 Hz, 4 ms latency • See http://doc-ok.org/?p=1478
- 27. Lighthouse Components Base station - IRLED array - 2 x scanned lasers Head Mounted Display - 37 photo sensors - 9 axis IMU
- 28.
- 29.
- 30. Example: Oculus Quest • Insideout tracking • Four cameras on corner of display • Searching for visual features • On setup creates map of room
- 31.
- 32. Occipital Bridge Engine/Structure Core •Inside out tracking • Uses structured light • Better than room scale tracking • Integrated into bridge HMD • https://structure.io/
- 33.
- 34. Tracking Coordinate Frames • Therecan be several coordinate frames to consider • Head pose with respect to real world • Coordinate fame of tracking system wrt HMD • Position of hand in coordinate frame of hand tracker
- 35. Example: Finding yourhand in VR • Using Lighthouse and LeapMotion • Multiple Coordinate Frames • LeapMotion tracks hand in LeapMotion coordinate frame (HLM) • LeapMotion is fixed in HMD coordinate frame (LMHMD) • HMD is tracked in VR coordinate frame (HMDVR) (using Lighthouse) • Where is your hand in VR coordinate frame? • Combine transformations in each coordinate frame • HVR = HLM x LMHMD x HMDVR
- 36.
- 37. Haptic Feedback • Greatly improvesrealism • Hands and wrist are most important • High density of touch receptors • Two kinds of feedback: • Touch Feedback • information on texture, temperature, etc. • Does not resist user contact • Force Feedback • information on weight, and inertia. • Actively resists contact motion
- 38. Active Haptics • Actively resistsmotion • Key properties • Force resistance • Frequency Response • Degrees of Freedom • Latency
- 39. Example: Phantom Omni • Combinedstylus input/haptic output • 6 DOF haptic feedback
- 40.
- 41. Haptic Glove • Many examplesof haptic gloves • Typically use mechanical device to provide haptic feedback
- 42. Passive Haptics • Not controlledby system • Use real props (Styrofoam for walls) • Pros • Cheap • Large scale • Accurate • Cons • Not dynamic • Limited use
- 43.
- 44. Passive Haptic Paddle • Usingphysical props to provide haptic feedback • http://www.cs.wpi.edu/~gogo/hive/
- 45. Tactile Feedback Interfaces •Goal: Stimulate skin tactile receptors • Using different technologies • Air bellows • Jets • Actuators (commercial) • Micropin arrays • Electrical (research) • Neuromuscular stimulations (research)
- 46. Vibrotactile Cueing Devices • Vibrotactilefeedback has been incorporated into many devices • Can we use this technology to provide scalable, wearable touch cues?
- 47.
- 48.
- 49. Teslasuit • Full bodyhaptic feedback - https://teslasuit.io/ • Electrical muscle stimulation
- 50.
- 51.
- 52. Audio Displays • Spatialization vs.Localization • Spatialization is the processing of sound signals to make them emanate from a point in space • This is a technical topic • Localization is the ability of people to identify the source position of a sound • This is a human topic, i.e., some people are better at it than others.
- 53. Audio Display Properties Presentation Properties •Number of channels • Sound stage • Localization • Masking • Amplification Logistical Properties • Noise pollution • User mobility • Interface with tracking • Integration • Portability • Throughput • Safety • Cost
- 54. Audio Displays: Head- worn EarBuds On Ear Open Back Closed Bone Conduction
- 55. Head-Related Transfer Functions (HRTFs) •A set of functions that model how sound from a source at a known location reaches the eardrum
- 56. Measuring HRTFs • Putting microphonesin Manikin or human ears • Playing sound from fixed positions • Record response
- 57. Capturing 3D Audiofor Playback • Binaural recording • 3D Sound recording, from microphones in simulated ears • Hear some examples (use headphones) • http://binauralenthusiast.com/examples/
- 58. OSSIC 3D Audio Headphones •https://www.ossic.com/3d-audio/
- 59.
- 60.
- 61. VR Input Devices • Physicaldevices that convey information into the application and support interaction in the Virtual Environment
- 62. Mapping Between Inputand Output Input Output
- 63. Motivatio n • Mouse andkeyboard are good for desktop UI tasks • Text entry, selection, drag and drop, scrolling, rubber banding, … • 2D mouse for 2D windows • What devices are best for 3D input in VR? • Use multiple 2D input devices? • Use new types of devices? vs.
- 64. Input Device Characteristics • Sizeand shape, encumbrance • Degrees of Freedom • Integrated (mouse) vs. separable (Etch-a-sketch) • Direct vs. indirect manipulation • Relative vs. Absolute input • Relative: measure difference between current and last input (mouse) • Absolute: measure input relative to a constant point of reference (tablet) • Rate control vs. position control • Isometric vs. Isotonic • Isometric: measure pressure or force with no actual movement • Isotonic: measure deflection from a center point (e.g. mouse)
- 65. Hand Input Devices • Devicesthat integrate hand input into VR • World-Grounded input devices • Devices fixed in real world (e.g. joystick) • Non-Tracked handheld controllers • Devices held in hand, but not tracked in 3D (e.g. xbox controller) • Tracked handheld controllers • Physical device with 6 DOF tracking inside (e.g. Vive controllers) • Hand-Worn Devices • Gloves, EMG bands, rings, or devices worn on hand/arm • Bare Hand Input • Using technology to recognize natural hand input
- 66. World Grounded Devices Disney AladdinMagic Carpet VR Ride • Devices constrained or fixed in real world • Not ideal for VR • Constrains user motion • Good for VR vehicle metaphor • Used in location based entertainment (e.g. Disney Aladdin ride)
- 67. Non-Tracked Handheld Controllers • Devicesheld in hand • Buttons, joysticks, game controllers, etc. • Traditional video game controllers • Xbox controller
- 68. Tracked Handheld Controllers(3 or 6 DoF) HTC Vive Controllers Oculus Touch Controllers • Handheld controller with 6 DOF tracking • Combines button/joystick input plus tracking • One of the best options for VR applications • Physical prop enhancing VR presence • Providing proprioceptive, passive haptic touch cues • Direct mapping to real hand motion
- 69. Example: Sixense STEM • Wirelessmotion tracking + button input • Electromagnetic tracking, 8 foot range, 5 tracked receivers • http://sixense.com/wireless
- 70.
- 71. Example: WMR Handheld Controllers •Windows Mixed Reality Controllers • Left and right hand • Combine computer vision + IMU tracking • Track both in and out of view • Button input, Vibration feedback
- 72.
- 73. Cubic Mouse • Plastic box •Polhemus Fastrack inside (magnetic 6 DOF tracking) • 3 translating rods, 6 buttons • Two handed interface • Supports object rotation, zooming, cutting plane, etc. Fröhlich, B., & Plate, J. (2000). The cubic mouse: a new device for three-dimensional input. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems (pp. 526- 531). ACM.
- 74.
- 75. Hand Worn Devices • Devicesworn on hands/arms • Glove, EMG sensors, rings, etc. • Advantages • Natural input with potentially rich gesture interaction • Hands can be held in comfortable positions – no line of sight issues • Hands and fingers can fully interact with real objects
- 76.
- 77. Data Gloves • Bend sensinggloves • Passive input device • Detecting hand posture and gestures • Continuous raw data from bend sensors • Fiber optic, resistive ink, strain-gauge • Large DOF output, natural hand output • Pinch gloves • Conductive material at fingertips • Determine if fingertips touching • Used for discrete input • Object selection, mode switching, etc.
- 78. How Pinch Gloves Work •Contact between conductive fabric completes circuit • Each finger receives voltage in turn (T3 – T7) • Look for output voltage at different times
- 79. Example: Cyberglove • Invented tosupport sign language • Technology • Thin electrical strain gauges over fingers • Bending sensors changes resistence • 18-22 sensors per glove, 120 Hz samples • Sensor resolution 0.5o • Very expensive • >$10,000/glove • http://www.cyb erglovesystem s.com
- 80. How CyberGlove Works • Straingauge at joints • Connected to A/D converter
- 81.
- 82. StretchSense • Wearable motioncapture sensors • Capacitive sensors • Measure stretch, pressure, bend, shear • Many applications • Garments, gloves, etc. • http://stretchsense.com/
- 83.
- 84. Comparison of Glove Performance FromBurdea, Virtual Reality Technology, 2003
- 85. Bare Hands • Using computervision to track bare hand input • Creates compelling sense of Presence, natural interaction • Challenges need to be solved • Not having sense of touch • Line of sight required to sensor • Fatigue from holding hands in front of sensor
- 86. Leap Motion • IR basedsensor for hand tracking ($50 USD) • HMD + Leap Motion = Hand input in VR • Technology • 3 IR LEDS and 2 wide angle cameras • The LEDS generate patternless IR light • IR reflections picked up by cameras • Software performs hand tracking • Performance • 1m range, 0.7 mm accuracy, 200Hz • https://www.leapmotion.com/
- 87.
- 88. Non-Hand Input Devices • Capturinginput from other parts of the body • Head Tracking • Use head motion for input • Eye Tracking • Largely unexplored for VR • Microphones • Audio input, speech • Full-Body tracking • Motion capture, body movement
- 89. Eye Tracking • Technology • ShineIR light into eye and look for reflections • Advantages • Provides natural hands-free input • Gaze provides cues as to user attention • Can be combined with other input technologies
- 90. Example: FOVE VR Headset •Eye tracker integrated into VR HMD • Gaze driven user interface, foveated rendering • https://www.youtube.com/watch?v=8dwdzPaqsDY
- 91. Pupil Labs VIVE/OculusAdd- ons • Adds eye-tracking to HTC Vive/Oculus Rift HMDs • Mono or stereo eye-tracking • 120 Hz eye tracking, gaze accuracy of 0.6° with precision of 0.08° • Open source software for eye-tracking • https://pupil-labs.com/pupil/
- 92. HTC Vive Pro Eye •HTC Vive Pro with integrated eye-tracking • Tobii systems eye-tracker • Easy calibration and set-up • Auto-calibration software compensates for HMD motion
- 93.
- 94. Full Body Tracking • Addingfull-body input into VR • Creates illusion of self-embodiment • Significantly enhances sense of Presence • Technologies • Motion capture suit, camera based systems • Can track large number of significant feature points
- 95. Camera Based Motion Capture •Use multiple cameras • Reflective markers on body • Eg – Opitrack (www.optitrack.com) • 120 – 360 fps, < 10ms latency, < 1mm accuracy
- 96.
- 97. Wearable Motion Capture: PrioVR •Wearable motion capture system • 8 – 17 inertial sensors + wireless data transmission • 30 – 40m range, 7.5 ms latency, 0.09o precision • Supports full range of motion, no occlusion • www.priovr.com
- 98.
- 99. Pedestrian Devices • Pedestrian inputin VR • Walking/running in VR • Virtuix Omni • Special shoes • http://www.virtuix.com • Cyberith Virtualizer • Socks + slippery surface • http://cyberith.com
- 100.
- 101. Virtusphere • Fully immersivesphere • Support walking, running in VR • Person inside trackball • http://www.virtusphere.com
- 102.
- 103. Omnidirectional Treadmills • Infinadeck • 2axis treadmill, flexible material • Tracks user to keep them in centre • Limitless walking input in VR • www.infinadeck.com
- 104.
- 105. Comparison Between Devices From Jerald(2015) Comparing between hand and non-hand input
- 106. Input Device Taxonomies • Helpsto determine: • Which devices can be used for each other • What devices to use for particular tasks • Many different approaches • Separate the input device from interaction technique (Foley 1974) • Mapping basic interactive tasks to devices (Foley 1984) • Basic tasks – select, position, orient, etc. • Devices – mouse, joystick, touch panel, etc. • Consider Degrees of Freedom and properties sensed (Buxton 1983) • motion, position, pressure • Distinguish bet. absolute/relative input, individual axes (Mackinlay 1990) • separate translation, rotation axes instead of using DOF
- 107. Foley and WallaceTaxonomy (1974) Separate device from interaction technique
- 108. Buxton Input DeviceTaxonomy (Buxton 1983) • Classified according to degrees of freedom and property sensed • M = devise uses an intermediary between hand and sensing system • T = touch sensitive