Output Devices:Graphics, 3-D Sound, Haptics and olfactory
Output Devices The human senses need specialized interfaces Graphics displays for visual feedback; 3-D audio hardware for localized sound; Haptic interfaces for force and touch feedback; Interested in smell and not yet in taste feedback.
Output Devices Definition: A graphics display is a computer interface that presents synthetic world images to one or several users interacting with the virtual world.
Output Devices Graphics Displays Human stereo viewing; Personal displays; Large volume displays: –Active glasses – Workbenches; – Microsoft Surface – Caves; – Walls;
Output Devices Human Visual System Vision is the dominant sensorial channel; Depth perception in mono images is based - on occlusion (one objects blocks another from view; - on shadows, textures and motion parallax (closer images appear to move more than distant ones)
Human Visual System-continued Depth perception in stereo is based on seteropsiswhen the brain registers and fuses two images; Image parallax means that the two eyes register different images (horizontal shift); The amount of shift depends on the “inter-pupillary distance” (IPD) (varies for each person inthe range of 53-73 mm); Works in the near field (to a few meters from theeye)
Output Devices (same principleused in new 3D HDTVs)Left eye image Right eye image
Output Devices Implications for Stereo Viewing devices Need to present two images of the same VR environment; The two images can be presented at the same time on two displays (HMD); The two images can also be presented time- sequenced on one display (active glasses); The two images can also be presented spatially- sequenced on one display (auto-stereoscopic displays).
Output Devices Personal Displays Definition: A graphics display that outputs a virtual scene destined to be viewed by a single user. Such image may be monoscopic or stereoscopic, monocular (for a single eye) or binocular (displayed on both eyes).
Output Devices Personal Displays Head Mounted Displays; 3-D Binoculars (hand supported); Booms (floor supported); Virtual windows (floor supported); Auto-stereoscopic displays (desk supported).
Output Devices HMD Characteristics Stereo or monoscopic Resolution; Field of view (horizontal); Field of view (vertical); Weight Price
HMD integration in a VR system Consumer HMD Professional HMD
Output Devices HMD Field of view (FoV) – what is good enough? Horizontal FoV Vertical FoV Sensics survey
Output Devices HMD Resolution– what is good enough? Sensics survey
Output Devices HMD Weight – what is good enough? Sensics survey
Output Devices HMD Characteristics (Summary)A field of view of at least 120x50 degrees.At least 1600x1200 resolution, but preferably HD1080.Bright displays with a very fast dynamic response.No more than 250 grams (8-10 oz) in weight.Easy user interface and cable management(based on responses from 84 universities)
AMLCD display, Resolution: 267x225 FOV: 30x23 degrees– Equivalent to 62 in at 2 m Weight: 100 grams Can be worn over glassesOlympus Eye Trek Face Mounted Display (FMD 200)
Olympus Eye Trek Head Mounted Display Optics – uses free-form lens to compensate for aberrations;- an eccentric optical system to reduce size (eliminate 45 degree mirror)
Daeyang “cy-visor” Face Mounted Display LCOS display, Resolution: 800x600 FOV: 60x43 degrees– Weight: 160 grams Can be worn over glasses Liquid Crystal on Silicon display (LCOS)
Daeyang “cy-visor” Face Mounted Display It is reflective – needs external lighting
Organic LEDs (OLED)Active-matrix OLED display, each pixel can be addressed independently via theassociated TFT’s and capacitors in the electronic back plane. Each pixel elementcan be selected to stay “on” during the entire frame time. Since OLED is anemissive device, the display aperture factor is not critical. There are no intrinsic limitations to the pixel count, resolution, orsize of an active-matrix OLED display, leaving the possibilities for commercial useopen to our imagination. Also, because of the TFT’s in the active-matrix design, adefective pixel produces only a dark effect, which is considered to be much lessobjectionable than a bright point defect, like found in LCD’s.
Organic LEDs (OLED)Robust Design - OLED’s are tough enough to use in portable devicesViewing Angles –up to 160 degrees screens provide a clear image, even in brightlight. High Resolution –Each pixel can be turned on or off independently to createmultiple colors in a fluid and smooth edged display. “Electronic Paper” – OLED’s are paper-thin. Due to the exclusion of certainhardware goods that normal LCD’s require, OLED’s are as thin as a dime. Production Advantages –20% to 50% cheaper than LCD processes. Video Capabilities –handle streamlined video, which could revolutionize the PDA andcellular phone market.Hardware Content – Lighter and faster than LCD’s. out of plastic and bendable. do not need lamps, polarizers, or diffusers. Takes less power to run (2 to 10 volts).
5DT Head Mounted Display 800x600 pixels 40o diagonal view Organic LED Frame sequential stereo 600 grams $4k
Samsung Emagin z800 OLED HMD Weight 8 oz PC connection - USB, RGB inputSVGA resolution (800x600 pixels) stereo Tracking - 360 degrees pan 60 degrees pitch $1200 USD www.3dvisor.com
Sensics piSight panoramic HMDsUses Organic LEDA series of micro-displays withspecial optics to generate apanoramic viewWeight 2 lbs (1 Kg)SVGA inputresolution (2400x1729 pixels)Field of view 179ο horizontalby 58ο verticalBinocular overlap 82οCost? USDsensics.com
Sensics xSight panoramic HMDsUses Organic LEDA series of micro-displays withspecial optics to generate apanoramic viewWeight 0.35 kgDVI inputresolution (1680x1050 pixels)Field of view 123ο horizontalby 58ο verticalBinocular overlap 123οCost? USDsensics.com
Sensics wireless HMDA wireless video link optimized HMDsLow latency (<30 msec)Support for HD1080p high definition video atfull 60Hz frame rates.Ability to use multiple transmitter/receiver pairssimultaneously to drive multiple independent HMDsAvailable battery option for both HMD and wireless videolink.Integrated stereo audioHigh-quality H.264 MPEG-4 compressionWi-Fi wireless N technology which does not require line ofsight and works over large distances
Sensics wireless smart goggles•On-board 1.2 GHz dual-core processor withgraphics and 3D co-processor running Android 4.0•Allows execution of on-board applications withoutrequiring connection to a PC or a gaming console.•First-person hand tracking provides real-timetracking and location information of the user’shands.•Hand position can be used to drive user interface,identify gestures and interact with the game.•Embedded head tracker for head angular position andlinear acceleration•Dual SXGA (1280×1024) OLED displays.•64 degree field of view for excellent immersion•Embedded stereo audio and microphone•Battery operated On-board WiFi and Bluetooth services.
Floor-supported displays Boom3C (courtesy of Fakespace Labs.)
21” LCD display, Resolution: SXGA (1600x1200) Weight: Counter- Balanced; No dead space but High latencies due to Third-party trackerVirtual Window 3-D Display (courtesy of Virtual Research Co.)
Output Devices Auto-stereoscopic displays Do not require use of special glasses; Passive auto-stereoscopic displays do not track user’s head and thus restrict user’s position; Active auto-stereoscopic displays track the head motion and give more freedom of motion.
18” LCD display, Resolution: 1280x1024 (mono) 640 x 1024 (stereo) Weight: 17 kgActive auto-stereoscopic 3-D Display (courtesy of Dresden 3D Co.)
Single-user Auto-stereoscopic display… The display redirects the appropriate frames to the right and left eye so that each eye can only see the relevant frame. The tracker locates each eye and sends the information to the control box. The control box then tells the LCD screen what pixels to display. Through the optics system in the screen, the image will get directed through the TFT directly to the appropriate eye. A split second later it would do the same to the other eye. Hence, creating a 3D image.
Active tracking accommodates ±25 degreeschange in view direction
Multi-user Auto-stereoscopic display… Multiple users can be tracked simultaneously and more pixelscan be opened up at any given time allowing light beams to bedirected simultaneously to more than one eye and more than one 3Duser.Position finders already track pupils of multiple viewers with verysmall delay. Good resolution but still shows some flicker. OLED’s becoming mainstream can help eliminate flickerWhat needs to be done?Better displays (100Hz…120Hz)Complete the multi-user concept
Sharp autostereoscopic laptop Pentium 4, 15” diagonal display, 1024x768 resolution, 2D and 3D mode, uses parallax barrier. http://www.inition.co.uk/inition/product_stereovis_sharp_actius_rd3d.php
Autostereoscopic cell phones! Ocuity (UK) and NEC make 2.5” diameter autostereoscopic cellphones. InTouch mobile handset (TTPCom) 2.1” Transflective 2D/3D TFT-LCD 132xRGBx176 pixel display Automatic control of 2D to 3D switching function Running TTPCom WGE 3D stereo game demonstration
Holographic displays • The image source is based on standard flat panel technology of which the image is seen upon a nine optical layer glass panel. Objects will appear to float in space. •For the maximum 3D effect, the background seen through the display should be several feet behind the display and dark in color.http://www.eonreality.com/files/brochures/eon_icrystal_hd.pdf
Holographic displays – EON TouchLight • Bare-hand 3D interaction virtual reality display system • VR scene can be zoomed, panned and rotated with both hands • Uses image processing techniques to combine the output of two video cameras placed behind a semi-transparent plane in front of the user. Incorporates IR cameras and image processing boardhttp://www.eonreality.com/files/brochures/eon_touchlight_hd.pdf
Output Devices Large Volume Displays Allow several co-located users to view a monoscopic or stereoscopic view of the virtual world; Can be classified as monitor-based large volume displays or projector-based large volume displays. Allow more freedom of motion vs. personal displays.
Output Devices Monitor-based Large Volume Displays Use active or passive glasses; Several users can look at a monitor; Can have a single monitor, or multiple side-by- side monitors; If side-by-side, image continuity becomes an issue.
Untracked and wirelessTrackedand wireless Active glasses
Output Devices Active glasses vs. FMDs Some advantages: no cables if head position is not tracked; light and ergonomic (can be used over vision glasses); work well with large volume displays. allows full screen resolution 1280x1024 Some disadvantages: lose 2/3 of image light intensity through LCD filtering; require special CRT “stereo ready” that has twice the hardware refresh rate (Hz) 120 Hz or more; require direct line of sight for IR controller; different viewing metaphor “through the window”.
Wireless – old model Active glassesWireless – new model Wired to the synchronizing jack of the graphics cardI-O Display Systems Inc. $99 vs. $1000 for StereoGraphicswireless glasses
Wired/wireless glasses need a “stereo enabler” when connectedto a VGA card without a 3-pin mini DIN output jack)
Through the window metaphorThe projection factor is changes by a factor K whichsuch that K = r (u – U) + UWhere: r is the responsiveness factor (optimally 1.25); u is the current head distance from the screen; U is the default distance (say 30 cm).Unfortunately tracker jitter is amplified as well
Output DevicesProjector-based Large-Volume Displays Old technology is CRT-based(analog) three projector tubes (R, G, B); Requires special “fast green” coating to avoid the fogging due to fast switching (at 120 Hz); Suffer from low luminosity problems (200-300 lumens)
Output Devices Projector-based Large-Volume Displays Technology made transition from CRT-based (analog) to Digital Micro-mirror Device (DMD) (digital) projectors; Workbench-type displays (Fakespace Responsive Workbench, Barco Baron, V-desk, etc.) Cave-type display (CAVE, RAVE) Wall-type displays Domes
Output Devices Digital Micro-mirror Device DisplayLight intensities are much largerthan for CRT-based projectors300 lumens to 1500 or more lumensThus ambient light does nothinder image quality
Tilted surface Viewing ConeReflector mirrorFloor CRT projector(not shown) The old Fakespace “ImmersaDesk” workbench
IR Controllers CRT Projector Mirrors Tilting mechanism Baron workbench (courtesy of BARCO Co.)
CRT Projector Screen MirrorCAVE 3-D large volume display (courtesy of Fakespace Co.)
CAVE 3-D large volume display (courtesy of Fakespace Co.)
RAVE (“Re-configurable Virtual Environment”) Modular construction that allows various viewingconfiguration, from flat wall, to angled theater, toCAVE; Vertical wall image 2.3 m X 2.4 m; Several CRT projectors (260 lumens, 1280x1024resolution); Takes 30 minutes or less to reconfigure
New types of stereo displays Such as BARCO Trace Driven by Barco Galaxy Stereo DLP projectors 3000 Lumens; 800:1 contrast ratio WARP geometry distortion for edge matching; 1400 x 1050 pixel resolution 70 inch diagonal screen active stereo glasses
Microsoft SURFACE one large display (projector) five infrared camerastracks user’s finger contact with the surfacePC included in the enclosure
40” diagonal Samsung SUR 40 new SURFACE 1920 x 1080 resolutionDual-core CPU, AMD HD6750M GPUViewing angle 178 degreesPixelSense™ to sense fingers and objects touching thescreen. It sees and reacts to light – taking sixty pictures everysecond in a way that is similar to a movie camera.
MultiTouch technologies* Two layers of electrodes. Electrodes parallel in same layer and orthogonal to to the other layer * Capacitive sensors can be constructed from copper or Indium tin oxide (ITO).* Methods for measuring capacitanceare Relaxation Oscillator(in figure),Charge Time, Voltage Divider, ChargeTransfer, Sigma-Delta Modulation.* Self capacitance, mutual capacitance
Resistive* Two ITO patterned plates separated by spacing dots.The top layer has ITO columns and bottom layer hasITO rows.* 5V is applied to a is applied to a column, and voltageis measured at every horizontal sensing line. In order toprevent masking, inactive sensing lines are set to highimpedance Multi touch Optical technologies Images from www.touchuserinterface.com Optical Imaging : two optical sensors track the movement of any object close to the surface by detecting the interruption of an infra-red light source.
Multi touch Optical technologies Rear Diffused Illumination : Infrared light is shone from the opposite side of the touch surface. Finger interrupts the infrared light, reflects back to the camera . Frustrated Total Internal Reflection (FTIR) : floods the inside of a piece of acrylic with infrared light by trapping the light rays within the acrylic. Kinect : Uses near infrared transmitter creates a pattern of near-infrared dots. In-Cell : 2D grid of retro-reflective optosensors which are placed behind an LCD panel. http://www.articlesbase.com/electronics-articles/technologies-of-multitouch-part-3-optical-technologies- 5485162.html
Output Devices Wall-type displays Accommodate more users Using a single projector on a large wall means small image resolution; Thus tiled displays place smaller images side-by-side so they need multiple projectors; Images need to have overlap, to assure continuity; However overlap from two projectors means intensity discontinuity (brighter images in the overlap areas) Projectors need to modulate intensities to dim their light for overlap pixels.
Pano-Wall display Three projectors; Approx. 7 x 2 m2
Tiled composite image from four projectors after adjustment
Wall and Dome-type displaysAdvantages: Accommodate more users (tens to hundreds) Give users more freedom of motion;Disadvantages: Large cost (up to millions of dollars); Even with multiple projectors, resolution is muchlower than for CRTs (because the area is large). Example PanoWall has 200,000 pixels/m2 while amonitor has 18,200,000 pixels/m2 To have equal numbers of pixels/unit are – moreprojectors (military)
Output Devices 3-D Audio Displays Definition: Sound displays are computer interfaces that provide synthetic sound feedback to the user interacting with the virtual world. The sound can be monoaural (both ears hear the same sound) or binaural (each ear hears a different sound).
Output Devices 3-D Audio Displays 3-D audio should not be confused with stereo sound; Human hearing model; HRTF-based 3-D sound; Convolvotron; 3-D sound cards.
Cross-talk effect Sound from one speaker reaches both ears:[ ] [ ][ ] Yleft Yright = Hl,l Hl,r Hr,l Hr,r Sleft Srightwhere Hl,l is the HRTF between the left speaker and the left ear, Hl,r is the HRTF between the right speaker and the left ear, Yleft is the sound reaching the left ear Yright is the sound reaching the right ear
Cross-talk effect cancellation Sound from both speakers is adjusted such that:[ ] [ ][ ] -1 Sleft Hl,l Hl,r Yleft = Sright Hr,l Hr,r Yrightwhere Yleft and Yright are known (the output if the convolvingprocess)
Commercial 3D Sound CardsWhat they have to offer: Digital Output Multi-speaker compatibility (7.1 channel format allows for 8 speakers) Positional Audio offers 3D dimensions of sound
USB 3D Sound Adapter•Supports 3-dimensional sound•Virtual 5.1 sound effects•USB powered, no external power required•Digital Class-B power amplifier•27 Environment sound effects•10 Band / Pre-set equalizer•$30
Sabrent 7.1 PCI Sound Card Internal connectors 8-channels of audio, a PCI interface, 3D sound and great quality $14 PCI Bus connectorAudio ports HRTF-base 3D positional audio, supporting DirectSound 3D, EAX and A3D interface. MIDI Game port Support multi-speaker output to 2/2.1/4.1/5.1/7.1 speakers. Support Karaoke key, Echo sound effects.
Creative Labs Sound Blaster Audigy 4 Pro Aureal 3D (A3D) •Two Versions •1.0 was very similar to DS3D plain Microsoft DirectX component for positional audio •2.0 could more accurately simulate how sound sources in a complex environment behave •An extension of DS3D (by itself, its just reverb) •PC environmental reverb standard created by Creative Lab •All sound cards can have EAX capability •$150
Output Devices Haptic Interfaces Haptics… Comes from Greek Hapthai meaning the sense of touch; Groups touch feedback and force feedback
Output Devices Touch Feedback Relies on sensors in and close to the skin; Conveys information on contact surface geometry, roughness, slippage, temperature; Does not actively resist user contact motion; Easier to implement than force feedback.
Output DevicesForce Feedback Relies on sensors on muscle tendons andbones/joints proprioception; Conveys information on contact surfacecompliance, object weight, inertia; Actively resist user contact motion; More difficult to implement than touch feedback(no commercial products until mid 90s).
Haptic InterfacesHuman touch sensing mechanism Most touch sensors are on the hand (much lessdensity on other parts of the body); Four primary types of sensors:40 % are Meissner’s corpuscles – detect movement across the skin – velocity detectors25% are Merkel’s disks – measure pressure and vibrations13 % are Pacinian corpuscles – deeper in skin (dermis) –acceleration sensors. Most sensitive to vibrations of about 250 Hz19% are Rufini corpuscles – detect skin shear and temperaturechanges
Haptic Interfaces Sensorial adaptation Measure the decrease in electrical signals from the skin sensor over time, for a constant stimulus; If the sensor produces a constant electrical discharge for a constant mechanical stimulus – It is called “Slow Adapting” (SA); If the rate of electrical discharge drops rapidly over time for a constant stimulus – called “Rapidly Adapting” (RA)
Haptic Interfaces Spatial resolution Measure the receptive field size of a sensor; If the sensor has a large receptive field – it has low spatial resolution (Pacinian and Ruffini) SA-II, RA-II If the receptive field is small – has high spatial resolution (Meissner and Merkel) SA-I, RA-I
Haptic InterfacesTwo-point limen test: 2.5 mm fingertip, 11 mm for palm, 67 mmfor thigh
Haptic Interfaces Human grasping configurations
Haptic Interfaces Maximum and sustained force exertion Maximum force exerted during “power” grasp Averages 400 N (male) and 225 N (female); Looking at body location, force output Grows from 50 N at PIP finger joint, to 100 N at shoulder; Sustained force feedback is much smaller than maximum, owing to fatigue and pain
Haptic Interfaces Fatigue measured as a function of % Maximum Voluntary Contraction (MVC) and rest cycle
Haptic Interfaces Haptic feedback actuators Need to maximize power/weight ratio; Need to have high power/volume ratio; Need to have high bandwidth; Need to have high dynamic range (fidelity); Need to be safe for the user - None of the current actuator technology satisfies all these requirements
Haptic Interfaces Actuator comparison based on P/W ratio
Output Devices Touch Feedback Interfaces… Can be desk-top or wearable (gloves); touch feedback mouse; CyberTouch glove; Temperature feedback actuators;
Haptic Interfaces The iFeel Mouse (0-125 Hz).
Haptic Interfaces 6 individually Controlled Vibrotactile actuators 0-125 Hz frequency 1.2 N amplitude at 125 Hz CyberTouch Glove (Virtex)
Output Devices Temperature feedback Added simulation realism by simulating surface thermal “feel”; No moving parts; Uses thermoelectric pumps made of solid-state materials sandwiched between “heat source” and “heat sink”; Single pump can produce 65°C differentials;
User comfort zone13-46°CIf system failsHeat travels backThrough the pumpand can burnskin Temperature feedback actuator control
Output Devices Force Feedback Interfaces… Need mechanical grounding to resist user motion; Can be grounded on desk, wall, or on user body; More difficult to construct and more expensive Than tactile feedback interfaces
NOVINT FALCON3 DOF Force Feedback• 3 DOF (right-left,forward/backward, up/down) 75 ×75× 75 mm• rumble, vibrations•3D exploration and textures• Dynamic effects (inertia, weight,momentum)• Cost less than $300• Resolution >0.06 mm• Max continuous force 10 N• Stiffness 5 N/mm• Connectivity USB 2.0• 1 kHz control bandwidth•http://home.novint.com/
The Haptic Master3 DOF cylindrical robotMax force output 250 NStiffness 50 N/mmUses force-in, position-outarrangement
Exoskeleton over CyberGlove) Cables and pulleys16 N/finger (continuous?); Weight 539 grams; remote electrical actuators in a control box. The CyberGrasp force feedback glove
The CyberGrasp force feedback gloveVC 3.4 on book CD
CyberGrasp gloveElectronicinterface box Tether Wrist Tracker The CyberPack (courtesy of Virtex Co.)
6 DOF mechanical armWrist position andForce feedback –No need for a trackerAllows simulation of weight and inertia, not possible withglove-only interfaces CyberForce interface (introduced recently)
Haptic Interfaces VC 3.5 on book CyberForce interface CD
Olfactory FeedbackOlfactory feedback refers to the inclusion of the sense ofsmell as an output in a virtual world or simulation.Aroma therapy has proven to be an effective form oftreatment for rehabilitation and treatment.The notion of “olfactory-evoked recall” can be vital toimmersion in a VR world. “Smell has the greatest impacton human emotions.” (Enviroscent)
Challenges in Smell TechnologiesNo “primary smells” unlike colors, makes smell synthesisdifficult.Consumable.More difficult to modulate and control compared to soundand light.Size.
Applications of Olfactory StimuliUsed by Dr. Albert Rizzo at USC for treatment ofPTSD patients.USAF researched FiVe FiRe training system for fire fighters whichincludes multiple odors for burning objects.Entertainment purposes; the originalSmell-o-Vision was used in theaters toadd additional movie going enjoyment.
Technologies for Olfactory StimuliEnviroscent has a line of smell machines suchas the ES-1 Fragrance Machine.Scents in Metagel dispersion cartridges-8 different scents gunpowder, diesel, burningrubber, body odor, etc.)- through which compressed air is pumped tocarry the smell temporarily into the userssimulation space.- researchers think PTSD that this adds to thesense of presence. Since the olfactory bulb isclosely linked to areas of the limbic systemimplicated to be involved with memory andemotion
Smell as InputSmell is actually a relatively common input source withapplications in medicine, security, and more.Can be used to detect pollutants, identify diseases,or even used to identify an individual.Typically designed as an array of chemical sensors.
VR for Addict RehabilitationVirtually Better, a company focused on VRrehabilitation techniques, in cooperation withaddiction researcher Dr. Bordnick of theUniversity of Houston, have created VRSimulations of addict tempting environments. Video Courtesy ScienCentral, Inc.
Further Reading and ResourcesAlbert Rizzo articleSmell CannonUoW Document on Olfaction for VRSony/Matsushita SmellovisionUsing VR with Olfaction for Addiction Rehabilitation Robot Nose