Haptic Interaction in Virtual and Mixed Reality EnvironmentsKonstantinos Moustakas, Assistant ProfessorElectrical and Comp...
Outline                                     Haptic interaction                                  Outline    • What is hapti...
Introduction                                  Haptic interaction                           Haptics    • Wikipedia, Haptics...
Introduction                         Haptic interaction                           Haptics    • Tactile information        ...
Introduction                                         Haptic interaction                 Haptic interaction - Importance   ...
Introduction                                            Haptic interaction    Haptic interaction - Importance             ...
Haptic Psychophysics                    Haptic interaction             Different types of tactile sensors
Haptic Psychophysics             Haptic interaction     Surface encoding –deformation matters
Haptic Psychophysics                        Haptic interaction                       Spatial resolution
Haptic Psychophysics                         Haptic interaction                       Temporal resolution                 ...
Haptic Psychophysics                        Haptic interaction                       What do we sense?   • Tactile        ...
Haptic Rendering                       Haptic interaction                   Haptics & Environment
Haptic Devices                                        Haptic interaction                                Haptic devices   •...
Haptic Devices                    Haptic interaction                 Haptic devices
Haptic Devices                    Haptic interaction                 Haptic devices
Haptic Devices                    Haptic interaction                 Haptic devices
Haptic Devices                    Haptic interaction                 Haptic devices
Haptic Devices                    Haptic interaction                 Haptic devices
Haptic Devices                    Haptic interaction                 Haptic devices
Haptic Devices                    Haptic interaction                 Haptic devices
Haptic Devices                    Haptic interaction                 Haptic devices
Haptic Devices                    Haptic interaction                 Haptic devices
Haptic Devices                            Haptic interaction           Haptic devices VS Human Sensors         Human senso...
Haptic Devices                     Haptic interaction           Haptic devices VS Human Sensors
Haptic Rendering                                          Haptic interaction     Haptic vs Graphics rendering processesApp...
Collision detection                                 Haptic interaction                        Collision detection    • Col...
Collision detection                         Haptic interaction                      Collision detection             In no ...
Collision detection                      Haptic interaction                  Bounding volume hierarchies
Collision detection                 Haptic interaction         Bounding volume hierarchies (video)
Collision detection                                                            Haptic interaction                         ...
Haptic Rendering                        Haptic interaction                   Haptic rendering basics
Haptic Rendering                                        Haptic interaction       Haptic rendering: Stiffness vs StabilityV...
Haptic Rendering                       Haptic interaction                   Force discontinuities
Haptic Rendering                   Haptic interaction                   Force shading
Haptic Rendering                      Haptic interaction                   Haptic texturing   •   Height maps   •   Fricti...
Haptic rendering                                                                Haptic interaction                The forc...
Haptic rendering                                                                Haptic interaction     From the distance f...
Haptic rendering                                        Haptic interaction                            Haptic rendering    ...
SQ-Map: Efficient layered collision              detection and haptic renderingK. Moustakas, D. Tzovaras and M.G. Strintzi...
Haptic rendering                                                               Haptic interaction                  SQ-    ...
Haptic rendering                      Haptic interaction                   SQ-                   SQ-Map distance maps
Haptic rendering                                                    Haptic interaction                   Collision detecti...
Haptic rendering                                                    Haptic interaction                   Collision detecti...
Haptic rendering                           Haptic interaction                   Collision detection: Layer 2
Haptic rendering                                             Haptic interaction                   Collision detection: Lay...
Haptic rendering                            Haptic interaction                   Collision detection: Layer 3    • Similar...
Haptic rendering                              Haptic interaction                     Accuracy analysis    • Lemma:        ...
Haptic rendering                          Haptic interaction                   Accuracy analysis                     Conca...
Haptic rendering                                Haptic interaction                   Haptic rendering                   Fo...
Haptic rendering                                    Haptic interaction                   Haptic rendering                 ...
Haptic rendering                                    Haptic interaction                         Haptic rendering           ...
Haptic rendering                 Haptic interaction                   Comparisons
Haptic rendering                                  Haptic interaction                         Qualitative results          ...
Haptic rendering                         Haptic interaction                   Qualitative results
Applications                       Haptic interaction                Application examples    • Surgical simulation    • Te...
Applications                                                                      Haptic interaction                      ...
Applications                    Haptic interaction               Haptic browser
Applications                                                                    Haptic interaction                Virtual ...
Applications                                                               Haptic interaction               Mixed reality ...
Applications                                                                 Haptic interaction               VR game for ...
Applications                    Haptic interaction               The ideal case
Applications                  Haptic interaction               Architecture
Applications                Haptic interaction               Game steps
Applications                Haptic interaction               Game video
Future work                                  Haptic interaction                         Future directions   • Haptic visua...
Haptic Interaction in Virtual and Mixed Reality Environments              Thank you for your                  attentionKon...
Upcoming SlideShare
Loading in …5
×

Haptic Interaction in Mixed Environments and Virtual Reality

3,158 views

Published on

Event: "IEEE Day 2012"

IEEE Student Branch-University of Patras

09.10.2012

"Haptic Interaction in Mixed Environments and Virtual Reality" by K.Moustakas

Museum of Sciences & Technology of the University of Patras

More info: http://ieee-upatras.gr/en/events/2012/123-ieee-day-2012

Published in: Technology
0 Comments
8 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,158
On SlideShare
0
From Embeds
0
Number of Embeds
16
Actions
Shares
0
Downloads
254
Comments
0
Likes
8
Embeds 0
No embeds

No notes for slide

Haptic Interaction in Mixed Environments and Virtual Reality

  1. 1. Haptic Interaction in Virtual and Mixed Reality EnvironmentsKonstantinos Moustakas, Assistant ProfessorElectrical and Computer Engineering Department, University of Patras
  2. 2. Outline Haptic interaction Outline • What is haptic interaction? • Haptic psychophysics outline • Haptic interaction essential processing: processing: – Collision detection – Haptic rendering • Applications • Future research directions
  3. 3. Introduction Haptic interaction Haptics • Wikipedia, Haptics: Haptics refers to the sense of touch (from Greek άπτω = "I fasten onto, I touch). • Merriam-Webster, Haptic: relating to or based on the sense of touch • Haptics is the science of applying touch (tactile) sensation and control to interaction with computer applications.
  4. 4. Introduction Haptic interaction Haptics • Tactile information – The responses of the receptors of the skin in contact with an object • Kinesthetic information – Net forces applied to limbs
  5. 5. Introduction Haptic interaction Haptic interaction - Importance • The sense of touch is underestimated! • Loss of the sense of touch can cause impairments that cannot be compensated by sight – Hand dexterity – Haptic capabilities, grasping, etc. – Walking – Limb position perception – Eating – Speaking • In virtual environments the loss of the sense of touch can impair performance and interaction efficiency
  6. 6. Introduction Haptic interaction Haptic interaction - Importance Strike a match (with vision) Normal case Anesthesia 7.5 sec 29 sec
  7. 7. Haptic Psychophysics Haptic interaction Different types of tactile sensors
  8. 8. Haptic Psychophysics Haptic interaction Surface encoding –deformation matters
  9. 9. Haptic Psychophysics Haptic interaction Spatial resolution
  10. 10. Haptic Psychophysics Haptic interaction Temporal resolution ~1 kHz constraint while in graphics ~25 Hz constraint
  11. 11. Haptic Psychophysics Haptic interaction What do we sense? • Tactile • Haptic – Surface texture – Mass/Weight – Orientation – Force – Shape – Stiffness – Size – Viscosity – Pressure • Thermal – Curvature – Cooling – Motion – Warming
  12. 12. Haptic Rendering Haptic interaction Haptics & Environment
  13. 13. Haptic Devices Haptic interaction Haptic devices • Haptic devices: – Tactile devices – Force feedback devices • Exoskeletons • One point of interaction devices
  14. 14. Haptic Devices Haptic interaction Haptic devices
  15. 15. Haptic Devices Haptic interaction Haptic devices
  16. 16. Haptic Devices Haptic interaction Haptic devices
  17. 17. Haptic Devices Haptic interaction Haptic devices
  18. 18. Haptic Devices Haptic interaction Haptic devices
  19. 19. Haptic Devices Haptic interaction Haptic devices
  20. 20. Haptic Devices Haptic interaction Haptic devices
  21. 21. Haptic Devices Haptic interaction Haptic devices
  22. 22. Haptic Devices Haptic interaction Haptic devices
  23. 23. Haptic Devices Haptic interaction Haptic devices VS Human Sensors Human sensors Haptic devices Many sensors One sensor (x N) Direct tactile sensation Only through motion Deformation N/A • Haptic devices act as a filter to the human tactile/haptic sensors. • How would you see a visual scene with such a filter?
  24. 24. Haptic Devices Haptic interaction Haptic devices VS Human Sensors
  25. 25. Haptic Rendering Haptic interaction Haptic vs Graphics rendering processesApplication Geometry Rasterizer Display Scene View Lighting Projection Texturing Traversal TransformTraversal Force Tactile Display Collision Force Force Force Haptic Detection Calculation Smoothing Mapping Texturing
  26. 26. Collision detection Haptic interaction Collision detection • Collision is one of natures powerful tools: – Object interaction • Walking • Machines are based on collisions – Sound source • In the real world physical laws and matter take-over • In the virtual world???
  27. 27. Collision detection Haptic interaction Collision detection In no case a trivial issue!!!
  28. 28. Collision detection Haptic interaction Bounding volume hierarchies
  29. 29. Collision detection Haptic interaction Bounding volume hierarchies (video)
  30. 30. Collision detection Haptic interaction Force feedback A. Vogiannou, K. Moustakas, D. Tzovaras and M.G. Strintzis, “Enhancing Bounding Volumes using Support Plane Mappings for Collision Detection”, Eurographics Computer Graphics Forum, vol. 29, no. 5, pp. 1595- 1604, August 2010.
  31. 31. Haptic Rendering Haptic interaction Haptic rendering basics
  32. 32. Haptic Rendering Haptic interaction Haptic rendering: Stiffness vs StabilityVirtual wall Moving into the wall Force Calculation – F Virtual wall V<0 Insufficient stiffness time Solutions High frequency F = K wall • ∆ x + B v Virtual coupling God object – virtual proxy
  33. 33. Haptic Rendering Haptic interaction Force discontinuities
  34. 34. Haptic Rendering Haptic interaction Force shading
  35. 35. Haptic Rendering Haptic interaction Haptic texturing • Height maps • Friction/Damping fields • Stochastic textures • Virtual textures/bumps
  36. 36. Haptic rendering Haptic interaction The force field haptic rendering method K. Moustakas, G. Nikolakis, K. Kostopoulos, D. Tzovaras and M.G. Strintzis, “Haptic Rendering of Visual Data for the Visually Impaired”, IEEE Multimedia, vol. 14, no. 1, pp. 62-72, January 2007.
  37. 37. Haptic rendering Haptic interaction From the distance field to the force field Distance field di -di -di Force field K. Moustakas, G. Nikolakis, K. Kostopoulos, D. Tzovaras and M.G. Strintzis, “Haptic Rendering of Visual Data for the Visually Impaired”, IEEE Multimedia, vol. 14, no. 1, pp. 62-72, January 2007.
  38. 38. Haptic rendering Haptic interaction Haptic rendering • The force feedback is calculated for a given point x ∈ R 3 through: • where D(x) and F(x) are the values and the vector of the distance and force field respectively Distance field Force field di -di -di
  39. 39. SQ-Map: Efficient layered collision detection and haptic renderingK. Moustakas, D. Tzovaras and M.G. Strintzis, “SQ-Map: Efficient Layered Collision Detection and HapticRendering”, IEEE Transactions on Visualization and Computer Graphics, vol. 13, no. 1, pp. 80 - 93,January 2007.
  40. 40. Haptic rendering Haptic interaction SQ- SQ-Map overview • Motivation: New method fast as the distance fields, without high memory requirements • SQ-Map steps – Preprocessing • Object approximate segmentation • Superquadric modeling • Distance map generation – Run-time processing • Layered collision detection (3 levels of processing) K. Moustakas, D. Tzovaras and M.G. Strintzis, “SQ-Map: Efficient Layered Collision Detection and Haptic Rendering”, IEEE Transactions on Visualization and Computer Graphics, vol. 13, no. 1, pp. 80 - 93, January 2007.
  41. 41. Haptic rendering Haptic interaction SQ- SQ-Map distance maps
  42. 42. Haptic rendering Haptic interaction Collision detection: Layer 1 ε2  2 2  ε1 2  x  2 +  y  2  +  z  1 > 1 ε ε ε F ( x, y , z ) =          a1    a2    a3   
  43. 43. Haptic rendering Haptic interaction Collision detection: Layer 1 ε2  2 2  ε1 2  x  2 +  y  2  +  z  1 < 1 ε ε ε F ( x, y , z ) =          a1    a2    a3   
  44. 44. Haptic rendering Haptic interaction Collision detection: Layer 2
  45. 45. Haptic rendering Haptic interaction Collision detection: Layer 2 P DSQ d If d>DSQ collision is detected
  46. 46. Haptic rendering Haptic interaction Collision detection: Layer 3 • Similar processing to layer 2 with two add- ons – Possibility to perform distance map subdivision so as to increase accuracy – The Layer 3 distance map is used that projects also mesh vertices onto the superquadric surfaces
  47. 47. Haptic rendering Haptic interaction Accuracy analysis • Lemma: – The distance map and the superquadric can provide an exact representation of the 3D model iff function fC that maps all points of the object’s surface onto the superquadric is injective. – Proof in the text
  48. 48. Haptic rendering Haptic interaction Accuracy analysis Concave region of type Q2 Concave region of type Q1
  49. 49. Haptic rendering Haptic interaction Haptic rendering Force magnitude estimation
  50. 50. Haptic rendering Haptic interaction Haptic rendering Force direction estimation Can be analytically calculated
  51. 51. Haptic rendering Haptic interaction Haptic rendering Advantages: • Rapid estimation of the reaction force • Possibility to analytically process and handle the force feedback Friction: Haptic texture: Gaussian noise
  52. 52. Haptic rendering Haptic interaction Comparisons
  53. 53. Haptic rendering Haptic interaction Qualitative results OBB SQ-Map, Layer 1 SQ-Map, Layer 2 SQ-Map, Layer 3
  54. 54. Haptic rendering Haptic interaction Qualitative results
  55. 55. Applications Haptic interaction Application examples • Surgical simulation • Telemanipulation, robot-assisted surgery • Computer aided design • Entertainment • Haptic Visualization • Applications for the disabled • …
  56. 56. Applications Haptic interaction Haptic maps Haptic map with embedded semantic iformation K. Moustakas, G. Nikolakis, K. Kostopoulos, D. Tzovaras and M.G. Strintzis, “Haptic Rendering of Visual Data for the Visually Impaired”, IEEE Multimedia, vol. 14, no. 1, pp. 62-72, January 2007.
  57. 57. Applications Haptic interaction Haptic browser
  58. 58. Applications Haptic interaction Virtual reality cane simulation D. Tzovaras et.al., “Design and implementation of haptic virtual environments for the training of the visually impaired, IEEE Neural Systems and Rehab. Eng., 2004
  59. 59. Applications Haptic interaction Mixed reality cane simulation Magnetic Virtual environment sensor 1CyberGrasp Virtual cane Cane replica Magnetic sensor 2 D. Tzovaras, K. Moustakas, G. Nikolakis and M.G. Strintzis, "Interactive Mixed Reality White Cane Simulation for the Training of the Blind and the Visually Impaired", Springer Journal on Personal and Ubiquitous Computing, vol.13, no.1, pp.51-58, January 2009.
  60. 60. Applications Haptic interaction VR game for the hearing and the visually impairedK. Moustakas, L. Dybkjaer, O. Aran, D. Tzovaras and N.O. Bernsen, "Communication Between Blind and HearingImpaired People Through a Multimodal Interactive Game", IEEE Multimedia, accepted for publication.
  61. 61. Applications Haptic interaction The ideal case
  62. 62. Applications Haptic interaction Architecture
  63. 63. Applications Haptic interaction Game steps
  64. 64. Applications Haptic interaction Game video
  65. 65. Future work Haptic interaction Future directions • Haptic visualization – Use the haptic channel to complement vision in information visualization applications • Haptic interaction with videos – Highly dynamic • Content • Interaction – Partial input of the environment, ill-posed • Towards a theory of haptic rendering…
  66. 66. Haptic Interaction in Virtual and Mixed Reality Environments Thank you for your attentionKonstantinos Moustakas, Assistant Professormoustakas@ece.upatras.grElectrical and Computer Engineering Department, University of Patras

×