Event: "IEEE Day 2012" IEEE Student Branch-University of Patras 09.10.2012 "Haptic Interaction in Mixed Environments and Virtual Reality" by K.Moustakas Museum of Sciences & Technology of the University of Patras More info: http://ieee-upatras.gr/en/events/2012/123-ieee-day-2012

1. Haptic Interaction in Virtual and Mixed Reality Environments
Konstantinos Moustakas, Assistant Professor
Electrical and Computer Engineering Department, University of Patras

2. Outline Haptic interaction
Outline
• What is haptic interaction?
• Haptic psychophysics outline
• Haptic interaction essential processing:
processing:
– Collision detection
– Haptic rendering
• Applications
• Future research directions

3. Introduction Haptic interaction
Haptics
• Wikipedia, Haptics: Haptics refers to the sense of
touch (from Greek άπτω = "I fasten onto, I touch).
• Merriam-Webster, Haptic: relating to or based on
the sense of touch
• Haptics is the science of applying touch
(tactile) sensation and control to interaction
with computer applications.

4. Introduction Haptic interaction
Haptics
• Tactile information
– The responses of
the receptors of the
skin in contact with
an object
• Kinesthetic
information
– Net forces applied
to limbs

5. Introduction Haptic interaction
Haptic interaction - Importance
• The sense of touch is underestimated!
• Loss of the sense of touch can cause impairments that
cannot be compensated by sight
– Hand dexterity
– Haptic capabilities, grasping, etc.
– Walking
– Limb position perception
– Eating
– Speaking
• In virtual environments the loss of the sense of touch can
impair performance and interaction efficiency

6. Introduction Haptic interaction
Haptic interaction - Importance
Strike a match (with vision)
Normal case Anesthesia
7.5 sec 29 sec

7. Haptic Psychophysics Haptic interaction
Different types of tactile sensors

8. Haptic Psychophysics Haptic interaction
Surface encoding –deformation matters

9. Haptic Psychophysics Haptic interaction
Spatial resolution

10. Haptic Psychophysics Haptic interaction
Temporal resolution
~1 kHz constraint
while in graphics
~25 Hz constraint

11. Haptic Psychophysics Haptic interaction
What do we sense?
• Tactile • Haptic
– Surface texture – Mass/Weight
– Orientation – Force
– Shape – Stiffness
– Size – Viscosity
– Pressure • Thermal
– Curvature – Cooling
– Motion – Warming

12. Haptic Rendering Haptic interaction
Haptics & Environment

13. Haptic Devices Haptic interaction
Haptic devices
• Haptic devices:
– Tactile devices
– Force feedback devices
• Exoskeletons
• One point of interaction devices

14. Haptic Devices Haptic interaction
Haptic devices

15. Haptic Devices Haptic interaction
Haptic devices

16. Haptic Devices Haptic interaction
Haptic devices

17. Haptic Devices Haptic interaction
Haptic devices

18. Haptic Devices Haptic interaction
Haptic devices

19. Haptic Devices Haptic interaction
Haptic devices

20. Haptic Devices Haptic interaction
Haptic devices

21. Haptic Devices Haptic interaction
Haptic devices

22. Haptic Devices Haptic interaction
Haptic devices

23. Haptic Devices Haptic interaction
Haptic devices VS Human Sensors
Human sensors Haptic devices
Many sensors One sensor (x N)
Direct tactile sensation Only through motion
Deformation N/A
• Haptic devices act as a filter to the human
tactile/haptic sensors.
• How would you see a visual scene with such a
filter?

24. Haptic Devices Haptic interaction
Haptic devices VS Human Sensors

25. Haptic Rendering Haptic interaction
Haptic vs Graphics rendering processes
Application Geometry Rasterizer Display
Scene View
Lighting Projection Texturing
Traversal Transform
Traversal Force Tactile Display
Collision Force Force Force Haptic
Detection Calculation Smoothing Mapping Texturing

26. Collision detection Haptic interaction
Collision detection
• Collision is one of natures powerful tools:
– Object interaction
• Walking
• Machines are based on collisions
– Sound source
• In the real world physical laws and matter
take-over
• In the virtual world???

27. Collision detection Haptic interaction
Collision detection
In no case a trivial issue!!!

28. Collision detection Haptic interaction
Bounding volume hierarchies

29. Collision detection Haptic interaction
Bounding volume hierarchies (video)

30. Collision detection Haptic interaction
Force feedback
A. Vogiannou, K. Moustakas, D. Tzovaras and M.G. Strintzis, “Enhancing Bounding Volumes using Support
Plane Mappings for Collision Detection”, Eurographics Computer Graphics Forum, vol. 29, no. 5, pp. 1595-
1604, August 2010.

31. Haptic Rendering Haptic interaction
Haptic rendering basics

32. Haptic Rendering Haptic interaction
Haptic rendering: Stiffness vs Stability
Virtual wall Moving into the wall Force Calculation –
F Virtual wall
V<0
Insufficient stiffness
time
Solutions
High frequency
F = K wall • ∆ x + B v
Virtual coupling
God object – virtual proxy

33. Haptic Rendering Haptic interaction
Force discontinuities

34. Haptic Rendering Haptic interaction
Force shading

35. Haptic Rendering Haptic interaction
Haptic texturing
• Height maps
• Friction/Damping fields
• Stochastic textures
• Virtual textures/bumps

36. Haptic rendering Haptic interaction
The force field haptic rendering
method
K. Moustakas, G. Nikolakis, K. Kostopoulos, D. Tzovaras and M.G. Strintzis, “Haptic Rendering of Visual
Data for the Visually Impaired”, IEEE Multimedia, vol. 14, no. 1, pp. 62-72, January 2007.

37. Haptic rendering Haptic interaction
From the distance field to the force field
Distance field
di
-di -di
Force field
K. Moustakas, G. Nikolakis, K. Kostopoulos, D. Tzovaras and M.G. Strintzis, “Haptic Rendering of Visual
Data for the Visually Impaired”, IEEE Multimedia, vol. 14, no. 1, pp. 62-72, January 2007.

38. Haptic rendering Haptic interaction
Haptic rendering
• The force feedback is calculated for a given
point x ∈ R 3 through:
• where D(x) and F(x) are the values and the
vector of the distance and force field respectively
Distance field Force field
di
-di -di

39. SQ-Map: Efficient layered collision
detection and haptic rendering
K. Moustakas, D. Tzovaras and M.G. Strintzis, “SQ-Map: Efficient Layered Collision Detection and Haptic
Rendering”, IEEE Transactions on Visualization and Computer Graphics, vol. 13, no. 1, pp. 80 - 93,
January 2007.

40. Haptic rendering Haptic interaction
SQ-
SQ-Map overview
• Motivation: New method fast as the
distance fields, without high memory
requirements
• SQ-Map steps
– Preprocessing
• Object approximate segmentation
• Superquadric modeling
• Distance map generation
– Run-time processing
• Layered collision detection (3 levels of processing)
K. Moustakas, D. Tzovaras and M.G. Strintzis, “SQ-Map: Efficient Layered Collision Detection and Haptic
Rendering”, IEEE Transactions on Visualization and Computer Graphics, vol. 13, no. 1, pp. 80 - 93,
January 2007.

41. Haptic rendering Haptic interaction
SQ-
SQ-Map distance maps

42. Haptic rendering Haptic interaction
Collision detection: Layer 1
ε2
 2 2

ε1 2
 x  2 +  y  2  +  z  1 > 1
ε ε ε
F ( x, y , z ) =       
  a1 
  a2    a3 
 

43. Haptic rendering Haptic interaction
Collision detection: Layer 1
ε2
 2 2

ε1 2
 x  2 +  y  2  +  z  1 < 1
ε ε ε
F ( x, y , z ) =       
  a1 
  a2    a3 
 

44. Haptic rendering Haptic interaction
Collision detection: Layer 2

45. Haptic rendering Haptic interaction
Collision detection: Layer 2
P
DSQ d
If d>DSQ collision is detected

46. Haptic rendering Haptic interaction
Collision detection: Layer 3
• Similar processing to layer 2 with two add-
ons
– Possibility to perform distance map
subdivision so as to increase accuracy
– The Layer 3 distance map is used that
projects also mesh vertices onto the
superquadric surfaces

47. Haptic rendering Haptic interaction
Accuracy analysis
• Lemma:
– The distance map and the superquadric can
provide an exact representation of the 3D
model iff function fC that maps all points of the
object’s surface onto the superquadric is
injective.
– Proof in the text

48. Haptic rendering Haptic interaction
Accuracy analysis
Concave
region of type
Q2
Concave
region of type
Q1

49. Haptic rendering Haptic interaction
Haptic rendering
Force magnitude estimation

50. Haptic rendering Haptic interaction
Haptic rendering
Force direction estimation
Can be analytically
calculated

51. Haptic rendering Haptic interaction
Haptic rendering
Advantages:
• Rapid estimation of the reaction force
• Possibility to analytically process and handle the
force feedback
Friction:
Haptic texture:
Gaussian noise

52. Haptic rendering Haptic interaction
Comparisons

53. Haptic rendering Haptic interaction
Qualitative results
OBB SQ-Map, Layer 1
SQ-Map, Layer 2 SQ-Map, Layer 3

54. Haptic rendering Haptic interaction
Qualitative results

55. Applications Haptic interaction
Application examples
• Surgical simulation
• Telemanipulation,
robot-assisted
surgery
• Computer aided
design
• Entertainment
• Haptic Visualization
• Applications for the
disabled
• …

56. Applications Haptic interaction
Haptic maps
Haptic map with embedded
semantic iformation
K. Moustakas, G. Nikolakis, K. Kostopoulos, D. Tzovaras and M.G. Strintzis, “Haptic
Rendering of Visual Data for the Visually Impaired”, IEEE Multimedia, vol. 14, no. 1, pp.
62-72, January 2007.

57. Applications Haptic interaction
Haptic browser

58. Applications Haptic interaction
Virtual reality cane simulation
D. Tzovaras et.al., “Design and implementation of haptic virtual environments for the
training of the visually impaired, IEEE Neural Systems and Rehab. Eng., 2004

59. Applications Haptic interaction
Mixed reality cane simulation
Magnetic Virtual environment
sensor 1
CyberGrasp Virtual
cane
Cane
replica
Magnetic
sensor 2
D. Tzovaras, K. Moustakas, G. Nikolakis and M.G. Strintzis, "Interactive Mixed Reality
White Cane Simulation for the Training of the Blind and the Visually Impaired", Springer
Journal on Personal and Ubiquitous Computing, vol.13, no.1, pp.51-58, January 2009.

60. Applications Haptic interaction
VR game for the hearing and the
visually impaired
K. Moustakas, L. Dybkjaer, O. Aran, D. Tzovaras and N.O. Bernsen, "Communication Between Blind and Hearing
Impaired People Through a Multimodal Interactive Game", IEEE Multimedia, accepted for publication.

61. Applications Haptic interaction
The ideal case

62. Applications Haptic interaction
Architecture

63. Applications Haptic interaction
Game steps

64. Applications Haptic interaction
Game video

65. Future work Haptic interaction
Future directions
• Haptic visualization
– Use the haptic channel to complement vision
in information visualization applications
• Haptic interaction with videos
– Highly dynamic
• Content
• Interaction
– Partial input of the environment, ill-posed
• Towards a theory of haptic rendering…

66. Haptic Interaction in Virtual and Mixed Reality Environments
Thank you for your
attention
Konstantinos Moustakas, Assistant Professor
moustakas@ece.upatras.gr
Electrical and Computer Engineering Department, University of Patras