This document discusses human perception and interaction in virtual reality. It explains that VR hijacks the human perception-action system by providing distal stimuli (the virtual environment) and tracking human actions like head and hand movements as input. It describes several common interaction patterns used in VR like hand selection, pointing, image-plane selection, 3D multitouch, and 3D tool manipulation. Throughout development, iteration is important to refine VR interactions based on how users perceive and interact with the virtual environment.

1. 1@TheVRBookJason Jerald, Phd
VR Interactions
Jason Jerald, PhD

2. 2@TheVRBookJason Jerald, Phd
60+ VR Projects, 30+ Organizations

3. 3@TheVRBookJason Jerald, Phd
The Psychology of Perception & Action
• Objective vs subjective reality
• Distal and proximal stimuli
• Sensation vs perception
• Bottom-up and top-down
processing
• Afference and efference
• Iterative perceptual processing
• The subconscious and conscious
• Mental models
• Visceral, behavioral, reflective,
and emotional processes
• Neuro-linguistic programming
• Space, time, & motion
• Perceptual constancies
• Adaptation
• Attention
• Action

4. 4@TheVRBookJason Jerald, Phd
VR Output = Distal stimuli
Image Courtesy of The VR Book
(adapted from Goldstein [2007])
The Iterative Perceptual Process
Human action = VR input/tracking
With VR, we hijack the
human perception-action
system

5. 5@TheVRBookJason Jerald, Phd
The Cycle of Interaction
Image Courtesy of The VR Book (adapted from Norman [2013])

6. 6@TheVRBookJason Jerald, Phd
Input Device Classes

7. 7@TheVRBookJason Jerald, Phd
The Most Important Input Device?

8. 8@TheVRBookJason Jerald, Phd
The Most Important Input Device?
The Human Hands!
– Large proportion of sensory and motor
cortex devoted to the hands
– Hand tracking technology is simply the
mediator that brings the hands into VR

9. 9@TheVRBookJason Jerald, Phd
The First Hand Technique
Image courtesy of NextGen Interactions

10. 10@TheVRBookJason Jerald, Phd
Interaction Patterns
• Hand
Selection
• Pointing
• Image-
Plane
• Volume
Based
Selection
• Walking
• Steering
• 3D Multi-
Touch
• Automated
Viewpoint
Control
• Direct Hand
• Proxy
• 3D Tool
Manipulation
• Widgets &
Panels
• Non Spatial
Indirect
Control
• Pointing
Hand
• World-in-
Miniature
• MultiModal
Compound

11. 11@TheVRBookJason Jerald, Phd
The Hand Selection Pattern
Hands with
Arms
Video Courtesy of Digital ArtForms

12. 12@TheVRBookJason Jerald, Phd
The Pointing Pattern
Image Courtesy of Digital ArtForms

13. 13@TheVRBookJason Jerald, Phd
The Image-Plane Pattern
The Head-Crusher
Technique
From Pierce et al. [1997]

14. 14@TheVRBookJason Jerald, Phd
Widgets & Panels Pattern
• Hand-held panels
– Buttons
– Sliders
– Dials
– Color cubes
Images Courtesy of Sixense & Digital ArtForms

15. 15@TheVRBookJason Jerald, Phd
World-In-Miniature Pattern
Referenced
world view
within a box
Images Courtesy of Digital ArtForms

16. 16@TheVRBookJason Jerald, Phd
3D Multitouch Pattern
• An extension of 2D Multitouch
• Manipulate the world as an object
– Reduces sickness
– Solves gorilla arm
• Low interaction fidelity
– Appropriate for abstract non-realistic interactions
• Content/data independent
– Emergency preparedness
– Medical Visualization
– MakeVR

17. 17@TheVRBookJason Jerald, Phd
3D Multitouch—Medical Visualization
Video Courtesy of ACM SIGGRAPH

18. 18@TheVRBookJason Jerald, Phd
3D Multitouch—Content Creation
MakeVR Collaborative
Video Courtesy of Sixense

19. 19@TheVRBookJason Jerald, Phd
3D Tools Pattern
The Jigs Technique
Images Courtesy of Digital ArtForms and Sixense

20. 20@TheVRBookJason Jerald, Phd
What is Common to All VR Dev?
Iteration!

21. 21@TheVRBookJason Jerald, Phd
Questions