CHI2021, gaze tracking, vr, object manipulation

1. Interaction Lab. Seoul National University of Science and Technology
Gaze-Supported 3D Object Manipulation
in Virtual Reality
Jeong Jae-Yeop
CHI Conference on Human Factors in Computing Systems (CHI ’21) , May 8–13, 2021, Yokohama, Japan
ACM, New York, NY, USA, 13 pages(https://doi.org/10.1145/3411764.3445343)
Difeng Yu, Xueshi Lu, Rongkai Shi, Hai-Ning Liang,
Tilman Dingler, Eduardo Velloso, and Jorge Goncalves. 2021

2. Interaction Lab., Seoul National University of Science and Technology
■Intro
■Design space
■Technique design
■Study 1 : Evaluation
■Study 2 : Application
■Discussion and conclusion
Agenda
2

3. Intro
Design space
3

4. Interaction Lab., Seoul National University of Science and Technology
■Object manipulation in VR(Virtual Reality)
 Manipulation
• Translation
• Rotation
• Scaling
 Application domains
• 3D modeling
• Game development
• Online collaboration
• Immersive data exploration
Intro(1/4)
4

5. Interaction Lab., Seoul National University of Science and Technology
■Input modality in VR
 Virtual hand
• Direct manipulation
• Inefficient and imprecise
• Arm-fatigue in longer interaction scenarios
 Gaze
• Light-weight and fast input
• Only used mostly for “target selection”
• Sub-phase of the whole “manipulate” phase
■Multimodal interaction in VR
 Gaze + Virtual hand
Intro(2/4)
5

6. Interaction Lab., Seoul National University of Science and Technology
■Aim
 Whether the incorporation of gaze input can benefit the hand manipulation in VR
 How gaze input should be combined with hand input for convenient and efficient
■Design space
 Integrated
 Coordinate
 Transition
■Four gaze-supported techniques and evaluated through two studies
Intro(3/4)
6

7. Interaction Lab., Seoul National University of Science and Technology
■Two study (evaluated)
 First study
• All objects located in front of the user and were within arm-reach distance
 Second study
• In a larger virtual environment with distant objects and embedded the designed techniques into realistic
workflow
Intro(4/4)
7

8. Design space
Technique design
8

9. Interaction Lab., Seoul National University of Science and Technology
■Target manipulation process(1/2)
 Indicate
 Confirm
 Manipulate
 Release
Design space(1/4)
9

10. Interaction Lab., Seoul National University of Science and Technology
■Target manipulation process(2/2)
 Indicate
• Action of determining the target of interest with an input device
• Less effort and faster than manual input
 Confirm
• “pick up” and start manipulating the indicated target
• Not gaze-based confirming (dwell, …), use hand-based method (hand-held device) for robust control
 Manipulate
• Translation, rotation, and scaling
• Hand alone or gaze and hand input together
 Release
Design space(2/4)
10

11. Interaction Lab., Seoul National University of Science and Technology
■Design dimensions(1/2)
 Existing design dimensions
• Target properties and input techniques
 Three-dimensional design space
• Integration, coordination, and transition of gaze and hand input for the manipulate phase
• Integration : input mechanism(s) of gaze and hand has(have) been integrated into the manipulation
• Coordination : target will snap to the hand position or remain in its original place
• Transition : whether the transition between gaze and hand input is explicit(trigger) or implicit
Design space(3/4)
11

12. Interaction Lab., Seoul National University of Science and Technology
■Design dimensions(2/2)
 Synthesis of prior work
• Existing gaze-supported manipulation techniques
Design space(4/4)
12

13. Technique design
Evaluation
13

14. Interaction Lab., Seoul National University of Science and Technology
Technique design(1/4)
■Four techniques
 𝐺𝑎𝑧𝑒𝐺𝑟𝑎𝑏
 𝑅𝑒𝑚𝑜𝑡𝑒𝐻𝑎𝑛𝑑
 3𝐷𝑀𝑎𝑔𝑖𝑐𝐺𝑎𝑧𝑒
 𝐼𝑚𝑝𝑙𝑖𝑐𝑖𝑡𝐺𝑎𝑧𝑒

15. Interaction Lab., Seoul National University of Science and Technology
Technique design(2/4)
■𝐺𝑎𝑧𝑒𝐺𝑟𝑎𝑏
 Target snaps to the hand position once the selection is confirmed
 Direct manipulation using hand
 The gaze-grabbed object is located slightly above the virtual hand, to avoid occlusion
■𝑅𝑒𝑚𝑜𝑡𝑒𝐻𝑎𝑛𝑑
 User first points at it with eye gaze and the confirms the selection with a hand trigger
 Indirect manipulation using remote hand movement
 “gaze selects, hand manipulates”, 3D extension of existing approaches in 2D

16. Interaction Lab., Seoul National University of Science and Technology
■3𝐷𝑀𝑎𝑔𝑖𝑐𝐺𝑎𝑧𝑒
 Circular safe region (10° radius, invisible to user)
 If the gaze point is within safe region
• Only the hand can control the transformation of the object
 If the gaze point is outside safe region and if the hand movement distance exceeds a threshold
• The object snaps to the gaze point direction, that is gaze can move object
 Explicit command to switch from gaze input to manual input
■𝐼𝑚𝑝𝑙𝑖𝑐𝑖𝑡𝐺𝑎𝑧𝑒
 Same as the 3DMagicGaze
 Not any hand trigger
 Implicit transition
 Dynamically-resized safe region
Technique design(3/4)
16

17. Interaction Lab., Seoul National University of Science and Technology
Technique design(4/4)

18. Study 1 : Evaluation
Study 2 : Application
18

19. Interaction Lab., Seoul National University of Science and Technology
■Intro
 Primary working space
• All target of interest are located in front of the user and within arm-reach distance
Study 1 : Evaluation(1/14)

20. Interaction Lab., Seoul National University of Science and Technology
■Participants and apparatus
 12 university students (3 women, 9 men) between the age of 18 to 29
 Pico Neo 2 Eye (6 DoF) + Tobii eye-tracking
 C# in Unity3D
Study 1 : Evaluation(2/14)
20

21. Interaction Lab., Seoul National University of Science and Technology
■Task
 Transform a 3D model from its initial configuration to a new target pose
• Target location : randomly selected within 30° of angle
• Lateral distance : the angular distance between the start and target location
• Depth : the differences in the depth dimension
• The target position was to be expected by participants
• Objects can exist out of sight but within primary working space
Study 1 : Evaluation(3/14)
21

22. Interaction Lab., Seoul National University of Science and Technology
■Design and procedure
 4 x 3 x 2 experiment
• Technique(𝑅𝑒𝑚𝑜𝑡𝑒𝐻𝑎𝑛𝑑, 𝐺𝑎𝑧𝑒𝐺𝑟𝑎𝑏, 𝐼𝑚𝑝𝑙𝑖𝑐𝑖𝑡𝐺𝑎𝑧𝑒, and 3𝐷𝑀𝑎𝑔𝑖𝑐𝐺𝑎𝑧𝑒)
• Depth (0.05m, 0.10m, 0.15m)
• Lateral Distance(35° and 55°)
 Data : 1440
• Participants(12) x techniques (4) x depths (3) x lateral distances (2) x repetitions (5)
 The whole experiment lasted approximately 50 minutes in total
 After each session, we collected user feedback
Study 1 : Evaluation(4/14)
22

23. Interaction Lab., Seoul National University of Science and Technology
■Evaluation metrics(1/3)
 Performance measures
 Hand manipulation measures
 Subject measures
Study 1 : Evaluation(5/14)
23

24. Interaction Lab., Seoul National University of Science and Technology
■Evaluation metrics(2/3)
 Performance measures
• Manipulation time : the target is correctly placed with errors under the pre-determined threshold
• Coarse translation time : the time elapsed between the selection confirmation and the first time
• Re-position time : Manipulation time – Coarse translation time
Study 1 : Evaluation(6/14)
24
Manipulation time
Coarse translation time
Re-position
time
Confirm Target position
End (Release)
Start

25. Interaction Lab., Seoul National University of Science and Technology
■Evaluation metrics(3/3)
 Hand manipulation measures
• Hand movement distance : the accumulated distance that the hand has travelled during the process
• Hand rotation angles : the accumulated angle that the hand has rotated during the process
 Subject measures
• Arm fatigue, ease of use, required workload
Study 1 : Evaluation(7/14)
25
Borg CR10 A categorical rating (0-10)
Arm exertion/fatigue
Single Easement Questionnaire 7-point scales
Ease-of-use of the techniques
Raw NASA-TLX 7-point scales
The task load induced
Subject Ranking Participants’ overall preference

26. Interaction Lab., Seoul National University of Science and Technology
■Result(1/3)
 Discarded the outliers
• Deviated more than three standard deviations from the mean value (𝑚𝑒𝑎𝑛 ± 3𝑠𝑡𝑑.)
• In each condition (20 𝑡𝑟𝑖𝑎𝑙𝑠, 1.3%)
 The data is non-normally distributed
• Shapiro-Wilk test
• Pre-processing through Aligned Rank Transform (ART)
 Repeated-measures ANOVAs (RM-ANOVA)
Study 1 : Evaluation(8/14)
26

27. Interaction Lab., Seoul National University of Science and Technology
■Result(2/3)
 Performance measures and hand manipulation measures
• RM-ANOVA (Repeated Measures ANOVA)
Study 1 : Evaluation(9/14)
27
Manipulation time
𝑇𝑒𝑐ℎ𝑛𝑖𝑞𝑢𝑒
(𝐹3,253 = 4.141, 𝑝 = .007)
𝐿𝑎𝑡𝑒𝑟𝑎𝑙 𝑑𝑖𝑠𝑡𝑎𝑛𝑐𝑒
(𝐹1,253 = 5.414, 𝑝 = .021)
𝐷𝑒𝑝𝑡ℎ
(𝐹2,253 = 0.186, 𝑝 = .831)
Coarse manipulation time
𝑇𝑒𝑐ℎ𝑛𝑖𝑞𝑢𝑒
(𝐹3,253 = 3.084, 𝑝 = .030)
𝐿𝑎𝑡𝑒𝑟𝑎𝑙 𝑑𝑖𝑠𝑡𝑎𝑛𝑐𝑒
(𝐹1,253 = 25.024, 𝑝 = .001)
𝐷𝑒𝑝𝑡ℎ
(𝐹2,253 = 0.452, 𝑝 = .637)
Re-position time
𝑇𝑒𝑐ℎ𝑛𝑖𝑞𝑢𝑒
(𝐹3,253 = 3.861, 𝑝 = .010)
𝐿𝑎𝑡𝑒𝑟𝑎𝑙 𝑑𝑖𝑠𝑡𝑎𝑛𝑐𝑒
(𝐹1,253 = 1.377, 𝑝 = .242)
𝐷𝑒𝑝𝑡ℎ
(𝐹2,253 = 0.452, 𝑝 = .637)
Movement distance
𝑇𝑒𝑐ℎ𝑛𝑖𝑞𝑢𝑒
(𝐹3,253 = 13.559, 𝑝 = .001)
𝐿𝑎𝑡𝑒𝑟𝑎𝑙 𝑑𝑖𝑠𝑡𝑎𝑛𝑐𝑒
(𝐹1,253 = 55.681, 𝑝 = .001)
𝐷𝑒𝑝𝑡ℎ
(𝐹2,253 = 0.126, 𝑝 = .881)
Movement rotation
𝑇𝑒𝑐ℎ𝑛𝑖𝑞𝑢𝑒
(𝐹3,253 = 15.663, 𝑝 = .001)
𝐿𝑎𝑡𝑒𝑟𝑎𝑙 𝑑𝑖𝑠𝑡𝑎𝑛𝑐𝑒
(𝐹1,253 = 26.569, 𝑝 = .001)
𝐷𝑒𝑝𝑡ℎ
(𝐹2,253 = 0.924, 𝑝 = .398)

28. Interaction Lab., Seoul National University of Science and Technology
■Result(3/3)
Study 1 : Evaluation(10/14)
28

29. Interaction Lab., Seoul National University of Science and Technology
■Discussion
 Hand-Only (𝑅𝑒𝑚𝑜𝑡𝑒𝐻𝑎𝑛𝑑) vs. Eye-Hand manipulation (3𝐷𝑀𝑎𝑔𝑖𝑐𝐺𝑎𝑧𝑒, 𝐼𝑚𝑝𝑙𝑖𝑐𝑖𝑡𝐺𝑎𝑧𝑒)
 Direct vs. Remote hand mappings
 Implicit vs. Explicit eye-hand transitions
 Effect of distance and depth
Study 1 : Evaluation(11/14)
29

30. Interaction Lab., Seoul National University of Science and Technology
■Hand-Only vs. Eye-Hand manipulation
 Not significant performance differences
• 𝑅𝑒𝑚𝑜𝑡𝑒𝐻𝑎𝑛𝑑 vs. 3𝐷𝑀𝑎𝑔𝑖𝑐𝐺𝑎𝑧𝑒, 𝐼𝑚𝑝𝑙𝑖𝑐𝑖𝑡𝐺𝑎𝑧𝑒
 Hand-Only
• 𝑅𝑒𝑚𝑜𝑡𝑒𝐻𝑎𝑛𝑑 required more hand movement and rotation to active same manipulation
 Eye-hand manipulation
• Participants quickly learned/adapted to new input method Transition between gaze and hand
 Borg CR10 and NASA-TLX
• Not significant benefits of eye-hand transitions over hand-only techniques regarding arm fatigue
Study 1 : Evaluation(12/14)
30

31. Interaction Lab., Seoul National University of Science and Technology
■Direct vs. Remote hand mappings
 Substantial differences in performance measures and subjective feedback
• 𝐺𝑎𝑧𝑒𝐺𝑟𝑎𝑏 required a much longer time frame to re-position an object than 𝑅𝑒𝑚𝑜𝑡𝑒𝐻𝑎𝑛𝑑
• 𝐺𝑎𝑧𝑒𝐺𝑟𝑎𝑏 caused significantly higher perceived arm fatigue
• 𝐺𝑎𝑧𝑒𝐺𝑟𝑎𝑏 had the smallest hand rotation angles
• “Direct manipulation” techniques are imprecise in nature
Study 1 : Evaluation(13/14)
31

32. Interaction Lab., Seoul National University of Science and Technology
■Implicit vs. Explicit eye-hand transitions
 Similar empirical performance
 Implicit
• Less hand movement
 Explicit eye-hand transitions
• 3DMagicGaze needs more hand movement and rotation (Explicit transition)
■ Side effect of hand movement
■ Longer periods of time and rotate more to achieve the same task
Study 1 : Evaluation(14/14)
32

33. Study 2 : Application
Discussion and conclusion
33

34. Interaction Lab., Seoul National University of Science and Technology
■Intro
 Primary working space
• Large Virtual environment
Study 2 : Application(1/7)

35. Interaction Lab., Seoul National University of Science and Technology
■Participants and apparatus
 8 university students (3 women, 5 men) with previous experience in 3D Modeling
 Their ages were between 21 – 29 years (𝑚𝑒𝑎𝑛 = 24.4)
■Procedure
 Approximately 60 minutes in total
 hand-only vs. hand-eye, direct vs. remote mappings, and implicit vs. explicit transitions
Study 2 : Application(2/7)
35

36. Interaction Lab., Seoul National University of Science and Technology
■Interaction scenario
Study 2 : Application(3/7)
36

37. Interaction Lab., Seoul National University of Science and Technology
■Discussion
 Hand-Only vs. Eye-Hand manipulation
 Direct vs. Remote hand mappings
 Implicit vs. Explicit eye-hand transitions
 Gaze-Supported techniques vs. Virtual hand
Study 2 : Application(4/7)
37

38. Interaction Lab., Seoul National University of Science and Technology
■Hand-Only vs. Eye-Hand manipulation
 Eye-Hand manipulation
• Unlike study 1, in larger environment, participants preferred eye-hand manipulation
• Eye-Hand manipulation became less useful for close and large objects
• Eye-Hand manipulation might occlude the user’s line-of-sight
 Hand-Only
• Some participants found hand-only manipulation to be more manageable
Study 2 : Application(5/7)
38

39. Interaction Lab., Seoul National University of Science and Technology
■Direct vs. Remote hand mappings
 Direct
• Remote transformation was efficient in transporting distant targets but can be cumbersome for close ones
• If the object under manipulation is quite large, it is difficult to transform
• Mini-map or semi-transparent
 Remote hand mappings
• Participants remain same standing position and transferred the object remotely
Study 2 : Application(6/7)
39

40. Interaction Lab., Seoul National University of Science and Technology
■Implicit vs. Explicit eye-hand transitions
 Implicit
• Hand movement to confirm the gaze action was somewhat redundant
 Explicit eye-hand transitions
• Robustness
• The rapid eye movement would not frequently bring the object to the user’s facing direction
■ Dynamically-resized safe region is not able to handle rapid in 𝐼𝑚𝑝𝑙𝑖𝑐𝑖𝑡𝐺𝑎𝑧𝑒
Study 2 : Application(7/7)
40

41. Discussion and conclusion
41

42. Interaction Lab., Seoul National University of Science and Technology
■Design implications(1/2)
 Embedding gaze input can be useful for a larger environment with distant objects
 Hand-Eye coordination strategies should be used in appropriate scenario
 Minimizing the duration of using direct-mapping and use indirect-mapping techniques
(𝑅𝑒𝑚𝑜𝑡𝑒𝐻𝑎𝑛𝑑) which allow users to rest their arms under a comfortable position
Discussion and conclusion(1/3)

43. Interaction Lab., Seoul National University of Science and Technology
■Design implications(2/2)
 Providing an implicit transition between gaze and hand input (𝐼𝑚𝑝𝑙𝑖𝑐𝑖𝑡𝐺𝑎𝑧𝑒) can enable the
smooth and concurrent transformation
 Adding a small widget to indicate which input modality is currently taking control of the
manipulation for explicit transition like 3𝐷𝑀𝑎𝑔𝑖𝑐𝐺𝑎𝑧𝑒
Discussion and conclusion(2/3)

44. Interaction Lab., Seoul National University of Science and Technology
■Limitations and future work
 Limitations
• Not embed techniques that enable non-linear mapping of hand input
• Not explore the long-term usage of gaze-supported manipulation techniques
• Not test the methods alongside more complex sculpturing and modeling tools/functions
 Future work
• Head gaze can be a cheaper solution than eye gaze for current VR systems
Discussion and conclusion(3/3)

45. Q&A
45