Touchless Circular Menus
•Download as PPTX, PDF•
1 like•846 views
A touchless menu system optimized for large displays, which enables users to make simple directional movements for selecting commands.
Recommended
Recommended
More Related Content
Similar to Touchless Circular Menus
Similar to Touchless Circular Menus (20)
More from Debaleena Chattopadhyay
More from Debaleena Chattopadhyay (10)
Recently uploaded
Recently uploaded (20)
Touchless Circular Menus
- 1. Touchless Circular Menus: Toward an Intuitive UI for Touchless Interactions with Large Displays Debaleena Chattopadhyay & Davide Bolchini AVI 2014 International Working Conference on Advanced Visual Interfaces Como (Italy) May 27-30, 2014
- 4. Touchless interaction with large displays still lacks a standardized user interface language for frequent user- operations, such as..
- 5. Command Selection Menu Marking Menu, 1994 FlowMenu, 2000 Toolglass, 1993 rapMenu, 2008 Grab, 2013 Finger-Count Menu, 2011
- 6. Purpose of the Work
- 7. To propose and validate a novel form of command selection language appropriate for touchless interaction with large displays.
- 8. Design Space
- 9. Current Debate on UI “naturalness” Norman, D. A. (2010). Natural user interfaces are not natural. interactions, 17(3), 6-10. Wigdor, D., & Wixon, D. (2011). Brave NUI world: designing natural user interfaces for touch and gesture. Elsevier. O'hara, K., Harper, R., Mentis, H., Sellen, A., & Taylor, A. (2013). On the naturalness of touchless: putting the “interaction” back into NUI. ACM Transactions on Computer-Human Interaction (TOCHI), 20(1), 5. “You are the controller” –Microsoft® “Most gestures are neither natural nor easy to learn or remember.” –Norman Natural is ‘a design philosophy and a source of metrics enabling an iterative process to create a product.’ –Wigdor & Wixon “Naturalness is not something to be represented but is rather an ‘occasioned property of action’ [..].” –O’Hara et al.
- 10. Intuitiveness of an interface Prior knowledge Expertise Culture Sensorimotor Innate Specialized knowledge acquired with practice. Knowledge of people specific to culture. Knowledge acquired since childhood through interaction with the world. Reflexes and intrinsic behavior. unconscious application of prior knowledge ∝ Hurtienne, J., & Israel, J. H., 2007. Image schemas and their metaphorical extensions: intuitive patterns for tangible interaction. Proc. TEI,127-134, ACM. Intuitive Interaction Framework
- 11. How can we use sensorimotor knowledge to inform the design of intuitive touchless selection mechanisms?
- 12. Device – less interaction Device – based interaction Menu Invocation Mouse click, pen down, or touching surface ? Menu-Selection Delimiter Breaking contact with the interface Menu Shape ? Linear, radial ? Hand posture Hand posture Linear, radial
- 13. What successful features of device-based menus can be "ported" to the touchless environment?
- 14. Prior sensorimotor knowledge Some successful features of device- based menus Touchless Circular Menus (TCM) + Directional strokes in mid-air Menu-invocation: Reaching a region-of- interest. Menu-selection: Crossing Shape: Radial
- 15. Touchless circular menus relieve users from both recalling a precise vocabulary of hand-postures and strictly complying with them.
- 16. Demo
- 17. To select a menu option, the user makes a directional stroke. To give feedback, when successfully selected, the menu- option changes color.
- 18. To mitigate accidental invocation, TCM appear opposite to the user’s direction of approach. To cancel the invoked menu, the user may continue in her direction of movement, or move away from the menu.
- 19. To invoke the menu, the user must reach the region-of- interest (ROI) of the target.
- 20. Hierarchical TCM: To select a command, the user first selects a menu option, and then accesses the submenu by continuing her trajectory.
- 21. The Design
- 22. To trigger the contextual menu, a user must cross the region-of-interest (ROI) of a display object. The ROI can be of any symmetrical shape around the center of the target, with its size directly proportional to the technique’s sensitivity. Menu Invocation
- 23. To select a command after triggering the menu, users cross it using a stroke in the command’s direction. Until the crossing happens, users can cancel TCM by moving in any direction away from the triggered menu. To allow easy escape routes, we designed the structure of TCM as a semicircular array of options appearing at the top-left or the bottom-right corner of the target. Menu Selection
- 24. As users approach the menu, to give them orientation, a trace is drawn connecting the target and the users’ hand position. To improve users’ pointing performance, the menu options would increase in amplitude as users approached them. To provide further feedback, menu options changed color when selected. Menu Selection
- 25. Currently, the menu design scales up to two levels (5 x 5), with users performing continuous strokes. To operate the submenu, users change their track and cross another command. Due to the lack of precision and control of freehand movements, TCM require users to make inflections in their continuing trajectories, and thereby avoid accidental command- selections. Hierarchical TCM
- 26. Evaluation
- 27. Experiments How effectiveness and efficiency of TCM is affected by their triggering locations on the visual interface? 1 Vs. Touchless circular menus Linear menus using a grab gesture 2 Participants (N = 15) were sitting away from a large display.
- 28. Experimental Design Sample Demographics N = 15 All right-handed 4/15 females 8/15 had prior familiarity with touchless gestures 11/15 were below 30 years of age Time/participant: 1 hour 1Experiment 9 trials X 7 blocks X 15 participants = 945 trials Experiment 2 6 trials X 7 blocks X 15 participants = 630 trials
- 29. Apparatus Display— Built by Fakespace, eight 50” projection cubes, each with a resolution of 1600 X 1200 pixels. 160” wide and 60” high display Total resolution: 15.3 M pixels. Motion Tracking Sensor— Kinect for Windows™ UITS Research Technologies Advanced Visualization Lab
- 30. Outcome Measures Efficiency: Time on task | Successful Trigger Rate Effectiveness: Error Rate User Satisfaction: SUS | NASA TLX
- 31. 1Experiment Hypotheses H1: Triggering location will affect the efficiency of TCM. H2: Triggering location will affect the effectiveness of TCM
- 32. Triggering location of TCM significantly affected selection time and successful trigger rate. “I felt I had to rush to select the menu option” “I was surprised that I could do so well.” 1Experiment Results
- 33. Arm posture affects users’ control on their hand movements and the required effort. 1Experiment 2D plane locations relative to the user’s body and arm configuration affects the consumed endurance in touchless interactions Hincapié-Ramos, J. D., Guo, X., Moghadasian, P., & Irani, P. (2014). Consumed Endurance: A metric to quantify arm fatigue of mid-air interactions. Proc. CHI, 1063-1072, ACM.
- 34. Menu triggering locations will significantly affect the user experience of contextual touchless menus for large displays. Arm posture affects users’ control on their hand movements and the required effort. Contextual touchless menus for large displays will have different menu- triggering locations. 1Experiment
- 35. Hypotheses H3: We predicted TCM would be more efficient than linear menus. H4: We hypothesized that TCM would be easier to use than linear menus. Experiment 2
- 36. Experiment 2 Compared with linear menus, users were more efficient with TCM, and perceived lower overall workload. But TCM were less effective than linear menus. “This is how I envision using touchless gestures.” Results “It was a lot of effort.”
- 37. Experiment 2 Compared with linear menus using grab gestures, participants using TCM were twice faster in selecting commands and perceived lower workload. However, TCM caused 3% more errors than linear menus.
- 38. Experiment 2Why TCM were less effective? 1st gesture registration to invoke the menu 2nd gesture registration to select the menu Gesture relaxation
- 39. Experiment 2Why TCM were less effective? 1st gesture registration to invoke the menu 2nd gesture registration to select the menu No gesture relaxation
- 40. Future Work
- 41. Minimize errors by constraining users’ freehand movements after triggering the TCM. Investigate bimanual gestures to operate menu levels in hierarchical TCM.
- 42. Takeaways Menu triggering locations significantly affect the user experience of contextual touchless menus for large displays. Simple directional movements for selecting commands is more efficient and causes lower workload than (hand) posture-based techniques. TCM relieve users from learning posture-based commands, and shift the interaction complexity from users’ input to the visual interface. Thank you. We thank the study participants and UITS AVL. This research is partially supported by an IUPUI Research Support Funds Grant. 1 2 3 Efficient gesture relaxation techniques may have the potential to increase effectiveness of touchless interactions. 4
Editor's Notes
- While interacting with large, distant displays…Touchless interaction frees users from employing any intermediate technology and almost has a universal appeal. Large-display touchless interaction is being increasingly used in a number of usage contexts. Public displays, where users interact for a brief amount of time and may not spend the time and effort to connect a hand-held device with the display. Sterile operating rooms, where surgeons may need to browse medical images and cannot touch devices. Interactive TVs, where users can use touchless interactions to browse multimedia sporadically or access their favorite commands. Given this range of different scenarios, different stakeholders, and varied user expertise, it is a challenge to design an optimal touchless user experience for interacting with large displays.
- However, touchless interaction is still in its infancy.
- Specifically touchless interactions with large displays lacks a standardized user interface language for frequent user-operations, such as..
- Command selection. Whereas an extensive body of works investigated optimal menu designs for mouse, pen-input, or multitouch surfaces, very few have looked into touchless command-selection techniques …..especially for large displays. Recent solutions that appeared in research venues require users to comply strictly with system-defined poses, such as pinching with fingers, or making different finger combinations. These require users to recall a vocabulary of gestures, and in-lab user-studies have reported their high mental and physical demand. On product platform, recent solutions include touchless menus using grab gestures ..closing and opening the hand…. Despite the potential of such ‘smart interactions’, experts have commented on such products’ low user satisfaction.
- The purpose of our work
- Was to propose and validate a novel form of command selection language appropriate for touchless interaction with large displays.
- Given the variety of contexts, stakeholders, and user expertise, while designing the touchless menu, we considered several aspects.. For example, the claimed naturalness of this interaction modality, As well as its differences form other input modalities, such as pen or touch.
- Ironically, when you are designing touchless interactions, the word “natural” comes both as your ally and your adversary. As markerless motion tracking became mainstream, users became the controller. Yet soon, gestures, lacking feedback and learnability, were critiqued to be neither natural, nor easy to learn or remember. Natural, was then described as a design philosophy to enable the iterative creation of a product. And naturalness was explained as not something to be represented by the mechanics of a touchless gesture, but rather experienced as a property of the action taking place.
- To operationalize the ‘naturalness’ construct, we used the intuitive interaction framework. This framework suggests that intuitiveness………..often the twin term of naturalness….. is directly proportional to the unconscious application of prior knowledge. Now prior knowledge can be classified as expertise, culture, sensorimotor or innate. Expertise level of knowledge is specialized knowledge acquired with practice. In the Culture level, we’d find knowledge of people specific to culture. Sensorimotor level of knowledge is acquired since childhood through interaction with the world. Finally innate knowledge is developed in a prenatal state, such as reflexes, or intrinsic behavior. One important thing to note here… is that …..The higher the degree of specialization of knowledge, the smaller would be the potential number of users applying that knowledge unconsciously. Touchless interaction with large displays appeals to a variety of domains where users would have different expertise and cultural backgrounds. Moreover, given the lack of users’ control in engaging or disengaging innate knowledge, our approach uses sensorimotor knowledge to inform the design of intuitive touchless command selection techniques.
- But then the question stands as…. how can we use sensorimotor knowledge to inform the design of intuitive touchless selection mechanisms?
- Specifically, how sensorimotor knowledge can inform the design of different features of a touchless menu? For example, how would we invoke a touchless menu. How would we end a touchless menu selection? What should be an optimal shape of the touchless menu? Until now, touchless command-selection techniques have been an extension of what has proven efficient for mouse-based, pen-based, or multitouch interfaces. However, touchless interaction has a very distinctive feature. It is device less. In existing touchless menus, user’s hand postures have strictly replaced the mechanics of other input devices. This trend may lead us to a local maxima where we’d fail to explore the potential of touchless interactions beyond the device-based paradigm.
- In designing our touchless menu, we wanted to port some successful features of ‘device-based’ menus, but not strictly emulate them.
- The outcome of our design exercise was touchless circular menus… Building upon prior sensorimotor knowledge of making directional strokes in mid-air… And some successful features of device-based menus, such as reaching a region-of-interest for menu-invocation….crossing using directional strokes for menu selection….and a radial shape
- A touchless circular menu relieves users from both recalling a precise vocabulary of hand-postures and strictly complying with them.
- Now let us see how the menu works..
- To select a menu option, the user makes a directional stroke. To give feedback, when successfully selected, the menu option changes color.
- To mitigate accidental invocation, TCM appear opposite to the user’s direction of approach. To cancel the invoked menu, the user may continue in her direction of movement, or move away from the menu.
- To invoke the menu, the user must reach the region-of-interest of the target.
- In hierarchical TCM, to select a command, the user first selects a menu option, and then she accesses the submenu by continuing her trajectory.
- Let’s now dig a bit deeper into the design rationale for the touchless circular menus.
- As we saw, to trigger the contextual menu, a user must cross the region-of-interest (ROI) of a display object. The ROI can be of any symmetrical shape around the center of the target, with its size directly proportional to the technique’s sensitivity.
- To select a command after triggering the menu, users cross it using a stroke in the command’s direction. Until the crossing happens, users can cancel TCM by moving in any direction away from the triggered menu. To allow easy escape routes, we designed the structure of TCM as a semicircular array of options appearing at the top-left or the bottom-right corner of the target.
- As users approach the menu, to give them orientation, a trace is drawn connecting the target and the users’ hand position. To improve users’ pointing performance, the menu options increase in amplitude as users approached them. To provide further feedback, menu options changed color when selected by crossing.
- Currently, our menu design scales up to two levels …5 x 5……with users performing continuous strokes. Users first select a root menu. Then to operate the submenu, users change their track and cross another command. In device-based hierarchical menus, submenus appear in the same direction of the root menu. Due to the lack of precision and control of freehand movements, TCM require users to make inflections in their continuing trajectories, and thereby avoid accidental command-selections.
- We evaluated single-level TCM in two experiments.
- TCM is a contextual menu for large displays. So in our first experiment, we investigated how effectiveness and efficiency of TCM is affected by their triggering locations? In the second experiment, we compared TCM with linear menus using grab gestures. For the linear menus, users invoked the menu with a grab gesture, and selected a menu –option using another grab gesture. A grab gesture was defined as an open hand…closing and opening again. For both these experiments, we had 15 participants, who were sitting away from a large display.
- In experiment 1, we had 9 trials and 7 blocks of repetition. In experiment 2, we had 6 trials and 7 blocks of repetition. For both the experiments, we used the same sample of participants. They were all right-handed. 4 participants were females. 8 out of 15 participants had prior familiarity with touchless gestures. And 11 out of 15 participants were below 30 years of age.
- As apparatus for our experiment we used a large display and a motion tracking sensor. The large display was built by Fakespace labs…It is 160 inch wide and 60 inch in height..with a total resolution of 15.3 M pixels. We would like to thank the IU Advanced visualization lab, a UITS Research Technologies division for the use of their facilities. As the tracking sensor, we used a Kinect for windows.
- For both of our experiments, we measured efficiency as the time on task and successful trigger rate. Successful triggers were When users triggered a menu, and continued to select an option from the triggered menu. If the menus was dismissed before any option was selected, it was considered an unsuccessful trigger. We measured effectiveness as error rate….. To measure user satisfaction, users were asked to complete the system usability scale and the NASA task load index.
- For experiment 1, we had two hypotheses: we predicted that triggering locations of the TCM will affect the performance time and the error rate.
- In our first experiment, we found that Triggering location of TCM significantly affected selection time and successful trigger rate. However, we did not find a significant effect of the triggering locations on the effectiveness of TCM. Users had mixed reaction about the menu. While one said “I felt I had to rush to select the menu option.”, another was surprised that he could do so well.
- Our findings suggest that arm posture affects users’ control on their hand movements and the required effort. This is completely in line with another study that was published last month in CHI. In this study, the authors found that “2D plane locations relative to the user’s body and arm configuration significantly affects the consumed endurance in touchless interactions.”
- Now Contextual touchless menus for large displays will always have different menu-triggering locations. Hence, Menu triggering locations will significantly affect the user experience of contextual touchless menus for large displays.
- For experiment 2, we predicted that TCM would be more efficient, and easier to use than linear menus.
- In our second experiment, we found that TCM were more efficient and cause lower workload than linear menus with grab gestures. However, TCM were less effective than linear menus. Again, users had mixed reaction about the linear menu. While a 20-something participant said “This is how I envision using touchless gestures.”…..for a 50-something participant …“It was a lot of effort.”
- Our findings suggest that Compared with linear menus using grab gestures, participants using TCM were twice faster in selecting commands and perceived lower workload. However, TCM caused 3% more errors than linear menus.
- So then the question was: “Why TCM were less effective?” To figure that out we can take a closer look at the mechanics of both linear menus using grab gestures, and TCM. For linear menus, one grab gesture was registered to invoke the menu, a second grab gesture was registered to select a menu option. The grab gesture was defined as an open hand..closing and opening again. In between the two gesture registrations, users could move their hand freely without affecting the on-going interaction. That is, user’s cursor would move, but not inadvertently select a menu option. Hence the gesture relaxation phase made the linear menus more effective.
- However, for TCM, after the menu was triggered, users could inadvertently move their hand and select a wrong command or..dismiss the menu. Unlike linear menus, TCM required users to strictly constrain their freehand movements after triggering the menu.
- As our future work, we plan to…
- To Minimize errors by constraining users’ freehand movements after triggering the TCM. And to Investigate bimanual gestures to operate menu levels in hierarchical TCM…for which we also plan to conduct empirical evaluations.
- So we proposed and evaluated TCM. Now The final takeaways. TCM relieve users from learning posture-based commands, and shift the interaction complexity from users’ input to the visual interface. We found that Menu triggering locations significantly affect the user experience of contextual touchless menus for large displays. Our results further suggest that Simple directional movements for selecting commands is more efficient and causes lower workload than (hand) posture-based techniques. Finally, our findings indicate that efficient gesture relaxation techniques may have the potential to increase effectiveness of touchless interactions.
- As a delimiter, dynamic gestures would be more efficient than static poses, as users would not have to halt-and execute a pose, but fluidly end the selection.