Download to read offline
![• Gestures offer many opportunities, but…
“When developing a gesture interface, the
objective should not be *to make a gesture
interface*. A gesture interface is not universally
the best interface for any application. The
objective is to *develop a more efficient interface*
to a given application.”
[Nielsen et al., 2004]
Source: Nielsen, M., Störring, M., Moeslund, T.B., Granum, E. (2004). A procedure for developing intuitive and ergonomic gesture
interfaces for HCI. GW 2003. LNCS (LNAI) 2915, Springer 2004](https://image.slidesharecdn.com/vanderdonckt-francquichair-18may2021-210519093008/85/Evaluating-Gestural-Interaction-Models-Methods-and-Measures-33-320.jpg)
Uploaded byJean Vanderdonckt
Evaluating Gestural Interaction: Models, Methods, and Measures
The document discusses various methods for evaluating gestural user interfaces (UIs), including comparing a UI to a reference model, collecting evaluation data on usability criteria, and using standardized scales and metrics. Common dimensions for evaluation are goals, utility, usability, and factors like system acceptance, ease of use, and cost. Methods mentioned include observations, questionnaires, heuristic evaluations, and measuring task performance and preferences using standardized scales. Guidelines are provided for designing and assessing the usability of different gestures.
More Related Content
Similar to Evaluating Gestural Interaction: Models, Methods, and Measures
More from Jean Vanderdonckt
Evaluating Gestural Interaction: Models, Methods, and Measures
- 1. Lecture 4: EvaluatingGestural Interaction: Models, Methods, and Measures? Jean Vanderdonckt, UCLouvain, Vrije Universiteit Brussel, on-line, 18 May 2021 Pleinlaan 9, B-1050 Brussels
- 2. 2 • UI Evaluation= comparing a UI of an interactive system with respect to a reference model Interactive system to be evaluated Reference Model Compare Evaluation dimensions: goals, utility, usability Evaluation data: ergonomic criteria Data collection methods: observations, metrics, analyzes
- 3. 3 • Evaluation dimensions •System acceptance • Social acceptability • Practical acceptability • Ease of use • Utility • Usability • Cost • Compatibility • Robustness • ISO 9241, ISO 2100, ISO 25000 Square • Quality Management
- 4.
- 5. Sampling error Error #2 Discoverability Recognition rate Recognition time Learnability Memorability Workload (NASA-TLX: mental, physical,temporal, perfor- mance, effort, frustration) Error #3 Error TLX .9*** .3** ? ? Effective- ness Efficiency ? ? ? Gesture recognition performance Other performance properties
- 6. Sampling error Error #2 Discoverability Recognition rate Recognition time Learnability Memorability Workload (NASA-TLX: mental, physical,temporal, perfor- mance, effort, frustration) Error #3 Error TLX .9*** .3** ? ? Effective- ness Efficiency ? ? ? Gesture recognition performance Other performance properties Source: https://arxiv.org/abs/2001.09963
- 7. Sampling error Error #2 Discoverability Recognition rate Recognition time Learnability Memorability Workload (NASA-TLX: mental, physical,temporal, perfor- mance, effort, frustration) Error #3 Error TLX .9*** .3** ? ? Effective- ness Efficiency ? ? ? Gesture recognition performance Other performance properties Reproductibility = of the motions with task intent, the rate of gestures achieved as intended Relevance = of the tasks, the rate of gestures performed as intended for the task
- 8. Sampling error Error #2 Discoverability Recognition rate Recognition time Learnability Memorability Workload (NASA-TLX: mental, physical,temporal, perfor- mance, effort, frustration) Intuitiveness Naturality Social acceptability Adjustment quality Error #3 Error TLX … .9*** .3** .3* ? ? ? Effective- ness Efficiency Satisfaction ? ? ? ? ? Gesture recognition performance Other performance properties
- 9. Sampling error Error #2 Discoverability Recognition rate Recognition time Learnability Memorability Workload (NASA-TLX: mental, physical,temporal, perfor- mance, effort, frustration) Intuitiveness Naturality Social acceptability Adjustment quality Attractiveness Playability Pleasurability Error #3 Error TLX Error #m … .9*** .3** .3* ? ? ? ? ? ? Effective- ness Efficiency Satisfaction Gestural Experience Other UX properties ? ? ? ? ? ? ? End user’s gesture preference Gesture recognition performance Other performance properties Other preference properties ? ?
- 10. Sampling error Error #2 Discoverability Recognition rate Recognition time Learnability Memorability Workload (NASA-TLX: mental, physical,temporal, perfor- mance, effort, frustration) Intuitiveness Naturality Social acceptability Adjustment quality Attractiveness Playability Pleasurability Error #3 Error TLX Error #m Quality of Gestural user experience … GUESS (Gesture User Experience Special Scale) .9*** .3** .3* ? ? ? ? ? ? Effective- ness Efficiency Satisfaction Gestural Experience Other UX properties ? ? ? ? ? ? ? End user’s gesture preference Gesture recognition performance Other performance properties Other preference properties ? ? ? ?
- 11. End user’s preference Gesture recognition performance SuperfluousToo system- oriented Desired System oriented Neutral User- centered Too user- centered Any GUESS evaluation Confidence interval low medium high low medium high
- 12. Source: Evaluation ofGesture-Based In-Vehicle Interaction: User Experience and the Potential to Reduce Driver Distraction Lisa Graichen, Matthias Graichen, and Josef F. Krems
- 13. Source: Evaluation ofGesture-Based In-Vehicle Interaction: User Experience and the Potential to Reduce Driver Distraction Lisa Graichen, Matthias Graichen, and Josef F. Krems, Human Factors, V61, N5, 2019
- 14. 14 • Whitefield classificationof evaluation methods Analytical methods User reports Expert reports Observational methods Represented Real User Represented Real Application Source: A. Whitefield, F. Wilson, J. Dowell, A framework for human factors evaluation, Behaviour and Information Technology 10(1), January 1991, pp. 65-79
- 15. 15 • Whitefield classificationof evaluation methods Analytical methods Observational methods Represented Real User Represented Real Application Questionnaires Design tests Alternative designs Iterative evaluation Code static analysis Source: A. Whitefield, F. Wilson, J. Dowell, A framework for human factors evaluation, Behaviour and Information Technology 10(1), January 1991, pp. 65-79
- 16. 16 • Nielsen classificationof evaluation methods Source: https://www.nngroup.com/articles/which-ux-research-methods/
- 17. 17 • Nielsen classificationof evaluation methods Source: https://www.nngroup.com/articles/which-ux-research-methods/
- 18. 18 • Nielsen classificationof evaluation methods Preference Performance
- 19. 19 • Nielsen classificationof evaluation methods Source: https://www.nngroup.com/articles/which-ux-research-methods/
- 20. 20 • Nielsen classificationof evaluation methods Product Development Phase Strategize Execute Assess Goal: Inspire, explore and choose new directions and opportunities Inform and optimize designs in order to reduce risk and improve usability Measure product performance against itself or its competition Approach: Qualitative and Quantitative Mainly Qualitative (formative) Mainly Quantitative (summative) Typical methods: Field studies, diary studies, surveys, data mining, or analytics Card sorting, field studies, participatory design, paper prototype, and usability studies, desirability studies, customer emails Usability benchmarking, online assessments, surveys, A/B testing Source: https://www.nngroup.com/articles/which-ux-research-methods/
- 21. • Evaluation ofgesture UIs: guideline review Gesture Gesture name Suggested behavior Fixed or application- specific Hot point Notes Scratch-out Erase content Fixed Starting point Make the strokes as horizontal as possible, and draw at least three strokes. If the height of the gesture increases, the number of back and forth strokes also needs to increase. Triangle Insert Application-specific Starting point Draw the triangle in a single stroke, without lifting the pen. Make sure that the top of the triangle points upward. Square Action item Application-specific Starting point Draw the square starting at the upper left corner. Draw the square with a single stroke, without lifting the pen. Star Action item Application-specific Starting point Draw the star with exactly five points. Do this in a single stroke without lifting the pen. Check Check-off Application-specific Corner The upward stroke of the check must be two to four times as long as the smaller downward stroke. Curlicue Cut Fixed Starting point is distinguishing hot point Draw the curlicue at an angle, from lower left to upper right. Start the curlicue on the word that you intend to cut. Double- Curlicue Copy Fixed Starting point is distinguishing hot point Draw the double-curlicue at an angle, from the lower left to the upper right. Start the double-curlicue on the word that you intend to copy.
- 22. • Evaluation ofgesture UIs: heuristic evaluation • Heuristic: Visibility of System Status • Violation: Originally, the system does not tell the user whether a Kinect was connected to the system. So, if the system goes un- responsive, the user might wonder what is going wrong, “did I click a wrong button?” or “how to fix this problem?” • Severity: Major usability problem: important to fix, so should be given high priority • Improvement: I added a Kinect status viewer that is able to show any problem the Kinect may be experience. In addition, if the Kinect is working as expected, it will act as a mirror, showing the user’s gesture so the user knows exactly what his gesture looks like to the system • Heuristic: Recognition rather than recall • Violation: while the user is doing an exercise, if he wants to pause the exercise, he can simply say “pause”. However, there is no way to for new users to know that this command even exists. Also, it is not a good idea to rely on user’s memory to let them remember a particular feature. Source: Zhaochen Liu, Design a Natural User Interface for Gesture Recognition Application, University Of California, Berkeley, 5/1/2013
- 23. • Evaluation ofgesture UIs: guideline review 0 20 40 60 80 100 120 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Distribution des règles en pondéré Résultats globaux en linéaire 29% 71% Nombre de règles ergonomiques enfreintes Nombre de règles ergonomiques respectées Résultats globaux en pondéré 39% 61% Nombre de règles ergonomiques enfreintes Nombre de règles ergonomiques respectées Nielsen’s scale for usability problems: 0 – it is not a usability problem; 1 – it is a superficial problem, it must be fixed unless extra time is available; 2 – it is a minor usability problem; 3 – it is a major usability problem. It is important fix the problem; 4 – It is usability catastrophic.
- 24. • Evaluation ofgesture UIs: by questionnaire Post-experiment questionnaire: Rating vs. Ranking (absolute vs. relative) Source: Vatavu, R.D., Vogel, D., Casiez, G., Grisoni, L. (2011). Estimating the perceived difficulty of pen gestures. Proceedings of INTERACT'11. Springer-Verlag, Berlin, Heidelberg, 89-106
- 25. • Evaluation ofgesture UIs: by questionnaire Post-experiment questionnaire: Rating vs. Ranking (absolute vs. relative) Source: Jean Vanderdonckt, Sara Bouzit, Gaëlle Calvary, Denis Chêne, Exploring a Design Space of Graphical Adaptive Menus: Normal vs. Small Screens. ACM Trans. Interact. Intell. Syst. 10(1): 2:1-2:40 (2020
- 26. Source: Vatavu, R.D.,Vogel, D., Casiez, G., Grisoni, L. (2011). Estimating the perceived difficulty of pen gestures. Proceedings of INTERACT'11. Springer-Verlag, Berlin, Heidelberg, 89-106 • Evaluation of gesture UIs: by questionnaire Post-experiment questionnaire: Rating vs. Ranking (absolute vs. relative)
- 27. Source: Vatavu, R.D.,Vogel, D., Casiez, G., Grisoni, L. (2011). Estimating the perceived difficulty of pen gestures. Proceedings of INTERACT'11. Springer-Verlag, Berlin, Heidelberg, 89-106 • Evaluation of gesture difficulty Rule #1: Relative Difficulty Ranking Gesture A is likely to be perceived as more difficult to execute than gesture B if the production time of A is greater than that of B: time(A) > time(B) suggests Ranking(A) > Ranking(B) 93% accuracy
- 28. Source: Vatavu, R.D.,Vogel, D., Casiez, G., Grisoni, L. (2011). Estimating the perceived difficulty of pen gestures. Proceedings of INTERACT'11. Springer-Verlag, Berlin, Heidelberg, 89-106 • Evaluation of gesture difficulty Rule #2: Classifying Difficulty Rating Mapping from production time to difficulty class (very easy, easy, moderate, difficult, and very difficult) 83% accuracy
- 29. Source: Vatavu, R.D.,Vogel, D., Casiez, G., Grisoni, L. (2011). Estimating the perceived difficulty of pen gestures. Proceedings of INTERACT'11. Springer-Verlag, Berlin, Heidelberg, 89-106 • Evaluation of multi-touch gesture difficulty
- 30. • Evaluation ofgesture UIs: Gesture Usability Scale (GUS) Source: Manipulating 3D Content using Gestures in Design Review Scenarios October 2009, Daniel Wickeroth, Paul Benoelken, Ulrich Lang GUS = 67.2 SUS = 72.8 Diff = 7.1
- 31. • Evaluation ofgesture UIs: Gesture Usability Scale (GUS) Source: Manipulating 3D Content using Gestures in Design Review Scenarios October 2009, Daniel Wickeroth, Paul Benoelken, Ulrich Lang GUS = 67.2 SUS = 72.8 Diff = 7.1
- 32. • Evaluation ofgesture UIs: by metrics Source: Performance evaluation of gesture-based interaction between different age groups using Fitts’ Law Diana Carvalho, Maximino Bessa, Luís Magalhães, Eurico Carrapatoso, Proc. of Interaccion ‘2015 According to MacKenzie, three-dimensional movements may follow the same predictive model as a one directional Task (A=amplitude, W=width)
- 33. • Gestures offermany opportunities, but… “When developing a gesture interface, the objective should not be *to make a gesture interface*. A gesture interface is not universally the best interface for any application. The objective is to *develop a more efficient interface* to a given application.” [Nielsen et al., 2004] Source: Nielsen, M., Störring, M., Moeslund, T.B., Granum, E. (2004). A procedure for developing intuitive and ergonomic gesture interfaces for HCI. GW 2003. LNCS (LNAI) 2915, Springer 2004