This document discusses new modes of interaction beyond the traditional mouse and keyboard as input methods. It presents several examples of new technologies that use touch, gestures, motion tracking and eye tracking as alternative forms of input. The author argues that these new input methods require new approaches to interface design to ensure perceptible affordances that allow users to understand how to control devices intuitively. The author proposes research to study how users visually perceive and make sense of interfaces for new input methods.

1. Beyond Mouse and Keyboard:
Post-WIMP and Novel Forms of
Interaction
Jacques Chueke
London, UK, May 2011
George Buchanan
(1st Supervisor)
Lecturer, Centre for HCI Design
Stephanie Wilson
(2nd Supervisor)
Lecturer, Centre for HCI Design
Master in Design, PUC-Rio, RJ, Brazil
PhD Researcher at the Centre for HCI Design
School of Informatics, City University London 1

2. Things are changing…
iPad: 1 year old growing among touch screens and print
2

3. Things are changing…
iPad for cats???
3

4. Overview
• The introduction of novel
hardware for computing and
gaming during the last decade
is changing the way we control
everyday devices.
• It provides NUI control methods,
such as haptic (e.g. iPhone,
iPad, MS Surface), gesture-
based and voice (e.g.
Nintendo Wii, Microsoft Xbox
360 console gaming with Kinect
sensor) and eye tracking
MIT Media Lab: DepthJS – 2011
interactions (e. g. Tobii P-10).
• One specific impact this has
had is on the user’s control of
such devices.
4

5. It’s not supposed to be like this…
Gmail Motion, April 2011
5

6. It’s not supposed to be like this…
Gmail Motion, April 2011
6

7. New Modes of Interaction
• PrimeSense / MS Kinect: Swim Browser
Prime sense browser competition winner: Stolarsky 'SwimBrowser’ – 2011
7

8. New Modes of Interaction
• KinVi: Kinect Virtual Interface
Prime sense browser competition 2n place: Windows Control with gestures - 2011
8

9. New Modes of Interaction
• eviGroup Paddle Pro
Front-facing webcam to track head movements for cursor control – 2011
9

10. New Modes of Interaction
• Hitachi: Hitachi's Gesture Remote Control TV Prototype
CES 2009: Hitachi's Gesture Remote Control TV Prototype
10

11. Problem Statement
• New command vocabularies
NEW CONTROL METHODS (INPUT)
have emerged and users do
not know how to access or NEW INTERFACES (OUTPUT)
activate them.
• This is a timely moment to
research how people make OLD OUTPUT X NEW INPUT
sense of these technologies
NEW OUTPUT X OLD INPUT
for CONTROL whilst some
Post-WIMP interfaces do not NEW OUTPUT X NEW INPUT
display appropriate visual
cues for NUI interactions.
WIMP-GUI (DESKTOP) X NUI (PHYSICAL)
POST-WIMP X MOUSE/KEYBOARD
POST-WIMP X NUI (PHYSICAL)
11

12. Case 1: Microsoft Surface
Media Player, Microsoft Surface. Nov, 2010
POST-WIMP + NUI (PHYSICAL) 12

13. Case 2: Windows 8 Metro Dashboard
• The new Windows 8 with similar features as used in Windows Phone and Xbox 360 Dashboard.
POST-WIMP + NUI (PHYSICAL) x Metro Dashboard: Windows 8 Start Screen. Feb, 2012
MOUSE/KEYBOARD 13

14. Example: GNOME 3 - desktop environment for GNU/Linux
and UNIX-type operating systems.
GNOME 3 hotcorners and responsive interface, 2012
POST-WIMP + NUI (PHYSICAL) x
MOUSE/KEYBOARD 14

15. Case 3: Tobii P-10 Eye Tracker: Gaze for Control
15

16. Case 3: Tobii P-10 Eye Tracker: Gaze for Control
Tobii P-10 at the SmartLab (UEL), Oct. 2010
16

17. Case 3: Tobii P-10 Eye Tracker: Gaze for Control
• Assistive Technology: Tobii P-10 at the SmartLab (UEL).
Tobii P-10 equipment, Oct. 2010
WIMP-GUI (DESKTOP) X NUI (PHYSICAL) 17

18. Case 3: Tobii P-10 Eye Tracker: Gaze for Control
Mouse configuration pop up for Windows Control, Tobii P-10
18

19. Case 4: Tobii LeNovo: Gaze for Control
• Tobii Lenovo / PCEye / Acuity
Tobii LeNovo, Jun 2011
WIMP-GUI (DESKTOP) X NUI (PHYSICAL) 19

20. New Visual Cues/Feedback
UI affordances are
Just-in-time chrome shown on tap. Applying
Can be triggered by the principle of
touch or proximity – scaffolding will lead you
hover effect (pg. 153) to far more successful
multi-touch and gesture
UI’s. (pg. 154)
Tethers indicate
that a size The marking
constraint (MS menu system
Surface) has teaches users to
been reached on make pen-based
an item being gestures (pg. 150)
scaled (pg. 91)
Wigdor, D. Wixton, D. Brave NUI World, 2011
20

21. Methodology
An empirical study with Eye Tracking (Tobii x60) was conducted to test the
Perceptible Affordances of Drag and Drop interactions within a iGoogle
Personal Web Portal. This study served to create a protocol for analysis which
focuses on the very first 10 seconds of a participant scrutinizing the screen while
trying to respond specific questions.
Participant, 54, Beginner Expertise Participant, 22, Advanced Expertise
Gazeplot Comparison: Beginner x Advanced 21

22. Quantitative Data Analysis: Gazeplot
Resulting gaze plot from seven participants during the first 10 seconds after question 01.
22

23. Quantitative Data Analysis: Heatmap
Resulting heat map from seven participants during the first 10 seconds after question 01.
23

24. Quantitative Data Analysis: Gazeplot
Resulting heat map from seven participants during the first 10 seconds after question 02.
24

25. Quantitative Data Analysis: Heatmap
Resulting heat map from seven participants during the first 10 seconds after question 02.
25

26. Quantitative Data Analysis: Gazeplot
Resulting heat map from seven participants during the first 10 seconds after question 03.
26

27. Quantitative Data Analysis: Heatmap
Resulting heat map from seven participants during the first 10 seconds after question 03.
27

28. Quantitative Data Analysis: Gazeplot
Resulting heat map from seven participants during the first 10 seconds after question 04.
28

29. Quantitative Data Analysis: Heatmap
Resulting heat map from seven participants during the first 10 seconds after question 04.
29

30. Model 1: Perceptible Affordances
• According to Nielsen (2008):
"Affordance" means what you can
do to an object. For example, a
checkbox affords turning on and off,
and a slider affords moving up or
down.
• "Perceived Affordances" are
actions you understand just by
looking at the object, before you
start using it (or feeling it, if it's a
physical device rather than an on-
screen UI element).
• In Gaver’s (1991) words, “…Perceptible Affordances are
inter-referential: the attributes of the object relevant for
action are available for perception. What is perceived is
what is acted upon.” 30

31. Perceptible Affordances in Post-WIMP
Post-WIMP GUI [INTERFACE LAYER]
OUTPUT
- LESS SYMBOLIC
- MORE INTUITIVE
(USER) PERCEPTIBLE AFFORDANCE
- MORE REACTIVE
(COMPUTER)
INPUT
NUI [MODE OF INTERACTION LAYER]
31

32. Model 2: Norman’s Theory of Action
Execution Cycle
Execution Specification of Formulation of
Actions Intention
Sequence
Interaction
Perception Interpretation Evaluation
Evaluation Cycle
Preece et al (2009: 121) quoting Norman, (1986)
32

33. Research Question 1: ACTIVATION
Post-WIMP GUI [INTERFACE LAYER]
Evaluation Cycle OUTPUT
PERCEPTIBLE AFFORDANCE
Execution Cycle INPUT
NUI [MODE OF INTERACTION LAYER]
33

34. Conclusions and Future Work
• By developing a methodology for an empirical study, which focuses on
observation prior to any interaction, we are willing to identify what elements
people will focus on NUI screens.
• I believe both Norman’s theory of Action and Perceptible Affordances theories
can be combined with the more recent Piaget’s theory of INRC and the
Scaffolding concept; and used on my protocol for analysis.
• A prototype with Post-WIMP characteristics and NUI mode of interaction will be
built in order to to understand how users visually scan such interfaces to obtain
the gist of its interactive potential.
• Quantitative (Eye Tracking) and Qualitative (Verbalizations) data will be
combined to produce conclusions about what kind of information can be obtained
with the protocol – and how can this data be adapted to indicate better design
interactions with NUI systems.
34

35. Thank you for your attention!
Jacques Chueke
Jacques.chueke.1@city.ac.uk
35

36. Bibliography
Beaudouin-Lafon, M. (November 2000). "Instrumental Interaction: An Interaction Model for Designing Post-WIMP
User Interfaces". CHI '00: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. The
Hague, The Netherlands: ACM Press. pp. 446–453. doi:10.1145/332040.332473. ISBN 1-58113-216-6.
http://www.daimi.au.dk/CPnets/CPN2000/download/chi2000.pdf.
Breeze, James. Eye Tracking: Best Way to Test Rich App Usability. UX Magazine, access on 25 November 2010.
(http://www.uxmag.com/technology/eye-tracking-the-best-way-to-test-rich-app-usability)
Buxton, W. (2001). Less is More (More or Less), in P. Denning (Ed.), The Invisible Future: The seamless
integration of technology in everyday life. New York: McGraw Hill, 145–179 ITU Internet Reports 2005: The
Internet of Things – Executive Summary.
Dam, A. (February 1997). "POST-WIMP User Interfaces". Communications of the ACM (ACM Press) 40 (2): pp.
63–67. doi:10.1145/253671.253708.
Dourish, P. Where the Action Is: The Foundations of Embodied Interaction. A Bradford Book: The MIT Press, USA,
2004.
Ehmke & Wilson, 2007. Identifying Web Usability Problems from Eye-Tracking Data. Published by the British
Computer Society. People and Computers XXI – HCI…but not the way we know it: Proceedings of HCI 2007.
Gaver, W. Technology Affordances. Copyright 1991 ACM 0-89791-383-3/91/0004/0079.
Gentner, D. and Nielsen, J. (April 1993). "The Anti-Mac Interface". Communications of the ACM (ACM Press) 39
(8): pp. 70–82. http://www.useit.com/papers/anti-mac.html.
Jacob, R. et al. (2008). "Reality-Based Interaction: A Framework for Post-WIMP Interfaces". CHI '08: Proceedings
of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems. Florence, Italy: ACM.
pp. 201–210. doi:http://doi.acm.org.ezproxy.lib.ucf.edu/10.1145/1357054.1357089. ISBN 978-1-60558-011-1. 36

37. Bibliography
McGrenere, J., Ho, W. (2000). Affordances: Clarifying and Evolving a Concept. Procs. of Graphic Interfaces 2000,
Montreal, May 2000.
McNaughton, J. Utilizing Emerging Multi-touch Table Designs. Technology Enhanced Learning Research Group -
Durham University. TR-TEL-10-01.
Nielsen, J. (April 1993). "Noncommand User Interfaces". Communications of the ACM (ACM Press) 36 (4): pp.
83–99. doi:10.1145/255950.153582. http://www.useit.com/papers/noncommand.html.
Norman, D. (1999). Affordance, Conventions and Design. In ACM Interactions, (May + June, 1999), 38-42.
Picard, R. Affective Computing. The MIT Press, Cambridge, Massachusetts. London, England, 1998.
PREECE, Jenny. SHARP, Helen. ROGERS, Yvonne. Interaction Design: Beyond Human-Computer Interaction
[2nd edition]. John Wiley & Sons, Ltd. West Sussex, UK, 2009.
Ramduny-Ellis, D.; Dix, A.; Hare, J.; Gill, S. Physicality: Towards a Less-GUI Interface (Preface). Procs. Third
International Workshop on Physicality. Cambridge, England, 2009.
Sorensen, M. Making a Case for Biological and Tangible Interfaces. Proceedings of the Third International
Workshop on Physicality. Cambridge, England, 2009.
Sternberg, R. Cognitive Psychology. Wadsworth, Cengage Learning. Belmont, CA, USA, 2009, 2006.
Vyas, D., Chisalita, C. Veer, G. Affordance in Interaction. ECCE '06 Proceedings of the 13th Eurpoean conference
on Cognitive ergonomics: trust and control in complex socio-technical systems. ACM New York, NY, USA ©2006
ISBN: 978-3-906509-23-5
WIGDOR, Deniel. WIXON, Dennis. Brave NUI World: designing natural user interfaces for touch and gesture. Morgan
Kauffman Publishers, USA, 2011. 37