Mobile Hci L Nigay
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Mobile Hci L Nigay

on

  • 1,710 views

Mobile Interaction (Laurence Nigay, Grenoble Informatics Laboratory (LIG), HCI Group, University of Grenoble 1)

Mobile Interaction (Laurence Nigay, Grenoble Informatics Laboratory (LIG), HCI Group, University of Grenoble 1)

Statistics

Views

Total Views
1,710
Views on SlideShare
1,710
Embed Views
0

Actions

Likes
0
Downloads
43
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Mobile Hci L Nigay Document Transcript

  • 1. Laurence Nigay - laurence.nigay@imag.fr Outline Mobile Human-Computer Interaction • Focus on Human-Computer Interaction Introductory Course • EHCI Group Laurence Nigay • Mobility: a broad concept University of Grenoble • Interaction with handheld devices Grenoble Informatics Laboratory - LIG Engineering Human-Computer Interaction Group - EHCI CNRS Summer School on Mobile Learning 4-9 July 2009, Brest, France EHCI group & Mobile HCI • EHCI Group • “EHCI” for “Engineering for Human-Computer Interaction” – Created in 1990 – 7 professors and assistant-professors – 15 PhD students – 4 research engineers and 1 post-doc EHCI group EHCI group: Research thematic • Software Engineering for HCI • EHCI group belongs to LIG (Computer Science Laboratory of Grenoble) Interaction • 165 academic researchers Computer science Supported by System Human • 240 doctoral and post-doctoral students contribution Human sciences • 65 technical and administrative staff – 24 autonomous research teams • Development of conceptual and technical tools based on HCI principles : Utility, usability, context – 4 principal themes • Establishing links between psychology- • Infrastructure (networks and data) ergonomics and software engineering • Software (foundations and design models) • Interaction (perception, action and dialog) • Knowledge (learning, agent models and web-ontologies) CNRS Summer School on Mobile Learning 4-9 July 2009, Brest, France
  • 2. Laurence Nigay - laurence.nigay@imag.fr EHCI group: Research axes EHCI group: Research axes • 4 complementary research axes • Models and processes for designing – Models and processes for designing HCI, HCI, formal testing formal testing – Interaction plasticity • Task Hierarchy: a model at runtime • Adaptation of HCI to interaction context – Post-WIMP interaction • Vision based finger tracking and two-handed interaction … • Mixed reality – Modality and multimodality EHCI group: Research axes EHCI group: Research axes • Interaction plasticity • Interaction plasticity • Adaptation of HCI to interaction context • Adaptation of HCI to interaction context • Context: interaction resources / user preferences / social context / location etc. EHCI group: Research axes EHCI group: Research axes • Interaction plasticity • Interaction plasticity Control panel Slides 2. PDA: Partial migration (only the 1. PC only 1. PC control panel, not the speaker video) CNRS Summer School on Mobile Learning 4-9 July 2009, Brest, France
  • 3. Laurence Nigay - laurence.nigay@imag.fr EHCI group: Research axes EHCI group: Research axes • Interaction plasticity • 4 complementary research axes » camnote_shortRepetWeaving.mov – Models and processes for designing HCI, formal testing – Interaction plasticity • Adaptation of HCI to interaction context – Post-WIMP interaction Control panel • Vision based finger tracking and two-handed interaction • Mixed reality – Modality and multimodality EHCI group: Research axes EHCI group: Research axes • WIMP • Vision-based tracking (finger, token) – Windows: scroll, resize, move – Icons: representations, drag/drop – Menus: pop-up, pull-down – Pointers: mouse, trackball • Post-WIMP EHCI group: Research axes EHCI group: Research axes • Mixed Reality • New technique of menu: multi-stroke » Video Geonote_26.avi marking menu •Submenus » Video MenuWave.wmv Previsualization » Video WaveMenuIphone.mov – Rapid scan of possibilities get •Parent menus: 1) Menu appears – Always visible 2) Menu is enlarged – Can be directly selected 3) Submenu appears 4) Final item is selected •Visual stability of the focus of attention CNRS Summer School on Mobile Learning 4-9 July 2009, Brest, France
  • 4. Laurence Nigay - laurence.nigay@imag.fr EHCI group: Research axes EHCI group: Research axes • 4 complementary research axes • Modality and multimodality – Models and processes for designing HCI, • RAZZLE: Mobile game – The goal of the player is to collect the pieces of a digital formal testing puzzle – Interaction plasticity – The digital puzzle pieces are scattered all over a • Adaptation of HCI to interaction context modeled playground – Post-WIMP interaction • Vision based finger tracking and two-handed interaction • Mixed reality – Modality and multimodality EHCI group: Research axes EHCI group: Research axes • Modality and multimodality • Modality and multimodality • Multimodality for the task of collecting a selected • RAZZLE experimental tests (10 subjects / 3 sessions) puzzle piece – Use of the modalities – Speech modality Get – All sessions / All subjects Video RAZZLE RAZZLEExpe2.dv – Gestural modality 24% Tactile 45% Gestural Speech Video RAZZLE – Tactile modality RAZZLEExpe1.dv 31% EHCI group: Research axes EHCI group: Research axes • Modality and multimodality • Modality and multimodality • RAZZLE experimental tests (10 subjects / 3 sessions) • RAZZLE experimental tests (10 subjects / 3 sessions) – Subject 4: Use of the modalities per action – Usage of the modalities per subject (for the three sessions) 100% 90% 100% 80% 90% 70% 80% 70% 60% Tactile 60% Tactile 50% Gestural 50% Gestural 40% Speech 40% Speech 30% 30% 20% 20% 10% 10% 0% 0% Rotate 1 Collect 1 Rotate 2 Collect 2 Rotate 3 Collect 3 1 2 3 4 5 6 7 8 9 10 CNRS Summer School on Mobile Learning 4-9 July 2009, Brest, France
  • 5. Laurence Nigay - laurence.nigay@imag.fr EHCI group: Research axes EHCI group: Research axes • Modality and multimodality • Modality and multimodality • RAZZLE experimental tests (10 subjects / 3 sessions) • RAZZLE experimental tests (10 subjects / 3 sessions) – Subject 9: Use of the modalities per action – The subjects used all of the modalities (for the three sessions) 100% => Usefulness of multimodality 90% 80% – Individual preferences leading in some cases to 70% 60% specialization Tactile • Subject 4: Tactile modality for rotating a puzzle piece 50% Gestural 40% • Subject 9: Speech for rotating and collecting a puzzle piece Speech 30% => Usefulness of multimodality 20% 10% 0% Rotate 1 Collect 1 Rotate 2 Collect 2 Rotate 3 Collect 3 EHCI group: Research axes • Engineering for HCI – Models and processes establishing links between • EHCI group psychology-ergonomics and software engineering • Mobility – Interaction plasticity – Post-WIMP interaction – Modality and multimodality – http://iihm.imag.fr/en/ Mobility: a broad concept Mobility: a broad concept (1) moving between devices - hot-desking (1) moving between devices - hot-desking (2) moving within an instrumented (2) moving within an instrumented environment - intelligent buildings and environment - intelligent buildings and intelligent appliances intelligent appliances (3) devices within moving vehicles- (3) devices within moving vehicles- computers in cars computers in cars (4) small devices that move with you - (4) small devices that move with you - PDAs, laptop computers etc PDAs, laptop computers etc Alan Dix (Lancaster University) Alan Dix (Lancaster University) CNRS Summer School on Mobile Learning 4-9 July 2009, Brest, France
  • 6. Laurence Nigay - laurence.nigay@imag.fr Mobility: Mobility: moving between devices moving between devices • Nomadic computing Mobility: moving between devices Mobility: a broad concept • In the 1990s, the Internet (1) moving between devices - hot-desking – a global transport for e-mail (2) moving within an instrumented and data exchange, environment - intelligent buildings and – but mostly for people intelligent appliances at fixed locations (3) devices within moving vehicles- computers in cars • More recently (4) small devices that move with you - – the office extended to employees on-the-go PDAs, laptop computers etc ("nomadic workers") => similar to the case 4 « small devices that move Alan Dix (Lancaster University) with you » Moving within an instrumented Moving within an instrumented environment environment Smart spaces How many people are in What is the the gym? Is nearest available the treadmill conference room? free? • Examples: Motion Sensor: – Smart spaces Room 3201 is Empty What is the – Augmented classroom / Augmented museum average temperature on the second – Wireless supermarket floor? CNRS Summer School on Mobile Learning 4-9 July 2009, Brest, France
  • 7. Laurence Nigay - laurence.nigay@imag.fr Moving within an instrumented Moving within an instrumented environment environment • Active Badge • Olivetti Research Laboratory in – Experimented with 32 staffs in ORL company in 2 weeks Cambridge, UK, in 1990-92 – All staff wearing badge that emits IR signal every 15 seconds • Active Badge: location-aware systems – Intended to aid telephone receptionist – FIND(name) » Provides current location of the named badge wireless IR network – WITH(name) Active Badge Badge Sensor Server/Application » Locates a named badge and other badges around the badge – LOOK(location) Active Badge » Provides badges near the specified location Badge Sensor – HISTORY(name) ... » Generates report of the location history for the named badge Active Badge Moving within an instrumented Moving within an instrumented environment environment • Server • ATT Lab Cambridge: Active Bat • Network Control – Polling all sensors on the network – Sentient Computing: a form of ubiquitous • Representation computing which uses sensors to perceive its – Builds linked list of (ID, location, time) in time order • Data Processing environment – Process large amount of data from the active badge network - A "follow-me phone" which would cause the • Display Interface telephone nearest the recipient to ring. – Showing textual information - Teleporting desktops via VNC just by clicking their Active Bat near the computer. Video Sentient.mpg Moving within an instrumented Moving within an instrumented environment environment • ParcTab (Rank Xerox)1995 – http://sandbox.parc.xerox.com/parctab/ • Projet EasyLiving Microsoft • Video EasyLiving.mpeg 7 ,8 cm cm 2 ,4 5 cm 10, CNRS Summer School on Mobile Learning 4-9 July 2009, Brest, France
  • 8. Laurence Nigay - laurence.nigay@imag.fr Moving within an instrumented Moving within an instrumented environment environment • Location-aware system (within a buidling) • Context-triggered actions are simple IF- THEN rules used to specify how context- aware systems should adapt Like living in a rule-based expert system • Examples: – Smart spaces – Augmented classroom / Augmented museum – Wireless supermarket Moving within an instrumented Moving within an instrumented environment environment • Augmented classroom: AR (video) • Examples: – Smart spaces – Augmented classroom / Augmented museum geometry and mathematics education – Wireless supermarket Moving within an instrumented Wireless supermarket concepts environment • Augmented surpermarket • The Shopping Buddy – Based on RFID for products wireless touch-screen device attached to a – WLan for communication inside the shopping cart supermarket – scans in items placed in the cart by shoppers – Intelligent chariots (with PC tablet) – delivers personalized services and incentives when activated with a frequent-shopper card – RFID which triggers certain offers and can help shoppers find anything in the store and draw a path to find it. CNRS Summer School on Mobile Learning 4-9 July 2009, Brest, France
  • 9. Laurence Nigay - laurence.nigay@imag.fr Wireless supermarket Operation Wireless supermarket Operation • As you shop, you can scan in each item – keeps a running total of how much you are spending – eliminates the need to wait in line at the check-out – you can also check the price of each item before you buy • As you walk down the aisles – promotions and paperless coupons “pop-up” on the screen – Reminders of articles you search – Directions to articles • You can place an order at the deli (ex. Half a kilo of ham) from anywhere – notification to pick it up (No more taking a number and standing in line) Wireless supermarket Operation Wireless supermarket Operation • The Everywhere Display (IBM) transforms any surface • The Everywhere Display (IBM) into an interactive computer • Beamed • A recent MIT project: Sixthsense from the supermarket ceiling – wearable gestural interface that augments the • transforms any surface physical world around us with digital infor into an interactive computer • On the floor • On the items Moving within an instrumented Mobility: a broad concept environment • Technological approach (1) moving between devices - hot-desking – 1. Augment the user (2) moving within an instrumented • The user wears or carries a device to obtain environment - intelligent buildings and information about physical objects. intelligent appliances • => similar to the case 4 « small devices that move (3) devices within moving vehicles- with you » computers in cars – 2. Augment the physical object (4) small devices that move with you - • The physical object is changed by embedding input, output or computational devices on or within PDAs, laptop computers etc it. – 3. Augment the environment surrounding the Alan Dix (Lancaster University) user CNRS Summer School on Mobile Learning 4-9 July 2009, Brest, France
  • 10. Laurence Nigay - laurence.nigay@imag.fr Devices within moving vehicles Devices within moving vehicles micro Tactile Surface Keyboard Devices within moving vehicles Devices within moving vehicles • FutureLab of Ars Electronica (Autria) • INTUITION DGA project • Augmented Reality • Augmented cockpit (Rafale) • Instar.mov video Devices within moving vehicles Mobility: a broad concept • INRETS-Lille (1) moving between devices - hot-desking (2) moving within an instrumented environment - intelligent buildings and intelligent appliances (3) devices within moving vehicles- computers in cars (4) small devices that move with you - PDAs, laptop computers etc Alan Dix (Lancaster University) CNRS Summer School on Mobile Learning 4-9 July 2009, Brest, France
  • 11. Laurence Nigay - laurence.nigay@imag.fr Evolution of personal computers: Mobility: small devices a continuum Desktops Laptops Palmtops Handhelds Wearables Eye-pod, projection on contact lenses (univ. Washington) Evolution of personal computers: Evolution of personal computers: a continuum a continuum External support + hand(s) 1 hand (at least) Hands-free Desktops Laptops Palmtops Handhelds Wearables Desktops Laptops Palmtops Handhelds Wearables Weight portability/mobility/maniability Wearable Computer Aim of Wearable Computing • A small portable computer that is designed to be worn on •To develop new interfaces that the body during use. mediate (augment, deliberately diminish, or otherwise modify) non- • In this, wearable computers differ from PDAs, which are computer activities designed for hand-held use. • Wearable computers are usually either integrated into the user's clothing or can be attached to the body • Without interfering with the user's through some other means, like a wristband. everyday tasks • They may also be integrated into everyday objects that are constantly worn on the body, like a wrist watch or a • The design of wearable computers is hands-free cell phone. still a topic of research, and a variety of user interfaces are being proposed University of Waterloo CA University of Waterloo CA CNRS Summer School on Mobile Learning 4-9 July 2009, Brest, France
  • 12. Laurence Nigay - laurence.nigay@imag.fr Evolution of Application domains Wearable Computers • Medical domain – Medical monitoring device – (blood pressure, temperature) • Military domain Amon: ICT project GlucoWatch Application domains Wearable Computing • Mobile workers • Key characteristics – Hands free • Constant: Always ready. It is always on and always running. • Personal: Human and computer are inextricably intertwined. Mobility: Huge variability of devices Mobility: small devices • Usage spaces of wireless devices Desktops Laptops Palmtops Handhelds Wearables Marcus A., Chen E “Designing the PDA of the Future” Interactions 2002 CNRS Summer School on Mobile Learning 4-9 July 2009, Brest, France
  • 13. Laurence Nigay - laurence.nigay@imag.fr Mobility: small devices Mobility: small devices • Information • M-Commerce – Personal Information Management – Electronic money, e-coupons – Business information: specific professionals, such as a drug interaction reference for doctors • Self-Enhancement – Active assistance • Ways in which the device extends one’s normal capabilities. (Finland: cell-phones "Kanny » extension of the hand ») – Health and Safety • Contact with emergency response teams or family – Education Mobility: small devices Mobility: small devices • M-Commerce • Usage spaces – Electronic money, e-coupons of wireless devices • Relationships – Core competence of phones to connect people • Huge variety • Entertainment of applications – Music, games – The mobile game market tends to follow the more mature console/PC game market – Game performance must match or exceed the users’ expectations Marcus A., Chen E “Designing the PDA of the Future” Interactions 2002 Variety of applications Variety of applications • Device: designed for a certain user task • Device: designed for a certain user task – impractical to have several different information Sony – Reader-ebook PRS-505 Screen has no backlight or flicker appliances for different purposes to give the experience of reading a paper page • Device: « Swiss-army-knife » useful in many different situations • + Different devices are appropriate for – HCI complicated / non intuituive different tasks and this should also guide the application design • the question of information appliance or • - Impractical to have several different information Swiss-army knife still remains unanswered appliances for different purposes CNRS Summer School on Mobile Learning 4-9 July 2009, Brest, France
  • 14. Laurence Nigay - laurence.nigay@imag.fr Mobility: Huge variability of Mobility: a broad concept devices (1) moving between devices - hot-desking (2) moving within an instrumented Desktops Laptops Palmtops Handhelds Wearables environment - intelligent buildings and intelligent appliances (3) devices within moving vehicles- computers in cars (4) small devices that move with you - PDAs, laptop computers etc Alan Dix (Lancaster University) Objectives • EHCI group • Mobility • Interaction with handheld devices From: MobileHCI 07 N. Savio J. Braiterman Design Sketch: The context of Mobile Interaction Challenges Variability of usage context • Key issue: Transparency/Usability of mobile devices • Challenges for HCI – Variability of platforms – Variability of usage context – Limited interactionnal resources From E. Kaasinen: User acceptance of mobile services - value, ease of use, trust and ease of adoption CNRS Summer School on Mobile Learning 4-9 July 2009, Brest, France
  • 15. Laurence Nigay - laurence.nigay@imag.fr Challenges Challenges • In mobile computing: the problem space • In conventional HCI: the problem space is is unbounded bounded – The target user (e.g., an office clerk) – accomplishes a well-defined set of tasks (e.g., writing a report) – using a fixed class of devices (e.g., a workstation) – in a predefined set of environments (e.g., in the office) Task User Env. Platform Task User Env. Platform Introduction: how to address this Handhelds: diversity, but in problem? common … • Approach1: to develop the systems on a case per case basis (ad- • Small screens hoc manner) • Limited input capabilities – Development cost and maintenance: very high! – Consistency problem between the various versions • No standards yet • Approach2: to improve the methods and tools in order to support • Recommandation – Portability • Apply the foundations of HCI - e.g., the method – Reusability • Create without re-inventing the wheel: draw your – Modifiability design on existing solutions (in order to support – Scalability in terms of consistency across applications/brands) and invent • Computing resources new design solutions to overcome current limitations • Services • Interaction techniques (multimodality, UI for small and large devices) => Plastic UI Objectives Focus on HCI Gulf of Evaluation Mobile Device Goals Gulf of Execution From: MobileHCI 07 N. Savio J. Braiterman Design Sketch: The context of Mobile Interaction CNRS Summer School on Mobile Learning 4-9 July 2009, Brest, France
  • 16. Laurence Nigay - laurence.nigay@imag.fr The design principle of The Design of Everyday Things transparency • How to design things that real people can a design principle is to try to make systems transparent so that people can understand them better and know what to use: successfully, easily, pleasantly do • 7 stages of performing an action • NOT to be understood as literal • Conceptual models – the device should • useful feedback explain itself • easy to understand • Visibility and Affordances • intuitive to use • clear & easy to follow instructions • Mappings • context sensitive guidance of • Feedback how to proceed when stuck From Prof. Steve Whittaker From Prof. Steve Whittaker Seven Stages of Action Gulfs of Execution & Evaluation Goals Goals Evaluation of Evaluation of Intention to act the interpretations Intention to act the interpretations GULF OF EXECUTION Interpreting Interpreting the perception Sequence of actions the perception Sequence of actions GULF OF EVALUATION Execution of the Perceiving the state Execution of the Perceiving the state action sequence of the world action sequence of the world The World The World From Prof. Steve Whittaker From Prof. Steve Whittaker Focus on HCI The Gulf of Execution • Does the system provide actions that correspond to the user’s intentions? Gulf of Evaluation • The difference between intentions and Mobile Device Goals allowable actions is the Gulf of Execution Gulf of Execution From Prof. Steve Whittaker CNRS Summer School on Mobile Learning 4-9 July 2009, Brest, France
  • 17. Laurence Nigay - laurence.nigay@imag.fr The Gulf of Evaluation The Seven Stages as Design aids • Does the system provide a physical How easily can the user: representation that can be readily Determine the function perceived and interpreted in terms of the of the system? user’s intentions and expectations? Tell what actions Tell if the system is in the desired state? • The Gulf of Evaluation reflects the amount are possible? of effort that the person must exert to Determine a mapping from Determine a mapping from interpret the physical state of the system intention to physical action? system state to interpretation and determine how well the intentions have been met. Perform the action? Tell what state the system is in? From Prof. Steve Whittaker From Prof. Steve Whittaker … principles for good design Mobile HCI: inputs/outputs • Visibility – by looking, users can tell the state of the device and possible actions • Conceptual model – operations are presented consistently; users can easily understand “how the system works” • Mappings – it is possible (easy) to determine the relationship between actions and results, between controls and their effects, and between the system state, and visible indicators • Feedback – users get full and continuous feedback about the results of actions From Prof. Steve Whittaker Mobile HCI Mobile HCI • A very active domain • A very active domain • Approach1: to develop the systems on a case per case basis • Approach2: to improve the methods and tools in order to support – Portability • Plasticity (approach 2) – Reusability – Modifiability • Innovative interaction techniques – Scalability in terms of (approach 1) • Computing resources • Services – Innovative interaction techniques • Interaction techniques (multimodality, UI for small and – Multimodality large devices) => Plastic UI CNRS Summer School on Mobile Learning 4-9 July 2009, Brest, France
  • 18. Laurence Nigay - laurence.nigay@imag.fr Plasticity of an interactive Plasticity system • Capacity of the system to adapt to the • Materials: capability to change their own context of use while preserving their own shape while preserving usage, they do not utility and usability break (within some limits, of course) • Animals, plants: they adapt to the • Context of use … environment to survive • Adaptation … • Utility and Usability … • By analogy in HCI … Context of use Plasticity of an interactive system • Context: it’s complex! Generally speaking, it is an information space – that is shared between actors, dynamic, structured, and – that serves interpretation (by the actors) • There is “no context without context” -> context of use • Capacity of the system to adapt to the context • Context of use: an information space of use while preserving their own utility and – that is shared (between the software components of the interactive system and between the system and the users), dynamic, and structured into 3 sub-spaces: usability • The users • The platform (from an elementary platform such as a mobile phone to a cluster of elementary devices) • The physical environment (location, light-sound-heat and social conditions) – that supports the adaptation process to preserve the system utility and usability • Context of use … • Adaptation … • Utility and Usability … Two means for UI adaptation UI remolding (reshaping) • Example: Plastic Clock • Remolding (reshaping) • Redistribution CNRS Summer School on Mobile Learning 4-9 July 2009, Brest, France
  • 19. Laurence Nigay - laurence.nigay@imag.fr UI remolding (reshaping) UI remolding (reshaping) • Physical Presentation level (PP) NSButton : Contrôleur de Nabaztag dialogue Button : ANF Présentation Interaction JButton : NF Interaction Interaction The classes of widgets are maintained but their behaviour (possibly, their spatial and temporal layout) has changed PC PDA UI remolding (reshaping): may UI remolding (reshaping) cover multiple levels of abstraction • Logical presentation level (LP) • Dialogue Control level (DC) Label + ComboBox : Contrôleur de dialogue Tasks are maintained but their ordering has changed DC Modification -> LP modification -> PP Contrôleur de dialogue Contrôleur de Contrôleur de modification dialogue ANF dialogue Présentation Présentation ANF Label + T extField : Présentation NF Interaction Interaction NF Interaction Interaction Substitution of widgets LP Modification -> PP Modification UI remolding (reshaping) UI remolding (reshaping) • Functional Core Adaptator level (FCA) • Functional Core level (FC) Contrôleur de Contrôleur de Book (workstation) Contrôleur de Contrôleur de ANF dialogue dialogue - title ANF dialogue dialogue The nature of the services of the functional core has changed ANF ANF ANF Présentation - authors ANF Présentation - summary FC modification ->FCA Modification -> DC modification NF Interaction NF Interaction - keywords … -> LP modification > PP modification Book(Handheld) - title - first author -keyword Tasks and domain-dependent concepts may be added/suppressed FCA Modification -> DC modification -> LP modification > PP modification CNRS Summer School on Mobile Learning 4-9 July 2009, Brest, France
  • 20. Laurence Nigay - laurence.nigay@imag.fr UI Redistribution UI Redistribution • Rekimoto’s “pick and drop” • The UI uses interaction resources that are distributed over multiple elementary platform (distributed UI) – Distribution is static • UI redistribution may be static or dynamic – Distribution at the workspace level • Dynamic redistribution => on-the-fly migration of the UI – No remolding – Migration may be total: the UI releases (frees) all of the interaction resources currently used and migrates entirely to other interaction resources – Migration may be partial: only portions of the UI migrate -> what is the • i-LAND granularity for UI migration? – Distribution is dynamic – Distribution at the pixel level – No remolding http://www.ipsi.fraunhofer.de/ambiente/english/projekte/projekte/i_land.html UI Redistribution Plasticity of an interactive 2. Arrival of the PDA: UI partial migration system • CamNote • Capacity of the system to adapt to the context of use while preserving their own utility and usability Remolding of the control panel • Context of use … at the FCA level 1. PC only: centralized UI • Adaptation … 3. Distributed UI (PC and PDA) + • Utility and Usability … remolding Usability of an interactive Utility of an interactive system system: many faces! • Capacity of the system to provide the target users with the appropriate functions/services (not less, not more) • To make it simple: Capacity of the system to allow the target users to access and manipulate the services • Appropriate = in conformity with users’need (not with provided by the interactive system in conformity with designers’ and developers’ needs!) their cognitive, motor and perceptual capabilities – Cognitive is the hard part ! Knowledge, pleasure, • How to elicit the appropriate services? emotion, mood come into play! – Talking to users is not a luxury, it is mandatory! – Questionnaires, focus group, evaluation, rapid • How to develop a usable system? prototyping (low-fidelity), observation of users in – Use theory-based knowledge (e.g., Theory of action, their daily activities, etc. Fitts law) – In short: user-centered design – Apply ergonomics rules – Design, evaluate with the end-user and iterate CNRS Summer School on Mobile Learning 4-9 July 2009, Brest, France
  • 21. Laurence Nigay - laurence.nigay@imag.fr Utility and Usability are not intrinsic to a product! It depends on the context of use Mobile HCI • Utility and usability depend on the “values” that the user associates to the system • A very active domain • The “values” depend on the context of use • Plasticity • Innovative interaction techniques – Innovative interaction techniques • Example: possible values for a heating control system that may depend on the context of use – Multimodality – Money saving – Comfort Interaction techniques: Interaction techniques small screen • Small screens • Fisheye technique • Limited input capabilities – Allows the user to see the details without loosing the context of the details (Furnas: seminal work) – Video – datelens-video-web-server.wmv Interaction techniques: Interaction techniques: small screen small screen • A peep hole on a large information space [Ka-Ping Yee, CHI2003] • Problem: loss of the context - • Microsoft Research Lab • Halo tends to alleviate this problem • Projet Halo Halo to make observable the information that is of interest to the user • Problem + CNRS Summer School on Mobile Learning 4-9 July 2009, Brest, France
  • 22. Laurence Nigay - laurence.nigay@imag.fr Interaction techniques: Interaction techniques: small screen small screen • Hybrid technique: thumbnail+editing Video Halo.avi • Collapse-to-Zoom – Pages that are to big to fit the screen are presented as a “summary thumbnail” – Thumbnails are editable (can be suppressed/enlarged, etc.) Solution expand collapse [Baudisch CHI 2004] Interaction techniques: Interaction techniques: small screen small screen • Hybrid technique: thumbnail+editing • Menu techniques • Collapse-to-Zoom – Space on screen – Pages that are to big to fit the screen are presented as a “summary thumbnail” – No keyboard for shortcuts (novice mode) – Thumbnails are editable (can be suppressed/enlarged, etc.) – The best way to interact is to use only one-hand Grey lines indicate the existence of collapsed areas Circular menus (advertisement) video collapsetozoom Interaction techniques: Interaction techniques: small screen small screen • Marking menus • Marking menus • Multi-stroke marking menus • Advantages • Instead of considering a spatial compound stroke, – Circular design • Multi-Stroke menus introduce a serie of simple strokes – Fluid transition – Scale independance • Require less physical input space in novice & expert modes – A submenu is displayed on top of • Limitations its parent menu – Screen space requirement – Overlapped marks – Number of commands • Error rate in expert mode • Increase accuracy in expert mode • Ambiguous marks in expert mode • Increase the number of items – No ambiguous gestures in expert mode video `Multistroke CNRS Summer School on Mobile Learning 4-9 July 2009, Brest, France
  • 23. Laurence Nigay - laurence.nigay@imag.fr Interaction techniques: Interaction techniques: small screen small screen Wave menus Wave menus • Few physical input space • A submenu is displayed at the center of the menu system. In order to remain visible, its parent menu is then enlarged to surround Novice Mode this submenu • In expert mode, Wave Menus (1) work exactly the same way as Multi-Stroke Menus (2) Expert Mode video WaveMenu Interaction techniques: Interaction techniques: small screen small screen Wavelet menus: extension on Iphone Wavelet menus: extension on Iphone •The Wavelet menu appears centered around the contact point. •By drawing a stroke towards the desired item, the first level is enlarged permitting progressive appearance of the submenu. •A second stroke selects an item in the submenu. Interaction techniques: Interaction techniques: small screen small screen • Menu techniques Wavelet menus: extension on Iphone – Space on screen – The best way to interact is to use only one-hand Long lists management: the linear list appears in the center of the Wavelet menu and is surrounded video Leafmenunovice Leafmenuexpert by its parent menus. ( video Wavelet.mov) CNRS Summer School on Mobile Learning 4-9 July 2009, Brest, France
  • 24. Laurence Nigay - laurence.nigay@imag.fr Interaction techniques: Interaction techniques: small screen small screen • Menu techniques • Non graphical techniques => Eye-Free interaction – Space on screen – Tactile & Sound feedback – The best way to interact is to use only one-hand – ArchMenu, ThumbMenu • Spatialized sound Meeting at 10 Soundbeam Neckset Interaction techniques: Interaction techniques: small screen small screen Vibro-Tactile feedback Shoogle senses data (and makes sense of them!) [ Univ. Vibro-tactile feedback Glasgow 07] Shape-change feedback: the dynamic knob [Deutsche Telecom Sensors: the Shake includes an accelerometer, gyroscope, et Univ. Postdam] magnetometer, vibro-tactile display communicating via bluetooth Expresses the existence of messages, remaining resources (e.g., the battery level) using a dynamic model inspired from mechanics (balls anchored with springs bouncing against walls) Interaction techniques Text entry • Small screens • Limited input capabilities CNRS Summer School on Mobile Learning 4-9 July 2009, Brest, France
  • 25. Laurence Nigay - laurence.nigay@imag.fr Text entry Keyboard: Text entry Text entry Text entry • LucidTouch: • LucidTouch: a see-through mobile device, pseudo-tranparency a see-through mobile device, pseudo-tranparency Keyboard: entry Text entry • Oulu University (Finland) • Hand writting + Movements CNRS Summer School on Mobile Learning 4-9 July 2009, Brest, France
  • 26. Laurence Nigay - laurence.nigay@imag.fr Pointing Techniques Pointing Techniques • The best way to interact is to use only • “Offset Cursor” [Potter, Shneiderman 1988)]: how to replace the stylus one-hand • with the finger without hiding the target Selection technique: finger touches the screen, display the cursor at 1/2 • Using the thumb inch over the finger, move the finger to bring the cursor over the target, remove finger Target Finger hides the target Finger moves to bring the cursor Gestures of the thumb Cursor is offset over the target MicroRolls from the target Pointing Techniques Pointing Techniques • Shift [P. Baudisch 07, MicroSoft]: improves the “offset cursor” technique Occlusion Occlusion of targets Occlusion of the interface Border accessibility Thickness of screen Areas more difficult to reach [Parhi 06] edges Accuracy Video: shiftMini.m4v Larger than the stylus Prediction hard Pointing Techniques New input Techniques • Tap-Tap Anne Roudaut (AVI 2008) Video: taptapMini.m4v • Temporal multiplexing strategy • Camera phone based motion sensing • 1ST tap: Specify the focus area Uni. of California, Berkeley • 2nd tap: Select the target at a larger scale & IBM Video: Phone3DMvt-UIST06.mov Target to select 1st tap Magnified popup 2nd tap CNRS Summer School on Mobile Learning 4-9 July 2009, Brest, France
  • 27. Laurence Nigay - laurence.nigay@imag.fr New input Techniques New input Techniques • Movement-based UI – 3D gesture • Tilting using accelerometers [Rekimoto 1996, Harrison&Fishkin 1998] • Movement-based UI – 3D gesture • Embodied UI: Rank Xerox Photos album Video: RocknScroll.mov New input Techniques New input Techniques • Embodied UI: Rank Xerox • Embodied UI • Prototyping tool for embodied UI – Stanford HCI group –Video: MapPDADesignplatform-UIST06.mov New input Techniques New input Techniques • RodDirect: the stylus as a peripheral device (2 degrees of freedom) + the second hand • Distorsion: bendable display • Two types of stylus movement in the stylus holder are • The Gummi interface: just bend it! (Sony, Schwesig CHI 2004) associated with parameters – (1) Twist (Rotation) (2) Push/Pull (Sliding) Map Browser Application Switching Scheduler JAIST: Japan Advanced Institute of Science and Technology, JAPAN Video: RodDirectDemo.mpg CNRS Summer School on Mobile Learning 4-9 July 2009, Brest, France
  • 28. Laurence Nigay - laurence.nigay@imag.fr New input Techniques Mobile HCI • Gummi: zoom-in a subway station • A very active domain • Plasticity • Innovative interaction techniques • Alpha-blending of 2 graphics layers – Innovative interaction techniques – Multimodality Which actor performs modalities Multimodal Interaction selection ? • Multimodal systems – Multi-Sensori-Motor Systems – extend the sensori-motor capabilities of computer systems = mobile devices Designer User System • Multimodal interaction makes use of several input and/or output (feedback) modalities in Information Selection of Multimodal to be one or several modalities interacting with a computer system. conveyed Expression – Examples of modalities: manual gestures, gaze, touch, speech, head & body movements Actor of the selection – Modality: human sensory channel, different representation modality, or different input method Multimodality selection Multimodality Adaptability Go to the middle of the message No adaptation Gestural modality Speech Adaptability Adaptivity Selection by Selection by Selection by the designer the user the system Embodied modality Direct manipulation CNRS Summer School on Mobile Learning 4-9 July 2009, Brest, France
  • 29. Laurence Nigay - laurence.nigay@imag.fr Multimodality Multimodality Adaptativity Adaptability • Usage of the modalities • Selection of the modalities by the system • All sessions / All subjects • Context-aware systems Speech Tata tata Direct manipulation ta ta ta taaaa tata Gesture Embodied Ring Vibration Multimodality: combining MATCH modalities • “Put that there” paradigm • Multimodal Access to City Help R. Bolt MIT • A Multimode Portable Device that accepts „Play this sound speech and pen gestures created by logo” ATT&T User selects a sound logo by clicking on the title with a stylus and speaking in order to hear it MATCH MATCH Testing Statistics • Part of a multi-year contract from DARPA • Exchanges 338 • Enables users to interact using speech, pen, or – Speech only 171 (51 %) synchronized combinations of speech and pen – Multimodal 93 (28%) • Essentially a testbed for designing portable – Pen only 66 (19%) multimodal applications – GUI actions 8 (2%) • Video: DEMOMULTI/CityPlannerATT.mpeg CNRS Summer School on Mobile Learning 4-9 July 2009, Brest, France
  • 30. Laurence Nigay - laurence.nigay@imag.fr Multimodality: combining Multimodality: a tool for modalities exploring multimodal interaction • OpenInterface (OI) • KIRUSA : Multimodal Interaction – Assembling components – speech, text, and touch input – graphics, text and audio output • CARE properties • C : Complementarity • A : Assignment • R : Redundancy • E : Equivalence OI: a tool for exploring OI Example: multimodal interaction Assembling components • Construction tool: Direct manipulation and Task: Tasks assembling of components in order to Center and Zoom Point (x,y) Go there specify a "pipeline” defining a multimodal interaction CARE Complementarity • Focus on multimodal interaction: The combination "pipeline” of components is sending elementary tasks (i.e., "commands") to the Transformation Designation Speech commands command rest of the interactive system (for example, processes a game) Microphon Devices Camera e Iterative user-centered design User-centered design Ergonomic evaluation Software prototype Final design: Final product – To support an iterative user-centered design Test-bed Detailed running on mobile Software specifications devices – To allow the rapid prototyping of the mobile prototype Design Test-bed interaction for conducting test with users Ergonomic Design Initial evaluation Requirement Ergonomic s evaluation Code Ergonomic Design Design evaluation Software prototype Software Software Test-bed prototype prototype Test-bed Test-bed Design Ergonomic evaluation Iterative design Developmen t CNRS Summer School on Mobile Learning 4-9 July 2009, Brest, France
  • 31. Laurence Nigay - laurence.nigay@imag.fr Game Test-Bed on Mobile Game testbed on mobile platform Devices • Mobile game: Running on mobile phones or PDAs • Test-bed: Running on PC Bluetooth / TCP-IP WiFi PC: Multimodal interaction Mobile device: Game Game Test-Bed on Mobile Game Test-Bed on Mobile Devices Devices • 1: Raw input from the devices (low level) • 2: Application-dependent messages (high level) Levels of • 3: Modality-independent and application-independent events abstraction of the exchanged messages ? PC: Multimodal interaction Mobile device: Game PC: Multimodal interaction Mobile device: Game Game Test-Bed on Mobile Funny Rabbit Devices • Examples: “ESC”, “UP”, “START” => virtual gamepad • 3D game on mobile phones • Goal: Find the rabbit that is hidden in a • 3: Modality-independent and application-independent events chest PC: Multimodal interaction Mobile device: Game CNRS Summer School on Mobile Learning 4-9 July 2009, Brest, France
  • 32. Laurence Nigay - laurence.nigay@imag.fr Funny Rabbit Funny Rabbit • Goal: Find the rabbit that is hidden in a chest • Tasks: <Move the character> <Select the chest> <Examine objects> Bluetooth • Sub-tasks: <Left> <Right> <Up> <Down> Input Funny Rabbit: Multimodal game and interaction graphical outputs (N93 and N95) Multimodal interaction Funny Rabbit Funny Rabbit Funny Rabbit <Left> <Right> <Up> <Down> Shake device <Left> <Right> <Up> <Down> Shake Evt Adapter MultiBT ARToolkit Adapter Evt Adapter MultiBT OI component pipeline Funny Rabbit OI component pipeline Funny Rabbit Funny Rabbit Funny Rabbit Open Left <Left> <Right> <Up> <Down> <Open the chest> Speech Evt Adapter MultiBT Speech Evt Adapter MultiBT Openin Shake g gesture OI component pipeline Funny Rabbit OI component pipeline Funny Rabbit CNRS Summer School on Mobile Learning 4-9 July 2009, Brest, France
  • 33. Laurence Nigay - laurence.nigay@imag.fr OpenInterface • OI: Running on PC • EHCI group • Connection to the mobile phone • Mobility • Interaction with handheld devices Bluetooth / TCP-IP WiFi PC: Multimodal interaction Mobile device http://www.oi-project.org/ Mobility: a broad concept • EHCI group (1) moving between devices - hot-desking • Mobility (2) moving within an instrumented environment - intelligent buildings and • Interaction with handheld devices intelligent appliances • Conclusion (3) devices within moving vehicles- computers in cars (4) small devices that move with you - PDAs, laptop computers etc Conclusion: Input/Output Interaction From WIMP tothe env. Integration into Post-WIMP Mobility Traditional interaction [Lyytinen & Yoo 2002] CNRS Summer School on Mobile Learning 4-9 July 2009, Brest, France
  • 34. Laurence Nigay - laurence.nigay@imag.fr Conclusion: Conclusion: From WIMP to Post-WIMP From WIMP to Post-WIMP Integration into the env. Integration into the env. Pervasive computing Mobility Mobility Traditional interaction Mobile Traditional interaction Mobile computing computing Conclusion: Thank you ? From WIMP to Post-WIMP Integration into the env. Pervasive Ambiant intelligence computing Mobility Traditional interaction Mobile computing Readings • Books – S. Weiss, Handheld usability, Wiley, 2002. ISBN 0 470 84446 9. – C. Lindholm, T. Keinonen, H. Kiljander, Mobile usability, McGraw-Hill, 2003. ISBN 0 07 138514 2. • Conferences – Mobile HCI ACM-Press – UBIMOB Mobilité et Ubiquité AICPS ACM-Press – CHI ACM-Press – UIST ACM-Press – AVI ACM-Press CNRS Summer School on Mobile Learning 4-9 July 2009, Brest, France