Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Adaptation and Continuity in Multi-Device Environments

2,428 views

Published on

This tutorial aims to help user interface designers and developers to understand the issues involved in multi-device interactive applications, which can be accessed through both mobile and stationary devices even exploiting different interaction modalities (graphical, vocal, gesture, ...). It will provide a discussion of the possible solutions in terms of concepts, techniques, languages, and tools, with particular attention to Web environments. The tutorial will deal with the various strategies in order to adapt the user interface according to the interaction resources available, also discussing what results can be obtained through model-based approaches when multi-device interfaces are considered. It will consider how to address such issues both when authoring multi-device interfaces and when user interfaces for different devices are dynamically adapted and can even migrate seamlessly across them to follow the mobile user. Thus, it will discuss how to support task continuity across multiple devices in examples of distributed and migratory interfaces and related usability issues.

  • Be the first to comment

Adaptation and Continuity in Multi-Device Environments

  1. 1. Adaptation and Continuity in Multi-Device Environments i Multi-D i E i M lti t Fabio Paternò HIIS Laboratory ISTI-C.N.R. Pisa, Italy http://giove.isti.cnr.it/ Tutorial Goals  How to obtain interfaces able to adapt to multiple device types while preserving usability?  State of art in terms of approaches design criteria approaches, criteria, tools  Particular attentiont to adaptation in Web applications and use of model-based techniques  Understanding the space of the possible solutions in order to better apply them and think about new pp y solutions  Consider how to address the device adaptation issue both at design time and at run-time  Discussion on how adaptation and continuity can be supported in migratory user interfaces 2 1
  2. 2. Structure  Introduction, Basic Concepts, Issues  Usability and Task in Multi-device environments Multi device  Authoring Multi-Device Interfaces  Model-based Support for Multi-device Interfaces  Tools for Desktop-to-Mobile Adaptation  Adaptation Customization  Migratory Interfaces  Partial / Trans-modal Migration  Conclusions and Discussion Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it 3 Why Adaptation to the Context of Use User: • Preferences Environment: • Knowledge •Position • Goals • Light • Background • Noise, … •… • Social aspects Device: •Privacy • Screen resolution •Collaboration • Connectivity •……. • Browser •… 4 2
  3. 3. Display Size and Resolution  Personal Computer (PC) usually varies between 800x600 and 1920x1200 pixels,  Mobile devices usually between 240x240 and 960x640 pixels (Iphone 4)  Simple phones have even lower resolutions  It varies more with mobile devices than desktop ones  The Moore Law continuously changes these numbers! b ! 5 Mobile Interaction Techniques Variation http://www.w3.org/TR/mwabp/#bp-presentation- http://www.w3.org/TR/mwabp/#bp-presentation-interaction  Focus Based: The browser focus moves through elements  The current focus of the page is easily determined because the focus element will be highlighted;  Focus area can move from one selectable element to another (e.g. from link to link) even when (e g widely spaced  Pointer Based: Key-based navigation controls a pointer that can cover any part of the screen  Selectable elements that are associated with each other need to be close as moving the pointer can be slow;  Selectable elements need to be large enough to be easily selected -- since the pointer often moves in steps of between 5 - 10 pixels;  Selectable elements should have rollovers to make it clear when the pointer has entered their active area.  Touch Based: Events are related directly to a finger or stylus touch position on the screen.  Selectable elements may be widely spaced since the user can select them directly;  Selectable elements must be large enough to be easily selected (e.g. list items should have a height of at least 30px);  No elements are in focus until they are selected so extra information cannot be passed to the user (e.g. rollovers will not work). Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it 6 3
  4. 4. Usability in Mobile Interaction  Minimize text input  Exploit the elements of the mobile device, such y as accesskeys  Consistency between platforms  Prevent user error  The purpose of the interface elements should be clear  Avoid overloading the user interface with many elements  Limit the need for scrolling  Short time access  Access to small pieces of information Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it 7 Supporting Tasks in Multi- Multi- platform Environments  Platform definition  Same task on multiple platforms in the same p p manner  Same task on multiple platforms but with different user interface elements  Same main task, with different levels of subtasks  Dependencies among tasks performed on different platforms  Tasks meaningful only on some platform types Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it 8 4
  5. 5. Same task on multiple platforms with different user interfaces Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it 9 Task Meaningful only on Some Platforms Search for flights Promotions Reservations Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – 10 http://hiis.isti.cnr.it 5
  6. 6. Authoring Multi-device Multi- Interfaces  Platform-specific Platform specific authoring (e.g. (e g Amazon)  Multiple-device authoring (e.g. with CSS)  Single authoring  Automatic re-authoring Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it 11 Multiple Device Authoring Damask (James Lin PhD – CHI’08) •Sketches •Layers •Patterns Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it 12 6
  7. 7. Automatic Re-Authoring Re-  Scaling, such as Safari on IPhone  Transducing, t T d i translates elements i t l t l t into other formats, and compresses and converts images to match device characteristics, such as Mowser or Skweezer,  Transforming goes f th t modify b th T f i further to dif both contents and structures originally designed for desktop systems to make them suitable to display on small screens Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – 13 http://hiis.isti.cnr.it Transforming (examples)  Single column, (for example Opera SSR) eliminates scrolling in one dimension, it g greatly increases the amount of scrolling in y g the other dimension.  Fisheye (for example Fishnet) is a fisheye Web browser that shows a focus region at a readable scale while spatially compressing page content outside the focus region  Overview + detail splits a Web page into multiple sections and provides an overview li l i d id i page with links to these sections. Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it 14 7
  8. 8. Narrow Solution (e.g. Opera SSR) 15 ………. A lot of vertical scrolling !!!! Narrow Solution  The order for the content follows that of the markup file starting with the top  The images are scaled to the size of the screen  The text is always visible and the content is compacted without blank spaces  Content that requires space such as maps and tables can become unreadable  Sometimes it is difficult to understand that the page has changed because the initial part is the same 16 8
  9. 9. Requirements for the Nokia Solution (CHI 2006, Roto et al.)  Remove the need of horizontal scrolling to read text  Provide enough contextual information to give an idea of the page structure and the current location in it  Don’t destroy the original page layout  Don’t introduce modal interaction Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it 17 MiniMap Method (CHI 2006, Roto et al.) Nokia S60 phones Overview + Detail Original Compacted Mini-Map Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it 18 9
  10. 10. Application of Information Visualization Techniques Fisheye calendar allows complex tasks to be completed more quickly Focus + Context In DateLens [Bederson et al 02, 04] 19 Model- Model-based approaches  Allow designers and developers to concentrate on main semantic aspects  Languages th t represent such aspects L that t h t  Avoid need to learn and manage many implementation languages  Linking semantic information and implemementation elements p  Interoperability through many possible implementation languages  Facilitate support of assistive technology Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it 20 10
  11. 11. Abstraction Levels in Interactive Systems  Task and object – Activity oriented  – I want to select a work of art  Abstract Interface – Platform Independent  – Single selection object with high cardinality  Concrete Interface –Platform Dependent  – List Interaction object with X elements  Implementation  – List object in Java or XHTML or .... Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it 21 The ConcurTaskTrees Notation for Task Models Hierarchical Temporal relations structure Task Allocation Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – 22 http://hiis.isti.cnr.it 11
  12. 12. Support for Applications based on Web Services (http://giove.isti.cnr.it/tools/Mariae) Task  Model Web  TM Services Language Multitouch  AUI  MARIA Desktop AUI PDA AUI  Language Phone AUI  Specification Specification Specification Annotations M‐touch     Multitouch  Phone         Desktop CUI  PDA CUI  PDA     Phone CUI  CUI  Specification Specification CUI  Desktopp Language Specification Language CUI  Language Specifications Languages Reverse Engineering Automatic Model Creation ! Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it 24 12
  13. 13. Adaptation in Authoring Environments (SUPPLE, Gajos and Weld) Weld)  Takes a functional specification of the interface, the device- specific constraints, a typical usage trace, and a cost function.  The cost function is based on user preferences and expected speed of operation  SUPPLE’s optimization algorithm, finds the user interface which minimizes the cost function while also satisfying all device constraints. Adaptation and Continuity in Multi-Device Environments 25 Architectural Solutions for Automatic Web Adaptation Application server-side adaptation Request/Device capability Server Client Adapted Content Proxy-side adaptation Request/ Request Device capability Server Proxy Client Content Adapted content Client-side adaptation Request Server Client Content Adapted content Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it 26 13
  14. 14. Dynamic User Interface Adaptation  Use of Logical Description Languages  Existing Web desktop Applications  Automatic user interface generation Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it Parametric Bidimensional Cost- Cost-based Semantic Redesign Font adjustments according user preferences Images Interactor replacement Long texts Grouping Relation Data tables Layout tables Paternò, Zichitella, HCSE 2010 14
  15. 15. Adaptation Solutions http://mowser.com http://www.skweezer.com •Limited reduction of image dimensions •Uses predefined style sheets •Aims to reduce •No support for tables or long texts horizontal scrolling Customizing Adaptation Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it 30 15
  16. 16. Customizing Adaptation 31 Browsing Large Table in Small Screens Ohnishi and Tajima, UIST’08 Ease comparison of cells far from each other Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it 32 16
  17. 17. Page Summarization  The Abstraction-based approach uses sentence manipulation techniques like reduction reduction, compression and reformulation.  The Extraction-based approach assigns scores to sentences in order to select those which better represents the whole text  Feature based (e.g. term frequency, sentence p position, attributes…); machine learning, graph , ); g, g p based techniques Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it 33 Micro- Micro-level Summarization (PowerBrowser, Buyukkokten et al.) PowerBrowser,  The importance of a keyword depends on the frequency it occurs in a text and in a larger collection  A word within a given text is considered most important if it occurs frequently within the text, but infrequently in the larger collection  The significance factor of a sentence is derived from an analysis of its constituent words  The sentences in which the greatest number of e se te ces c t e g eatest u be o frequently occurring distinct words are found in closest proximity to each other are probably important  MEAD is a public multi-document summarization system, which provides more flexible support in this area (see http://www.summarization.com/mead/) 17
  18. 18. Adaptation in Multi-User and Multi- Multi- Multi-Device environments  WebSplitter (Han et al., CSCW’00)- collaborative Web browsing by creating personalized partial views of the same Web page depending on the user and the device. Developers have to specify the Web content in XML and define a policy file indicating what content tags should be shown for each device and user. Concepts and Tools for Multi-Device User Interfaces 35 How People Use Multiple Devices (Dearman & Pierce ‘08)  A recent study in US of 27 people from academic and industrial research revealed that on average they employ more than five computing devices  Associating a user’s activities with a particular device is problematic for multiple device users because many activities span multiple devices  Device use varies by user and circumstance; users assign different roles to devices both by choice and by constraint.  Users employ a variety of techniques for accessing information across devices but participants reported managing information across their devices as the most challenging aspect of using multiple devices. 36 18
  19. 19. Flexible Access in Multi- Multi- Device environments  Moving objects: across interactive devices th d i through pick-and-drop h i k dd  Distributed user interfaces: application logic receiving input from multiple devices  Migratory user interfaces: change g y g device, interface migration with state preservation Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it 37 Pick-and- Pick-and-drop Jun Rekimoto, UIST’97 / CHI’98 A Direct-manipulation Technique for Multiple-Computer Environments 38 19
  20. 20. PUC (Personal Universal Controller) [CMU, Nichols, Myers] Personal Mobile Automatically Device Generated G t d Interface Control Abstract Specification State Feedback 39 Why Migratory Interfaces  Our life is becoming a multi-device experience  One of the main source of frustration is that we need to restart for each device change  Need for continuous access to interactive services across various devices  Migratory user interfaces can transfer among different devices (from ‘source’ devices to ( ‘target’ devices), so as to allow the users to continue their tasks  Application domains such as shopping, bids for auction on line, games, making reservations Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – 40 http://hiis.isti.cnr.it 20
  21. 21. Migration Phases  Device Discovery: Devices notify their presence  When to Migrate: Migration Trigger (User or System or Mixed initiative)  Where to Migrate: Identification of Target and its resources  What to Migrate: User interface and/or Application logic  How to Migrate: Adaptation (depending on the type of target)  State Persistence: Source state extraction and association to target version  Activation in the target device: Upload the adapted version at the point in which the user left off on the source device 41 Usability in Migration  Relevant aspects to continuity  Time  Time required by the migration process db h  Time from the last interaction in the source device  Adaptation process  When adaptation makes unclear how to continue the task  Predictability:  The target device  Which part of the user interface migrate  Where the result of an interaction will be presented  Learning: to get familiar with the migration process 42 21
  22. 22. What is the state to preserve  User Input  Focus  Cookies  Session  History  Bookmarks  JavaScripts J S i t Task: SelectMenu value: Vegetable Your Selected 43 Menu is Vegetable Web Session Migration Using Dynamic 2D Barcodes A. Alapetite, PUC 14(1): 45-52 (2010) Append the ID of the current session (token) to the current URL query string When the Web server receives a request from a new device (the mobile phone) with a known existing session ID, the server will provide the same content as the one served to the previous device (the desktop). Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it 44 22
  23. 23. Approaches to Migration  Bharat and Cardelli (UIST’95) - the migration of entire applications  Chung and D Ch d Dewan (UIST’96) - when migration is h i ti i triggered the environment starts a fresh copy of the application process in the target system, and replays the saved sequence of input events to the copy. This solution does not support interface adaptation.  Kozuch and Satyanarayanan (2002) - solution for migration based on the encapsulation of all volatile execution state of a virtual machine (migration of applications among desktop or laptop systems)  Melchior, Grolaux, Vanderdonckt (EICS’09) - solution for distributed user interfaces with extension of Tcl/Tk Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – 45 http://hiis.isti.cnr.it Architectural Framework Device 1 Migration Platform Application • User Interface Migration Framework Mi ti F k • Application Logic •Server • Connection to OPEN Migration Services Client Orchestration OPEN Client Migration Services g •R Runs i b k in background d User Interface migration • Connects to OPEN Application logic migration platform framework State persistence Context management Trigger management 46 23
  24. 24. Device discovery and selection Target Device Selection •same environment •device availability •same user (or shared) •interaction features Example representation of the migration environment Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it 47 Device Selection Ghiani, PhD Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it 48 24
  25. 25. Example Architecture Migration Device Discovery (1) Migration Trigger (6) Migration Client Request Page (2) Desktop Application Cookies Reverse Semantic Redesign Provide Annotated Page (5) Proxy State Mapper Server Generator Upload (8) Trasmission of DOM + current state through callback PDA AJAX (7) Provide Page (4) Request Page (3) Application Server 49 Migrating between mobile and Digital Tv Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it 50 25
  26. 26. Example Migration (Video) (shopping scenario) Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it 51 Trans- Trans-Modal Migration Vocabularyy Sender Subject Date Size &Time  Luois Last minute offer May 25 1k lestat 9:23  Information organization George Clow Re: Cameleon meeting June 3 11:02 3k Different concrete objects Luois Learn about Java June 5 2k Lestat 12:20  Mary Rex Re: Cameleon June 7 1k meeting 10:00  Not a simple one to one mapping  Task not Supported  Task implemented with different concrete objects  Adaptation of runtime data 52 26
  27. 27. Differences in task support implementation (grouping sound) Please say your name. Please say your surname. yy Please say the date of reservation. What type of menu do you prefer: fish; meat; vegetable (grouping sound) Your name is …. Your surname is …. You have booked a table on … and You prefer… (grouping sound) If you want to cancel the reservation , say delete; If you want to confirm the reservation, say ok (grouping sound). Grouping Operator Object that Support the task “Provide date” 53 Adaptation of runtime data Your name is Louis You have booked Louis. a table on 3 July July. (grouping sound) Please say your name. Please say your surname. Please say the date of reservation. Name: Louis Date: 3 July What type of menu do you prefer: fish; meat; vegetable (grouping sound) Your name is …. Your surname is …. You have booked a table on … and You prefer… (grouping sound) If you want to cancel the reservation , say delete; If you want to confirm the reservation, say ok (grouping sound). 54 27
  28. 28. Application Logic Reconfiguration Clausthal University in OPEN Project Partial Migration (Mobile HCI 2010 paper) 56 28
  29. 29. Partial Migration Partial Migration with Interface Structure Representation 29
  30. 30. Partial Migration with Direct Selection 59 Partial Migration with Direct Selection (video) Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – 60 http://hiis.isti.cnr.it 30
  31. 31. Research Agenda  Automatic Desktop-to-Vocal Adaptation  Migration involving multiple users g g p  Integration of wider set of interaction modalities in migratory environments  Migration from multiple devices to multiple devices  Authoring Environments for Distributed User Interfaces I t f  Privacy and Security of Migratory User Interfaces  Applying Machine Learning Techniques to User Interface Adaptation Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – 61 http://hiis.isti.cnr.it More Info at  EU OPEN Project http://www.ict-open.eu  EU SERENOA Project htt // http://www.serenoa-fp7.eu/ f 7 /  W3C group on model-based interfaces http://www.w3.org/2005/Incubator/model-based-ui/charter/  EU Artemis SMARCOS Project http://www.smarcos- project.eu/  Forthcoming S i F th i Spinger Book on Mi t B k Migratory Interactive Applications in Ubiquitous Environments Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – 62 http://hiis.isti.cnr.it 31
  32. 32. Don’t Forget EICS 2011 ! Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – 63 http://hiis.isti.cnr.it 32

×