An Intelligent Editor for Multi-Presentation User Interfaces Benoît Collignon 1 , Jean Vanderdonckt 1 , Gaëlle Calvary 2 1 Université catholique de Louvain (UCL) Louvain School of Management (LSM) Belgian Laboratory of Computer-Human Interaction (BCHI) http://www.isys.ucl.ac.be/bchi 2 Université Joseph Fourier – Grenoble I, Laboratoire LIG 385, rue de la Bibliothèque BP 53 - F-38041 Grenoble Cedex 9 (France)

Introduction The problem of designing multi-target user interfaces Context of use = {user, computing platform, physical environment} Multiple contexts of use => multiple variations of users, platforms, and environments Independently Simultaneously Notion of plasticity Plasticity = user interface ability to change according to a change of the context of use while preserving predefined usability properties Context-aware adaptation = mere adaptation depending on context changes Plasticity = more than context-aware adaptation since usability should be guaranteed at a minimum level of satisfaction To address the problem, we combine Model-driven engineering of user interfaces Plasticity as a mean to address context-aware adaptation

The problem of designing multi-target user interfaces

Context of use = {user, computing platform, physical environment}

Multiple contexts of use => multiple variations of users, platforms, and environments

Independently

Simultaneously

Notion of plasticity

Plasticity = user interface ability to change according to a change of the context of use while preserving predefined usability properties

Context-aware adaptation = mere adaptation depending on context changes

Plasticity = more than context-aware adaptation since usability should be guaranteed at a minimum level of satisfaction

To address the problem, we combine

Model-driven engineering of user interfaces

Plasticity as a mean to address context-aware adaptation

Multiplicity of contexts of use TV is multi-media family device #1 Family Device Booking notification Everywhere connectivity for simple data exchange Travelling Travel booking site Powerful interface for complex operations Working Multimedia Travel programme Sporting Experience Role Location

Example of a Plastic User interface Property of plasticity = adaptation to the context of use while preserving predefined usability properties of interest [Grolaux et al.,2002]

Property of plasticity = adaptation to the context of use while preserving predefined usability properties of interest

Example of a multi-target UI Traditional approach: Atomica HTML/Java UI code Target 1: Web terminal Target 2: Personal computer Visual C++ UI code Target 3: Pocket PC Mobile Visual Basic UI

Traditional approach: Atomica

Related work Some solutions to address this problem: The mediator solution (model-based approach) [Eisenstein et al.,2001]

Some solutions to address this problem:

The mediator solution (model-based approach)

Related work Some solutions to address this problem: Multiple variations based on a model (here, a platform model) [Märtin et al.,1990]

Some solutions to address this problem:

Multiple variations based on a model (here, a platform model)

Related work Some solutions to address this problem: Adaptive UI layout: widgets are (un)displayed and laid out according to resolution [Plomp & Keranen,2002]

Some solutions to address this problem:

Adaptive UI layout: widgets are (un)displayed and laid out according to resolution

Related work Some solutions to address this problem: Artistic resizing : widgets have different variations according to resolution [Dragicevic et al.,2005] w h w L h L y L x L r h B w B  x L = (w-w L ) / 2  y L = (h-h L ) / 2  w L = 20 h L = 10  w B = 5 h B = 5  r = 20

Some solutions to address this problem:

Artistic resizing : widgets have different variations according to resolution

 x L = (w-w L ) / 2

 y L = (h-h L ) / 2

 w L = 20

h L = 10

 w B = 5

h B = 5

 r = 20

Our solution Combine model-driven engineering and plasticity domains

Combine model-driven engineering and plasticity domains

Case study Case study: multi-target plastic UI for date & time (FlexClock) [Grolaux et al.,2002]

Case study: multi-target plastic UI for date & time (FlexClock)

Case study Dialogue modeled through a Moore machine 16 alternative variations 6 resizing operations Observations Visualization is hard to achieve

Dialogue modeled through a Moore machine

16 alternative variations

6 resizing operations

Observations

Visualization is hard to achieve

W1 HH:MM W2 HH:MM:SS W3 HH:MM JJ/MM W4 HH:MM:SS JJ/MM/AAAA W5 HH:MM:SS DAY JJ/MM/AAAA W6 HH:MM:SS DAY JJ MONTH AAAA W7  W8 HH:MM  JJ/MM W9 HH:MM:SS  JJ/MM/AAAA W10 HH:MM:SS  DAY JJ/MM/AAAA W11 HH:MM:SS  DAY JJ MONTH AAAA W13  HH:MM:SS DAY JJ/MM/AAAA W12  HH:MM:SS JJ/MM/AAAA W14  HH:MM:SS DAY JJ MONTH AAAA W15  HH:MM:SS DAY JJ MONTH AAAA CALENDAR W16 HH:MM:SS  DAY JJ MONTH AAAA CALENDAR

W1 HH:MM W2 HH:MM:SS W3 HH:MM JJ/MM W4 HH:MM:SS JJ/MM/AAAA W5 HH:MM:SS DAY JJ/MM/AAAA W6 HH:MM:SS DAY JJ MONTH AAAA W7  W8 HH:MM  JJ/MM W9 HH:MM:SS  JJ/MM/AAAA W10 HH:MM:SS  DAY JJ/MM/AAAA W11 HH:MM:SS  DAY JJ MONTH AAAA W13  HH:MM:SS DAY JJ/MM/AAAA W12  HH:MM:SS JJ/MM/AAAA W14  HH:MM:SS DAY JJ MONTH AAAA W15  HH:MM:SS DAY JJ MONTH AAAA CALENDAR W16 HH:MM:SS  DAY JJ MONTH AAAA CALENDAR

W1 HH:MM W2 HH:MM:SS W3 HH:MM JJ/MM W4 HH:MM:SS JJ/MM/AAAA W5 HH:MM:SS DAY JJ/MM/AAAA W6 HH:MM:SS DAY JJ MONTH AAAA W7  W8 HH:MM  JJ/MM W9 HH:MM:SS  JJ/MM/AAAA W10 HH:MM:SS  DAY JJ/MM/AAAA W11 HH:MM:SS  DAY JJ MONTH AAAA W13  HH:MM:SS DAY JJ/MM/AAAA W12  HH:MM:SS JJ/MM/AAAA W14  HH:MM:SS DAY JJ MONTH AAAA W15  HH:MM:SS DAY JJ MONTH AAAA CALENDAR W16 HH:MM:SS  DAY JJ MONTH AAAA CALENDAR

W1 HH:MM W2 HH:MM:SS W3 HH:MM JJ/MM W4 HH:MM:SS JJ/MM/AAAA W5 HH:MM:SS DAY JJ/MM/AAAA W6 HH:MM:SS DAY JJ MONTH AAAA W7  W8 HH:MM  JJ/MM W9 HH:MM:SS  JJ/MM/AAAA W10 HH:MM:SS  DAY JJ/MM/AAAA W11 HH:MM:SS  DAY JJ MONTH AAAA W13  HH:MM:SS DAY JJ/MM/AAAA W12  HH:MM:SS JJ/MM/AAAA W14  HH:MM:SS DAY JJ MONTH AAAA W15  HH:MM:SS DAY JJ MONTH AAAA CALENDAR W16 HH:MM:SS  DAY JJ MONTH AAAA CALENDAR

W1 HH:MM W2 HH:MM:SS W3 HH:MM JJ/MM W4 HH:MM:SS JJ/MM/AAAA W5 HH:MM:SS DAY JJ/MM/AAAA W6 HH:MM:SS DAY JJ MONTH AAAA W7  W8 HH:MM  JJ/MM W9 HH:MM:SS  JJ/MM/AAAA W10 HH:MM:SS  DAY JJ/MM/AAAA W11 HH:MM:SS  DAY JJ MONTH AAAA W13  HH:MM:SS DAY JJ/MM/AAAA W12  HH:MM:SS JJ/MM/AAAA W14  HH:MM:SS DAY JJ MONTH AAAA W15  HH:MM:SS DAY JJ MONTH AAAA CALENDAR W16 HH:MM:SS  DAY JJ MONTH AAAA CALENDAR

W1 HH:MM W2 HH:MM:SS W3 HH:MM JJ/MM W4 HH:MM:SS JJ/MM/AAAA W5 HH:MM:SS DAY JJ/MM/AAAA W6 HH:MM:SS DAY JJ MONTH AAAA W7  W8 HH:MM  JJ/MM W9 HH:MM:SS  JJ/MM/AAAA W10 HH:MM:SS  DAY JJ/MM/AAAA W11 HH:MM:SS  DAY JJ MONTH AAAA W13  HH:MM:SS DAY JJ/MM/AAAA W12  HH:MM:SS JJ/MM/AAAA W14  HH:MM:SS DAY JJ MONTH AAAA W15  HH:MM:SS DAY JJ MONTH AAAA CALENDAR W16 HH:MM:SS  DAY JJ MONTH AAAA CALENDAR

W1 HH:MM W2 HH:MM:SS W3 HH:MM JJ/MM W4 HH:MM:SS JJ/MM/AAAA W5 HH:MM:SS DAY JJ/MM/AAAA W6 HH:MM:SS DAY JJ MONTH AAAA W7  W8 HH:MM  JJ/MM W9 HH:MM:SS  JJ/MM/AAAA W10 HH:MM:SS  DAY JJ/MM/AAAA W11 HH:MM:SS  DAY JJ MONTH AAAA W13  HH:MM:SS DAY JJ/MM/AAAA W12  HH:MM:SS JJ/MM/AAAA W14  HH:MM:SS DAY JJ MONTH AAAA W15  HH:MM:SS DAY JJ MONTH AAAA CALENDAR W16 HH:MM:SS  DAY JJ MONTH AAAA CALENDAR W1 HH:MM W2 HH:MM:SS W3 HH:MM JJ/MM W4 HH:MM:SS JJ/MM/AAAA W5 HH:MM:SS DAY JJ/MM/AAAA W6 HH:MM:SS DAY JJ MONTH AAAA W7  W8 HH:MM  JJ/MM W9 HH:MM:SS  JJ/MM/AAAA W10 HH:MM:SS  DAY JJ/MM/AAAA W11 HH:MM:SS  DAY JJ MONTH AAAA W13  HH:MM:SS DAY JJ/MM/AAAA W12  HH:MM:SS JJ/MM/AAAA W14  HH:MM:SS DAY JJ MONTH AAAA W15  HH:MM:SS DAY JJ MONTH AAAA CALENDAR W16 HH:MM:SS  DAY JJ MONTH AAAA CALENDAR W1 HH:MM W2 HH:MM:SS W3 HH:MM JJ/MM W4 HH:MM:SS JJ/MM/AAAA W5 HH:MM:SS DAY JJ/MM/AAAA W6 HH:MM:SS DAY JJ MONTH AAAA W7  W8 HH:MM  JJ/MM W9 HH:MM:SS  JJ/MM/AAAA W10 HH:MM:SS  DAY JJ/MM/AAAA W11 HH:MM:SS  DAY JJ MONTH AAAA W13  HH:MM:SS DAY JJ/MM/AAAA W12  HH:MM:SS JJ/MM/AAAA W14  HH:MM:SS DAY JJ MONTH AAAA W15  HH:MM:SS DAY JJ MONTH AAAA CALENDAR W16 HH:MM:SS  DAY JJ MONTH AAAA CALENDAR

Case study Dialogue modeled through a Mealy machine W2 W1 W2 W2 W2 W2 W2 W1 2 / 1 1 11 / 1 2 17 / 13 3 30/15 4 19 / 9 5 42 / 16 6 98 / 16 7 102 / 16 8

Dialogue modeled through a Mealy machine

Case study Main idea: design one UI variation for one context at a time, connect related variations afterwards Horizontal screen resolution Vertical screen resolution Plasticity domain

Main idea: design one UI variation for one context at a time, connect related variations afterwards

Case study Main idea: design one UI variation for one context at a time, connect related variations afterwards

Main idea: design one UI variation for one context at a time, connect related variations afterwards

Tool support: PlastiXML Graphical editor for plasticity domains Direct manipulation of plasticity domains (here, rectangles) Assignment of a UI variation to each plasticity domain Automated generation of UsiXML for Layout Navigation between variations <contextModel id=&quot;test-contextModel_14&quot; name=&quot;test-contextModel&quot;> <contextRessource contextId=&quot;cont1&quot;> <Environment id=&quot;envir1&quot; name=&quot;envir1&quot; … /> <userStereotype id=&quot;ustr1&quot; stereotypeName=&quot;ustr1&quot; … /> <Platform id=&quot;plat1&quot; name=&quot;plat1&quot;> <HardwarePlatform …screenSize=&quot;640x480&quot;/> <SoftwarePlatform … /> <NetworkCharacteristics … /> <WapCharacteristics … /> <BrowserUA … /> </Platform> </contextRessource> <PlasticityDomainSet id=&quot;plasts1&quot;> <PlasticityDomain id=&quot;plast1&quot; contextId=&quot;cont1&quot; > <PlatformPlasticityDomain id=&quot;platp1&quot; PlatformId=&quot;plat1&quot; > <ScreenSizeAspect id=&quot;scrd1&quot; shape=&quot;rectangle&quot; corners=&quot;{(80,10);(80,30);(ScreenSizeXLimit,10); (ScreenSizeXLimit,30)}&quot; allowedOperations=&quot;{vertical shrinkage}&quot;/> </PlatformPlasticityDomain> </PlasticityDomain> <PlasticityDomain id=&quot;plast2&quot; contextId=&quot;cont1&quot; > <PlatformPlasticityDomain id=&quot;platp2&quot; PlatformId=&quot;plat1&quot; > <ScreenSizeAspect id=&quot;scrd2&quot; shape=&quot;polygon&quot; corners=&quot;{(80,30);(ScreenSizeXLimit,30); (ScreenSizeXLimit,60);(100,60); (100,ScreenSizeYLimit); (80,ScreenSizeYLimit)}&quot; allowedOperations=&quot;{vertical shrinkage}&quot;/> </PlatformPlasticityDomain> </PlasticityDomain> </PlasticityDomainSet> </contextModel> < mappingModel id=&quot;map1&quot; name=&quot;map1&quot; …> <isShapedFor id=&quot;adp1&quot; name=&quot;adp1&quot;> <source sourceId=&quot;W2&quot;/> <target targetId=&quot;plast1&quot;/> </isShapedFor> <isShapedFor id=&quot;adp2&quot; name=&quot;adp2&quot;> <source sourceId=&quot;W4&quot;/> <target targetId=&quot;plast2&quot;/> </isShapedFor> <source sourceId=&quot;W12&quot;/> <target targetId=&quot;plast4&quot;/> </isShapedFor> </mappingModel>

Graphical editor for plasticity domains

Direct manipulation of plasticity domains (here, rectangles)

Assignment of a UI variation to each plasticity domain

Automated generation of UsiXML for

Layout

Navigation between variations

Scenario First plasticity domain

First plasticity domain

Scenario Second plasticity domain

Second plasticity domain

Scenario Third plasticity domain

Third plasticity domain

Running applications Example #1: country selector

Example #1: country selector

Running applications Example #2: multi-presentation medical teaching application

Example #2: multi-presentation medical teaching application

Running applications Example #3: multi-presentation simple information systems

Example #3: multi-presentation simple information systems

Conclusion (1/2) Advantages One UI variation is assigned to one context (here, a resolution) at a time Connect related variations afterwards Visual design of connected variations insted of separate (unrelated) design Automated generation of UsiXML specifications Presentation Navigation Mealy machine Moore machine Limitations Apart from copy/paste, no direct reuse of components from one UI variation to another No factoring out of common components Possible extension Generate a default UI variation to feed the various resolutions to be supported Edit afterwards Generalization needed

Advantages

One UI variation is assigned to one context (here, a resolution) at a time

Connect related variations afterwards

Visual design of connected variations insted of separate (unrelated) design

Automated generation of UsiXML specifications

Presentation

Navigation

Mealy machine

Moore machine

Limitations

Apart from copy/paste, no direct reuse of components from one UI variation to another

No factoring out of common components

Possible extension

Generate a default UI variation to feed the various resolutions to be supported

Edit afterwards

Generalization needed

Conclusion (2/2) Feedback from designers Like the visual, colourful design aspects Dislike the modelling aspects (fortunately generated by software) Limited to GUIs for the moment Feedback from users Adaptation always induce some user disruption No explanation of the adaptation But would it be better is explanation is provided

Feedback from designers

Like the visual, colourful design aspects

Dislike the modelling aspects (fortunately generated by software)

Limited to GUIs for the moment

Feedback from users

Adaptation always induce some user disruption

No explanation of the adaptation

But would it be better is explanation is provided

Thank you very much for your attention For more information and downloading, http://www.isys.ucl.ac.be/bchi http://www.usixml.org User Interface eXtensible Markup Language http://www.similar.cc European network on Multimodal UIs Special thanks to all members of the team!