Search

Model-driven engineering of user interfaces

Loading...

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

1 comments

Comments 1 - 1 of 1 previous next Post a comment

Post a comment
Embed Video
Edit your comment Cancel

Notes on slide 1

6 Favorites

Model-driven engineering of user interfaces - Presentation Transcript

  1. Model-Driven Engineering of User Interfaces Jean Vanderdonckt Université catholique de Louvain (UCL) Louvain School of Management (LSM) Information Systems Unit (ISYS) Belgian Laboratory of Computer-Human Interaction (BCHI) http://www.isys.ucl.ac.be/bchi
  2. Course outline
    • Day 1: Why do we need to model the user interface?
      • H1: Motivations and introduction to usability evaluation of user interfaces
        • Case studies and evaluation
      • H2: Concepts used for usability evaluation for every UI
        • Usability guideline, ergonomic criteria, IFIP properties
    • Day 2: What do we need to model to cover UI aspects? (Part 1)
      • H1: The Cameleon reference framework for multi-target UIs
        • Underlying metamodel
        • Terminology and revisitation of IEEE Taxonomy of RE
        • Task modelling, domain modelling in IdealXML
      • H2: Abstract UI
        • Mapping model
        • Model-to-model transformation in IdealXML
  3. Course outline
    • Day 3: What do we need to model to cover UI aspects? (Part 2)
      • H1: Concrete UI
        • Model-to-model transformation in TransformiXML
        • Model-to-code generation in GrafiXML
        • Graph grammars and other techniques
      • H2: Final UI
        • UI rendering: by interpretation, by compilation
        • UI rendering in multiple computing platforms
    • Day 4: When do we need to model what to cover UI aspects?
      • H1: Multi-path development of UIs
        • Forward, reverse, lateral engineering
        • UI adaptation: graceful degradation
      • H2: Multiple targets
        • Multiple users, platforms, environments
  4. Course outline
    • Day 5: How do we assess the UI modelled?
      • H1: Automated evaluation of guidelines
        • Evolution of code static analysis
        • Evaluation for web and non-web applications
      • H2: Towards genuine model-checking of UIs
        • Usability advisor: natural language evaluation
        • Evaluation of usability guidelines, design rules, properties of interest
        • Final conclusion: evolution of MDA for UIs
          • Low threshold
          • High ceiling
          • Wide walls
  5. What is the situation today?
    • Technological aspects of user interfaces progress significantly faster than
      • Software engineering aspects
        • It takes time to develop a user interface with a new device, a new interaction technique
        • It takes more time to develop a toolkit
        • It takes even more time to rely on a model-driven approach
      • Usability engineering aspects
        • New user interfaces are shipped with usability problems because
          • Little or no experience
          • No past, no empirical evidence
        • Empirical experiments require a lot of resource if done carefully
    Day 1: Why? [Dragicevic et al.,2004]
  6. Adapted from [Palanque,2002] Target Applications, Domains Notations and tools User Interface Interaction Techniques 2006 TODAY No Interaction Technique Automated, batch systems Nothing Character UIs Business applications Screen definitions Graphical User Interfaces Information systems Entity-relationship Attribute model State-transition diagrams
  7. What’s the situation of today?
    • Interactive Software evolution: context of use =(U,P,E)
    time Day 1: Why? Platform User Environment Language
  8. Software evolution Day 1: Why?
  9. Diversity of users
    • Traditional users
    • People with disabilities
      • Visual, motor, cognitive
    Day 1: Why?
  10. Variety of tasks
    • Users want to determine their path on each platform
    Day 1: Why? [Forrester Research,2003]
  11. Heterogeneousness of platforms Day 1: Why?
  12. Multiplicity of contexts of use Day 1: Why? TV is multi-media family device #1 Family Device Booking notification Everywhere connectivity for simple data exchange Travelling Travel booking site Powerful interface for complex operations Working Multimedia Travel programme Sporting Experience Role Location
  13. Consequence
    • Proliferation of developments
    Day 1: Why? [Abrams et al.,2001] UI #12 UI #11 UI #10 UI #9 Application 3 UI #8 UI #7 UI #6 UI #5 Application 2 UI #4 UI #3 UI #2 UI #1 Application 1 Platform #4 Platform #3 Platform #2 Platform #1 Application 1 Application 2 Application 3 UI model #1 UI model #2 UI model #3 Platform #1 Platform #2 Platform #3 Platform #4 Platform model #1 Platform model #2 Platform model #3 Platform model #4
  14. Is this web site usable? Day 1: Why?
  15. Ergonomic criteria
    • Criteria for assessing the usability of any UI
      • A priori and/or a posteriori
      • At design and/or evaluation time
    • Structured into 8 first-level criteria
      • Compatibility
      • Consistency
      • Work load
      • Adaptation
      • Dialog control
      • Guidance
      • Error management
    • Implicit order: sorted by importance
    Day 1: Why? [Scapin & Bastien,1997] [Vanderdonckt,1995]
  16. Navigation
    • Use a reactive image as a navigation map
    Day 1: Why? No map without any relation Metaphoric map
  17. Navigation
    • Include navigation clues
    Day 1: Why? Navigation bar Landmark Global vs local diagram
  18. Navigation
    • Navigation clues should be consistent
    Day 1: Why? Inconsistencies between the menu and navigation bar Use the same clues at the same location for the same purpose
  19. Navigation
    • No more "Clic here"
    Day 1: Why? Clic here to go to the Publications page Clic here to go to the LSM page Clic here to go to the UCL page Go to : UCL / LSM / Publications
  20. Navigation
    • Do not use "Return to..."  links
    Day 1: Why? Aller à :... Button Avoid : come back, return, redo, cancel, undo, redirect Home page Page A Page B Page C Page D Page E First site Page 1 Page 2 Page 3 Page 4 Page 5 Second site
  21. Presentation
    • Information wich are semantically related should be presented together
    Day 1: Why? Related label Related label
  22. Presentation
    • Example of a web page format (before)
    Day 1: Why? Display zone of browser navigation buttons Browser status bar Area for accessing to main commands, langages, map, about, contact User category picture Site menus zone Informational contents Display zone for external links and external applications Organisation logo area Area for accessing to local commands Structure indicator
  23. Presentation
    • How to format? Position of home page link, internal links
    Day 1: Why? [Shaikh & Lenz,2006]
  24. Presentation
    • How to format? Position of search engine, advertisements, about us
    Day 1: Why? [Shaikh & Lenz,2006]
  25. Presentation
    • Example of a web page format (after)
    Day 1: Why? Logo to home page Site menus zone Informational contents External links You are here: structure indictor NL -FR-DL Title, sub-title Contact - Mentions légales – Vie privée – Médiateur - Accessibilité Log in – Site map - Help Search Update – Printer-friendly v.
  26. Simple choice Mixed domain: Know domain: Day 1: Why? [Vanderdonckt & Faulkner,2002] Unknown domain : Amount of possible values  [2,3] Amount of possible values  [8,50] Amount of possible values  [4,7] Amount of possible values  ]50,+  [ Amount of possible values  [2,3] Amount of possible values  [4,7] Amount of possible values  [8,50] Amount of possible values  ]50,+  [
  27. Choix multiple Mixed domain: Known domain: Day 1: Why? [Vanderdonckt & Faulkner,2002] Domaine inconnu : Amount of possible values  [4,7] Amount of possible values  [8,50] Amount of possible values  ]50,+  [ Amount of possible values  [2,3] Amount of possible values  [4,7] Amount of possible values  [8,50] Amount of possible values  ]50,+  [ Amount of possible values  [2,3]
  28. Graphics
    • Use graphics for headers
    Day 1: Why? Use moderately artistic fonts for headers Keep hear in a separate layer Use anti-aliasing
  29. Multimedia elements
    • The background should be appropriate for the task
    Use various horizon lines depending on the context Day 1: Why? Horizon line Example Meaning Low Brings attention to the sky, the abstract, high values High Stresses immediate results, the concrete Median Suggests balance, equilibrium, peace Diagonal Creates some instability, a feeling of moving Sharp Suggests dynamic aspects, changing, aggressivity
  30. Multimedia elements
    • Low horizon line
    Day 1: Why?
  31. Multimedia elements
    • High horizon line
    Day 1: Why?
  32. Multimedia elements
    • Median horizon line
    Day 1: Why?
  33. Multimedia elements
    • Diagonal horizon line
    Day 1: Why?
  34. Multimedia elements
    • Sharp horizon line
    Day 1: Why?
  35. Multimedia elements
    • Do prefer low-intensity, light backgrounds
    Day 1: Why? No white on black
  36. Multimedia elements
    • Do not use backgrounds with textures
    Day 1: Why? Pale backgrounds, submit and test Pages with automatically built background
    • Keep the good color combinations
    Multimedia elements Day 1: Why? [Murch,1987] Blue (94%) Black (63%) Red (25%) White (75%) Yellow (63%) Yellow (75%) White (56%) Black (44%) Black (100%) Blue (56%) Red (25%) White (75%) Yellow (63%) Cyan (25%) Blue (69%) Black (56%) Red (37%) Black (63%) White (56%) Blue (44%) Background Thin lines and text White Black Red Green Blue Cyan Magenta Yellow Background Thin lines and text White Black Red Green Blue Cyan Magenta Yellow Red (63%) Blue (63%) Black (56%)
    • Keep the good color combinations
    Multimedia elements Day 1: Why? [Murch,1987] Black (69%) Blue (63%) Red (31%) Yellow (69%) White (50%) Green (25%) Black (50%) Yellow (44%) White (44%) Black (69%) Red (63%) Blue (31%) Yellow (38%) Magenta (31%) Black (31%) Red (56%) Blue (50%) Black (44%) Blue (50%) Black (44%) Yellow (25%) Red (75%) Blue (63%) Black (50%) Background Bold lines and panels White Black Red Green Blue Cyan Magenta Yellow Black (69%) Blue (63%) Red (31%) Yellow (69%) White (50%) Green (25%) Black (50%) Yellow (44%) White (44%) Black (69%) Red (63%) Blue (31%) Yellow (38%) Magenta (31%) Black (31%) Red (56%) Blue (50%) Black (44%) Blue (50%) Black (44%) Yellow (25%) Red (75%) Blue (63%) Black (50%) Background Bold lines and panels White Black Red Green Blue Cyan Magenta Yellow Background Bold lines and panels White Black Red Green Blue Cyan Magenta Yellow
    • Avoid the bad color combinations
    Multimedia elements Day 1: Why? [Murch,1987] Yellow (100%) Cyan (94%) Blue (87%) Red (37%) Magenta (25%) Magenta (81%) Blue (44%) Green (25%) Cyan (81%) Magenta (50%) Yellow (37%) Green (62%) Red (37%) Black (37%) Green (81%) Yellow (75%) White (31%) Green (75%) Red (56%) Cyan (44%) White (81%) Cyan (81%) Background Thin lines and text White Black Red Green Blue Cyan Magenta Yellow Yellow (100%) Cyan (94%) Blue (87%) Red (37%) Magenta (25%) Magenta (81%) Blue (44%) Green (25%) Cyan (81%) Magenta (50%) Yellow (37%) Green (62%) Red (37%) Black (37%) Green (81%) Yellow (75%) White (31%) Green (75%) Red (56%) Cyan (44%) White (81%) Cyan (81%) Yellow (100%) Cyan (94%) Blue (87%) Red (37%) Magenta (25%) Magenta (81%) Blue (44%) Green (25%) Cyan (81%) Magenta (50%) Yellow (37%) Green (62%) Red (37%) Black (37%) Green (81%) Yellow (75%) White (31%) Green (75%) Red (56%) Cyan (44%) White (81%) Cyan (81%) Background Thin lines and text White Black Red Green Blue Cyan Magenta Yellow Background Thin lines and text White Black Red Green Blue Cyan Magenta Yellow
    • Avoid the bad color combinations
    Multimedia elements Day 1: Why? [Murch,1987] Yellow (95%) Cyan (75%) Blue (81%) Magenta (31%) Magenta (69%) Blue (50%) Green (37%) Cyan (81%) Magenta (44%) Yellow (44%) Green (44%) Red (31%) Black (31%) Yellow (69%) Green (62%) White (56%) Cyan (81%) Green (69%) Red (44%) White (81%) Cyan (56%) Green (25%) Background Bold lines and panels White Black Red Green Blue Cyan Magenta Yellow Yellow (95%) Cyan (75%) Blue (81%) Magenta (31%) Magenta (69%) Blue (50%) Green (37%) Cyan (81%) Magenta (44%) Yellow (44%) Green (44%) Red (31%) Black (31%) Yellow (69%) Green (62%) White (56%) Cyan (81%) Green (69%) Red (44%) White (81%) Cyan (56%) Green (25%) Background Bold lines and panels White Black Red Green Blue Cyan Magenta Yellow Background Bold lines and panels White Black Red Green Blue Cyan Magenta Yellow
  41. Multimedia elements
    • Consider font variations induced by different platforms
    Day 1: Why? Fixed size font Label on top of field Same fonts Two browsers
  42. IFIP Quality Properties for UIs
    • Ten properties in 3 categories
    • Category 1: information presentation
      • Multiplicity of devices: keyboard, mouse, tablet,…
      • Multiplicity of representations: alternate representations both in input and output, honesty
      • Input/output reusability (use output produced by one action as input for another)
    • Category 2: ordering of task planning
      • Multiplicity of user roles
      • Multiplicity of execution paths: users should be able to be engaged in different tasks simultaneously
      • Non-preemptiveness: degree of freedom for the user to decide what’s next
      • Reachability: possibility to navigate in the system (undo, redo)
      • Observability vs Browsability
    • Category 3: dialog adaptation
      • Reconfigurability: system ability to support user personalization
      • Adaptivity: system ability to support automated adaptation
      • Migrability: system ability to transfer responsibility from one user to another, among users, among users and systems
      • Plasticity: system ability to adapt to the context of use while preserving predefined usability properties
    • Dix’s Principle of Effort Maximility
      • A complex action should be easy to do, but complex to undo
    Day 1: Why? [Gram & Cockton,1986]
  43. IFIP Quality Properties for UIs
    • Multiplicity of devices and representations
    Day 1: Why? [Stephanidis,2001]
  44. Derivation panel with preview CTTE Operator Operator parameters Design comment resulting from Applying guidelines Design Preview that dynamically changes according to parameters Legends Day 1: Why? [Vanderdonckt et al.,2003]
  45. Plastic UI
    • Example: the Virtual keyboard
    Day 1: Why? [Grolaux et al.,2001]
  46. Plastic UI
    • Plastic UI: adaptable bounded value
    Day 1: Why? [Vanderdonckt et al.,2005]
  47. Plastic User interface
    • Property of plasticity = adaptation to the context of use while satisfying predefined usability properties of interest
    Day 1: Why? [Grolaux et al.,2002]
  48. How to address this issue?
    • Capture essence of concepts through models
      • Separation of concerns, Correlability
      • Parsability, editability
      • If possible, human readability
    • Typical models
    Day 2: What? [Calvary et al.,2003]
  49. Typical models
  50. What do we want to get? Final user Interface T Concrete user Interface T Task and Domain T Abstract user Interface T T=Target context of use Concrete user Interface S Final user Interface S Task and Domain S Abstract user Interface S S=Source context of use http://www.plasticity.org UsiXML unsupported model UsiXML supported model User S User T Day 2: What? [Calvary et al.,2003] Environment T Reification Abstraction Reflexion Translation Platform S Environment S Platform T
  51. Software engineering interpretation
    • Different types of engineering
      • Forward engineering
      • Reverse engineering
      • Lateral engineering
      • Diagonal engineering (with or without shortcuts)
    • Different approaches
      • Top-down
      • Bottom-up
      • Middle-out
    • Different development paths
      • Example: Round-trip engineering = composition of
        • Reification CUI -> FUI
        • Reflexion FUI -> FUI
        • Abstraction FUI -> CUI
        • Reflexion CUI -> CUI
  52. IEEE Reverse Engineering Taxonomy Reference: Elliot J. Chikofsky , James H. Cross II, Reverse Engineering and Design Recovery: A Taxonomy, IEEE Software, v.7 n.1, p.13-17, January 1990: http://labs.cs.utt.ro/labs/acs/html/resources/ReengineeringTaxonomy.pdf
    • Reengineering
    • "the examination of a subject system to reconstitute it in a new form and the subsequent implementation of the new form.“
    • Restructuring
    • "a transformation from one form of representation to another at the same relative level of abstraction." The new representation is meant to preserve the semantics and external behavior of the original.
    • Redocumentation
    • "a form of restructuring where the resulting semantically-equivalent representation is an alternate view intended for a human audience.“
    • Design recovery
    • "a subset of reverse engineering in which domain knowledge, external information, and deduction or fuzzy reasoning are added to the observations of the subject system." The objective of design recovery is to identify meaningful higher-level abstractions beyond those obtained directly by examining the system itself
    [Chikofsky & Cross,1990]
  53. Revisitation [Bouillon,2006]
  54. Multi-Path Development of UI
    • Multi-path development qualifies a methodology that support the realization of multiples types of development paths withing a single framework
    Forward engineering Reverse Engineering Adaptation Any Relevant Composition [Limbourg,2004] Development step Development step Development path Development path * 1 Development Scenario Development Scenario * * Development step Development step Development path Development path * 1 Development Scenario Development Scenario * *
  55. Our goals
    • Objectives
      • Description of interactive systems
        • Using pre existing domain specific meta-models
        • Used both at design and run time
      • Capitalization
        • Properties
        • Transformations
    Interactive system Model of the IS Model of the IS ‘ Interactive system ‘ Checks of properties Transformation Models, instances of Meta-Models described in MOF (even for properties and transformations…)
  56. UsiXML
    • UsiXML =
      • USer Interface exTensible Markup Language
      • XML-compliant specification language for user interfaces suitable for any interface
        • Web
        • Java
        • Windows-based applications
        • Multimodal applications, 3D applications
        • Virtual, mixed reality applications
      • http://www.usixml.org
      • Join the UsiXML Consortium by registering on line
      • Download the CD image from http://www.usixml.org/index.php?download= UsiXML RelOne.iso
  57. UsiXML = User Interface eXtensible Markup Language
    • What is UsiXML?
      • It is a XML-compliant User Interface Description Language
      • Publicly available from http://www.usixml.org
      • Free to use, open for access, easy to expand
      • Definition of the language
    UML Class Diagrams UsiXML Reference manual XSD XML Schema Descriptions UsiXML Models
  58. Adapted from [Palanque,2002] Target Applications, Domains Notations and tools User Interface Interaction Technique 2006 TODAY 2007 No Interaction Technique Automated, batch systems Nothing Character UIs Business applications Screen definitions Graphical User Interfaces Information systems Entity-relationship Attribute model State-transition diagrams Multi-platform User Interfaces Web, Java applications Task model, context model, UML ,…
  59. The problem
    • To develop user interfaces (UIs) simultaneously for multiple contexts of use
    • A context of use = triple
      • User
      • Computing platform
      • Surrounding environment
        • Organisation
        • Socio-psychological factors
    [Calvary et al.,2003]
  60. The problem
    • Why is it difficult?
      • For the designer:
        • Multiplicity of contexts of use
        • No systematic design method
      • For the user:
        • Poor usability of resulting UI
        • UI not adapted to the genuine context of use
      • For the developer:
        • Increase of development and maintenance costs
        • Increasing development complexity
        • Little of no factoring out of common elements
  61. The problem
    • Why should it be systematic?
      • Some consider it noble to have a method
      • Other consider it noble not to have a method
      • Not to have a method is bad
      • But to stop entirely at any method is worse still
      • One should at first observe rules severely, then change them in an intelligent way
      • The aim of possessing method is to seem finally as if one had no method
    [Lao Tch’ai Tche, many many years ago]
  62. Mono-platform UI
    • Traditional approach
      • Visual development
  63. Mono-platform UI
    • Advantages of traditional approach
      • UI is graphical by nature
      • A UI prototype can be
        • Rapidly produced
        • Easily modified
        • Visually demonstrated
  64. Mono-platform UI
    • Shortcomings of traditional approach
      • No structured method for developing a UI
      • All the knowledge remain implicit
      • Modification can lead to unstructured design
      • Little or no tool support for iterative design
        • Selection of widgets can be inappropriate
        • UI Layout can be tedious, repetitive
        • Problem of the spaghetti of callbacks
      • Early prototyping is hard to achieve and time consuming
      • Limited reusability (throw away)
  65. Mono-platform UI
    • Shortcomings of traditional approach
      • Limited use of UI builder
    [Szekely,1996]
    • Table with dynamic data
    • Gantt chart
    • Direct manipulation
    Desired interface Obtained interface
    • Menu bar and pull-down menus
    • Toolbar
    • Scrollbars
  66. Mono-platform UI
    • Interface builders cannot build their own UI
    [Szekely,1996]
  67. Mono-platform UI
    • Any development method (or methodology) is decomposed into 3 axes:
      • Models : explicitly capture knowledge about UI and Interactive Applications with appropriate abstractions
      • Method : structures the definition and use of underlying models in a stage-wise approach
      • Supporting tools : support the use of the method by providing tools for models and their related operations. Ideally, one model should be supported by at least one tool
    [Bodart,1989]
  68. Mono-platform UI
    • Goal: to integrate all three facets
    Interface 1 Method Model 1 Model … Model n Models Model Model Model Tools
  69. MDE based on UsiXML Model to Model Platform Independent Model (PIM) Platform Specific Model (PSM) Model to Code Source code MDA Components Techniques proposed based on UsiXML Computing Independent Model (CIM) Model to Model UsiXML model: Abstract user interface UsiXML model: Concrete user interface Rendering Final user interface UsiXML models: task, domain Graph transformations Graph transformations
  70. CIM Step 1: Task model
  71. New Abstraction: the user’s task
    • Task = set of actions carried out by a user in a given context to reach a goal
    • Logical decomposition of task into sub-tasks
    • Temporal ordering: LOTOS operators (in CTTE)
      • T1 >> T2 Enabling
      • T1[ ]>>T2 Enabling + information passing
      • T1 |> T2 Suspend/resume
      • T1 [ ] T2 Non-deterministic choice
      • T1  T2 Deterministic choice
      • T1 [> T2 Disabling (e.g. Form submit)
      • T1 |=| T2 Independence (any order, but finished)
      • T1* Iteration
      • T1{n} Finite iteration
      • T1 ||| T2 Concurrency
      • T1 |[x]| T2 Concurrency + information passing
      • [T] Optional
      • T Recursion
    [Markopoulos,1992]
  72. New Abstraction: the user’s task
    • Task definition = action + object
      • Action types
        • CRUD pattern: create, read, update, delete
        • Select, control,…
        • Acquire, render, modify, publish, compute, derive,…
      • Object types:
        • Element, list, table, collection, compound,…
  73. New Abstraction: the task meta-model [Limbourg,2004]
  74. CIM Step 2: domain model [Limbourg,2004]
  75. CIM Step 3: Task-domain mappings IdealXML [Limbourg,2004]
  76. New Abstraction: the abstract UI
    • Different CIOs can be used for the same purpose, but with different interaction modalities
    • Definition
      • Abstract Container = set of Abstract Individual Component
      • AIC = abstraction of CIOs of the same type, but independently of any interaction modality
      • Abstract User Interface (AUI) = decomposition into AC+AIC
    [Vanderdonckt & Bodart,1993]
  77. Abstraction: the abstract UI
  78. Abstraction: the abstract UI
    • Notation: based on L. Constantine’s notation for canonical abstract prototypes
    IdealXML [Montero et al.,2005] [Constantine,2003] Abs.container Abs. component Input Output Navigation Control Select
  79. IdealXML [Montero,2005]
  80. Example of AUI produced
  81. Mapping the models [Montero,2005] triggers (tg): { , } x updates (up): x observes (ob): x isExecutedIn (ex): x manipulates (ma): { , } x These mappings can be established:
  82. Mapping the models
    • Mapping the models with a mapping model (!!)
  83. Your turn now!
    • Virtual Polling System
      • Characterization of a participant
        • Age
        • Gender
        • Region
        • Country
      • Polling
        • Series of questions & answers
        • Each answer may have
          • Simple choice
          • Multiple choices
        • More capabilities…
  84. Example solution (first variant)
  85. Example solution (first variant)
  86. Task and domain models [Montero,2005]
  87. Typed Model-to-Model Transformation Uses language Meta-Meta-Model Graph Structure is instance of Meta-Model Our Meta-Model Meta-Model Subset 1 e.g., Task+Domain Model is instance of Meta-Model Subset 2 e.g., Concrete UI Model is instance of Initial UI Model e.g., MyTaskAndDomainModel Resultant UI Model e.g., MyConcreteUIModel Transformation Rule Our transformation catalog [Limbourg,2004]
  88. Expression of models as graphs
    • All transformations are in UsiXML
      • Each model = instance of meta-model
      • Each model = graph as instance of graph type
      • Each model transformation =
        • graph transformation
        • Set of productions
  89. Definition of a production
    • Find an occurrence of LHS in G (this occurrence is called a match). If several occurrences exist, choose one non-deterministically.
    • Check preconditions of both type PAC and NAC. If not verified, then skip.
    • Remove the part of G which corresponds to (LHS – K), where K is the morphism specified between LHS and RHS.
    • Add RHS – K into G – (LHS – K) as it is given by the corresponding relation between RHS – K and K
    • Check postconditions of both type PAC (and notably that the resulting graph is properly typed) and NAC. If not verified, then undo the transformation rule.
    LHS G RHS G’ r m m* r* LHS G RHS G’ r m m* r*
  90. Transformation system [Limbourg,2004] G Host USIXM specification G’ Resultant USIXM specification LHS RHS Matches - Co-Matches Is Transformed Into Is Transformed Into Transformation Rule 1 Transformation Rule 2 … Transformation Rule N Transformation System NAC Not Matches +
  91. PIM step: task+domain to AUI
    • Abstract UI (AUI) = UI independent of any interaction modality
    • Definition of AUI structure in terms of Abtract Containers (AC)
      • Which tasks should be logically grouped?
    • Definition of Abstract Individual Components (AIC) types
      • Which « functionnality » should assume AICs and what data do they manipulate ?
    • Definition of spatio-temporal arrangement
      • How should AIC be arranged in space and time ?
    • Definition of dialog control
      • What is the valid flow of action on AICs ?
    UsiXML model: Abstract user interface UsiXML models: task, domain Graph transformations
  92. PIM step: task+domain to AUI Identification of AUI structure Spatio-Temporal Arrangement of AIOs Selection of AIC Definition of Abstract Dialog Control STEP : From Task & Domain to AUI SUB-STEPS Derivation of AUI to Domain Relationships
  93. How to read a graph transformation? Node type Node (Attribute,value) Edge type Edge (Attribute,value) Node Edge
  94. Rule 1 LHS RHS ::= Ø
  95. Rule 2 LHS RHS ::=
  96. Rule 3 LHS RHS ::=
  97. Rule 4 LHS RHS ::= NAC
  98. Rule 5 LHS RHS ::= NAC1 NAC2
  99. Rule 6 LHS RHS ::=
  100. Rule 7 LHS RHS ::= PAC “ X < 3000”
  101. Rule 8 LHS RHS ::= PAC “ X > 3000” NAC
  102. Rule 9 LHS RHS ::=
  103. Rule 10 PAC LHS RHS ::= “ X > Y”
  104. PIM sub-step 1: Definition of AUI structure (AC)
  105. PIM sub-step 1: Definition of AUI structure (AC)
  106. Facet type
  107.  
  108. AC Participate to OpinionPoll AC Answer Poll AC Answer Questionnaire AC Answer Question AIC (Input/Single Selection) Select Proposition AIC (Output) See Statistics AIC (Input/Single Selection ) Chose Poll AIC (Control) Validate Question AIC (Output) Provide Personal Data AIC (Output) Read Question AIC (Input/Single Selection) Chose Questionnaire
  109. PIM sub-step 2: definition of AIC types AC = Abstract Container AIC = Abstract Individual Component CIC = Concrete Interaction Component
  110. PIM sub-step 3: Definition of spatio-temporal arrangement
  111. PSM Step: AUI to CUI
    • Concrete UI (CUI) = UI independent of toolkit
    • Definition of CUI structure
      • Which AIC is a window?
    • Definition of Concrete Interaction Component (CIC) type
      • Which « widget » should represent which AIC ?
    • Definition of placement
      • What layout can be specified between CICs,…
    • Definition of navigation
      • Which container can be started or closed from which container?
    • Definition of dialog control
      • What is the valid flow of action on AIOs
    UsiXML model: Abstract user interface UsiXML model: Concrete user interface UsiXML models: task, domain Graph transformations Graph transformations
  112. PSM Step: AUI to CUI Reification of AC into CC Arrangement of CICs Selection of CIC Concrete Dialog Control Definition STEP : From AUI to CUI SUB-STEPS Definition of Navigation Derivation of CUI to Domain Relationships
  113. PSM sub-step 3: definition of navigation An example of a complex rule
  114. PSM: Concrete User Interface
  115. Example: Platform adaptation widget substitution (1) <?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot; standalone=&quot;yes&quot;?> < cuiModel name =&quot; MyModel &quot;> < version modifDate =&quot; 2004-03-24T17:09:17.402+01:00 &quot; xmlns =&quot;&quot;> 7 </ version > < authorName xmlns =&quot;&quot;> Youri </ authorName > < window height =&quot; 500 &quot; width =&quot; 600 &quot; name =&quot; Formulaire (2/5) &quot; id =&quot; window_1 &quot;> < box relativeHeight =&quot; 100 &quot; name =&quot; box1_0 &quot; id =&quot; box1_0 &quot;> < box type =&quot; vert &quot; name =&quot; boxTodo &quot; id =&quot; boxTodo &quot;> ... < box type =&quot; horiz &quot; name =&quot; box_2_2_2_1 &quot; id =&quot; box_2_2_2_1 &quot;> < textComponent defaultContent =&quot; Sexe &quot; isBold =&quot; true &quot; id =&quot; label_2 &quot;/> < radioButton groupName =“ grupo01 &quot; defaultContent =&quot; Femme &quot; defaultState =&quot; false &quot; id =&quot; radiobutton_0 &quot;/> < radioButton groupName =&quot; grupo01 &quot; defaultContent =&quot; Homme &quot; defaultState =&quot; true &quot; id =&quot; radiobutton_1 &quot;/> </ box > ... </ box > </ box > </ window > </ cuiModel > Excerpt for a UsiXML CUI specification
  116. Example: widget substitution (2)
  117. Example: widget substitution (3) The UsiXML graph before applying any rule
  118. Example: widget substitution (4) LHS RHS NAC Rule 1: Create a new comboBox with the same id and name as the name of the group of radioButtons.
  119. Example: widget substitution (5) Rule 1: Create a new comboBox with the same id and name as the name of the group of radioButtons. The UsiXML graph after applying the first rule
  120. Example: widget substitution (6) LHS RHS ::= Rule 2: Convert every radio button within the group “x” into an item for the comboBox “x” that we have just created thanks to Rule 1
  121. Example: widget substitution (7) Rule 2: Convert every radioButton within the group “x” into an item for the comboBox “x”, we have just created. The UsiXML graph after applying the second rule: radio buttons disappeared
  122. Example: widget substitution (8) <?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot; standalone=&quot;yes&quot;?> < cuiModel name =&quot; MyModel &quot;> < version modifDate =&quot; 2004-03-24T17:09:17.402+01:00 &quot; xmlns =&quot;&quot;> 7 </ version > < authorName xmlns =&quot;&quot;> Youri </ authorName > < window height =&quot; 500 &quot; width =&quot; 600 &quot; name =&quot; Formulaire (2/5) &quot; id =&quot; window_1 &quot;> < box relativeHeight =&quot; 100 &quot; name =&quot; box1_0 &quot; id =&quot; box1_0 &quot;> < box type =&quot; vert &quot; name =&quot; boxTodo &quot; id =&quot; boxTodo &quot;> ... < box type =&quot; horiz &quot; name =&quot; box_2_2_2_1 &quot; id =&quot; box_2_2_2_1 &quot;> < textComponent defaultContent =&quot; Sexe &quot; isBold =&quot; true &quot; id =&quot; label_2 &quot;/> < comboBox id =&quot; comboBox001 &quot; name =&quot; label_3 &quot; isDropDown =&quot; true &quot;> < item id =&quot; radiobutton_0 &quot; name =&quot; radiobutton_0 &quot; defaultContent =&quot; Femme &quot;/> < item id =&quot; radiobutton_1 &quot; name =&quot; radiobutton_1 &quot; defaultContent =&quot; Homme &quot;/> </ comboBox > ... </ box > </ box > </ window > </ cuiModel > Excerpt from the final transformated UsiXML specification
  123. Example: widget substitution (9)
  124. What do we have so far? Concrete user Interface S Final user Interface S Task and Domain S Abstract user Interface S S=Source context of use UsiXML unsupported model UsiXML supported model User S Reification Platform S Environment S
  125. Multiple development paths
  126. Mapping the Methodological World onto the Transformation World A development library stores (in usiXML textual format) paths, steps and sub-steps definition and their associated transformation systems and transformation rules Development step Development step Development sub - step Development sub - step Development path Development path Transformation System Transformation System Transformation Rule Transformation Rule isComposedOf isRealizedBy isComposedOf isComposedOf * 1 * 1 1 * 1 0..1 Development step Development step Development sub - step Development sub - step Development path Development path Transformation System Transformation System Transformation Rule Transformation Rule isComposedOf isRealizedBy isComposedOf isComposedOf * 1 * 1 1 * 1 0..1 Methodological World Development step Development step Development sub - step Development sub - step Development path Development path Transformation System Transformation System Transformation Rule Transformation Rule isComposedOf isRealizedBy isComposedOf isComposedOf * 1 * 1 1 * 1 0..1 Transformation World [Limbourg,2004]
  127. Multiple development paths Rule n Transformation System 1 Rule 1 Rule 2 … Rule n Transformation System 2 Rule 1 Rule 2 … Rule n Transformation System ... Rule 1 Rule 2 … Rule n Transformation System n Rule 1 Rule 2 … : when source terminates apply target : execute development step Development Step α
  128. TransformiXML API/GUI
    • API = set of transformations
  129. From T&D to AUI
    • TransformiXML
    TransformiXML [Bouillon et al.,2005]
  130. Final user interface
    • Two forms of UI rendering
      • Interpretation
        • By run-time static analysis and direct rendering (InterpiXML & FormiXML)
      • Code generation
        • By program synthesis (GrafiXML)
        • By generative programming (Angie)
          • Feature model
          • Components assembling
    UsiXML model: Abstract user interface UsiXML model: Concrete user interface Rendering Final user interface UsiXML models: task, domain Graph transformations Graph transformations Generative programming
  131. What is a Feature Model? C F1 C F1 F2 C F1 F2 Dependencies 1. 2. Optional Exclusive Alternate [Schlee & Vanderdonckt,2004]
  132. Example of a Feature Model Alternate [Schlee & Vanderdonckt,2004]
  133. Interpretation of a feature model C F1 F2 F3 F2a F2b F3a F3b C F1 F2 F3 F2a F2b F3a F3b 0 1 1 1 1 0 0 UsiXML Specifications <C> <F1>0</F1> <F2>1 <F2a>1</F2a> <F2b>0</F2b> </F2> <F3>1 <F3a>0</F3a> <F3b>1</F3b> </F3> </C> ABA ANGIE-Part [Schlee & Vanderdonckt,2004]
  134. Generative Programming void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void nlinFilterCommon (const int& nID, Mask*, Pl*); int pot (const unsigned int& ) const; ImgProcessT<T>& channelXor (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL); ImgProcessT<T>& chann elMix ImgProcessT<T> & channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL); V oid channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void nlinFilterCommon (const int& nID, Mask*, Pl*); int pot (const unsigned int& ) const; ImgProcessT<T>& channelXor (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL); public: ImgProcessT(const char* const psz = &quot;&quot;, Pl* ppl = NULL); ImgProcessT(const int&, const int& nWidth = 20, const int& nHeight = 20, const int& nFillColor = 0, ProgressLine* ppl = NULL ); ImgProcessT(const ImgProcessT<T>&); virtual ~ImgProcessT(); virtual char* getClassName() const; virtual char getClassID () const; ImgProcessT<T>& operator+=(const ImgProcessT<T>&); ImgProcessT<T>& operator-=(const ImgProcessT<T>&); ImgProcessT<T>& operator^=(const ImgProcessT<T>&); ImgProcessT<T> operator+ (const ImgProcessT<T>&); ImgProcessT<T> operator- (const ImgProcessT<T>&); ImgProcessT<T> operator^ (const ImgProcessT<T>&); ImgProcessT<T>& channelNLinFilter(const int&, Mask*, T*, Pl* ppl=NULL); mgProcessT<T>& operator+=(const ImgProcessT<T>&); ImgProcessT<T>& operator-=(const ImgProcessT<T>&); ImgProcessT<T>& operator^=(const ImgProcessT<T>&); ImgProcessT<T> operator+ (const ImgProcessT<T>&); ImgProcessT<T> operator- (const ImgProcessT<T>&); ImgProcessT<T> operator^ (const ImgProcessT<T>&); ImgProcessT<T>& channelNLinFilter(const int&, Mask*, T*, Pl* ppl=NULL); void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void nlinFilterCommon (const int& nID, Mask*, Pl*); int pot (const unsigned int& ) const; ImgProcessT<T>& channelXor (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL); USIXML Specifications ImgProcessT<T>& channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL) ImgProcessT<T>& channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL) Components void areaOperation(const int&, const int&, const int&, const int&, ImgProcessT<T>& (ImgProcessT<T>::*pimgpr)(const int&, Pl*), const int& nAngle, Pl* ppl ); void areaOperation(const int&, const int&, const int&, const int&, ImgProcessT<T>& (ImgProcessT<T>::*pimgpr)(Pl*), Pl* ppl ); void areaOperation(const int&, const int&, const int&, const int&, ImgProcessT<T>& (ImgProcessT<T>::*pimgpr)(const int&, const int&, const int&, const int& ), const int&, const int&, const int&, const int& ); void channelShowHistogram( int*, const int&, const int&, const int&, const int&, const int&, const int&, const int& n = 0 ) const; void channelFft ( T*, T*, T*, const unsigned int&, const unsigned int&, Pl* ppl = NULL ); void channelIfft ( T*, T*, T*, const unsigned int&, const unsigned int&, Pl* ppl = NULL ); void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void nlinFilterCommon (const int& nID, Mask*, Pl*); int pot (const unsigned int& ) const; ImgProcessT<T>& channelXor (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelMix (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSub (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL); public: ImgProcessT(const char* const psz = &quot;&quot;, Pl* ppl = NULL); ImgProcessT(const int&, const int& nWidth = 20, const int& nHeight = 20, const int& nFillColor = 0, ProgressLine* ppl = NULL ); ImgProcessT(const ImgProcessT<T>&); virtual ~ImgProcessT(); virtual char* getClassName() const; virtual char getClassID () const; ImgProcessT<T>& operator+=(const ImgProcessT<T>&); I Final code ImgProcessT<T>& channelAdd (T*, T*, Pl* ppl = NULL); ImgProcessT<T>& channelSubBalanced(T*, T*, Pl* ppl = NULL); public: ImgProcessT(const char* const psz = &quot;&quot;, Pl* ppl = NULL); ImgProcessT(const int&, const int& nWidth = 20, const int& nHeight = 20, const int& nFillColor = 0, ProgressLine* ppl = NULL ); void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); void channelFftSpectrum(T*, T*, T*, Pl* ppl = NULL); void channelRange (T*, Pl* ppl = NULL ); [Schlee & Vanderdonckt,2004]
  135. Developed by Benjamin Michotte
    • GrafiXML consists of a user interface builder for designing a graphical user interface (GUI) independently of the platform and save it in UsiXML format language.
    • Features
    • Exports GUI in
      • Java source (using Swing)
      • XHTML 1.0 Strict
      • XUL
    • Works on Windows, Linux and MacOs X
    • Available in English, French and Spanish
    • Based on plugins
      • Export
      • Import
      • Project management
      • Graceful degradation of user interfaces
    • Allows creating context models
    [Michotte,2004]
  136. Design Tab Design window Components toolbar Components options
  137. Localisation of UIs
    • GrafiXML allows the user to create multi-language GUI
    Support for mnemonics and shortcuts
  138. Preview
    • At any time, you can preview the UI in the language you want
  139. XML Editor
    • GrafiXML contains a XML editor which shows the UsiXML specification of your work
    • You can edit yourself some part of the XML
    Edit the UsiXML Show attributes A click on the tree highlights the corresponding UsiXML
  140. Plugins
    • GrafiXML works with plugins
      • Install / remove using the plug-in manager
      • Updated if needed using one click
    Click on « add » to open the downloader Choose the plugins you want install And install them Double-click on a plugin And look at the plugin informations
  141. What do we have so far? Concrete user Interface S Final user Interface S Task and Domain S Abstract user Interface S S=Source context of use UsiXML unsupported model UsiXML supported model User S Reification Platform S Environment S
  142. Reverse engineering
    • From FUI to CUI
    • From CUI to AUI, task & domain
  143. From FUI to CUI
    • Do you have the source code?
      • Markup languages (e.g., HTML): static code analysis (ReversiXML)
    • Do you have the compiled code?
      • Programming languages (e.g., compiled C): resource file extraction and static code analysis
    • Do you have the running application?
      • Video analysis: computer vision
    • Do you have only the documentation?
      • Image analysis: image processing
    [Bouillon & Vanderdonckt,2004]
  144. From compiled code [Marion,2005]
  145. The calculator
    • Begin VB.Form Calculator BorderStyle = 1 'Fixed Single Caption = &quot;Calculator&quot; ClientHeight = 2970 ClientLeft = 2580 ClientTop = 1485 ClientWidth = 3270 ClipControls = 0 'False BeginProperty Font name = &quot;System&quot; charset = 1 weight = 700 size = 9.75 underline = 0 'False italic = 0 'False strikethrough = 0 'False EndProperty Height = 3375 Icon = &quot;CALC.frx&quot;:0000 Left = 2520 LinkMode = 1 'Source LinkTopic = &quot;Form1&quot; MaxButton = 0 'False ScaleHeight = 2970 ScaleWidth = 3270 Top = 1140 Width = 3390 Begin VB.CommandButton Number Caption = &quot;7&quot; Height = 480 Index = 7 Left = 120
    <Window id=“1” name=“1” isVisible=“true” isEnabled=“true” defaultBorderTitle=“Calculator” borderWidth=“1”height=“358” width=“309”> <Box id=“2” name=“2” type =“verticall” isVisible=“true” isEnabled=“true”> <button isEnabled=“true” … USIXML [Marion,2005]
  146. From CUI to AUI, task & domain
    • Re-engineering = reverse + forward
    • Possible re-interpretation by QtkiXML
  147. Example of Derivation Rules
    • Examples:
      • For a textbox in HTML/WML
    •  x  Ts : x = input ٨ (x.type=“text” ٧ x.type=“password” ٧ x.type=NULL)  Addnode (“textComponent”, idtext)
    • where idtext= NodeAmount(Tt)
      • For a table in HTML/WML
    •  x  Ts : x = table ٨ x.border>0
    •  Addnode (“table”, idtable) where idtable = NodeAmount(Tt)
    •   x  Ts : x = table ٨ x.border=0  Addnode (“box”, idbox) where idbox = NodeAmount(Tt)
    •  AddAttribute (idbox, “type”, “vertical”)
    Input Name=in1 Maxlength=15 Value=login Type=text Element B TextComponent Name=in1 Maxlength=15 defaultContent=login Id=in1 Element Y [Bouillon,2006]
  148. Derivation rules categories
    • Rules can be classified into different categories, outlining the common issues in a RE process for various source UIs.
      • Element recovery
      • Attribute detection
      • Layout / temporal ordering analysis
      • Dialog recuperation
      • Hierarchy recovery
      • Multi-model transformations
      • Retargeting operations
     x  to Tt i ,y  to Tt 0 : x = window ٨ (y=box ٧ y=window) ٨ x.filename =y.targetfile  CloneNode(x.id, idnew, Tt0) where idnew =∑ node  Tt 0 ٨ RemoveNode(x, x.id) ٨ RemoveArc(ParentNode(x).id, x.id) ٨ z=root(Tt i ) ٨ Remove Node(z,z.id) ٨ AddArc(y.id, idnew)  x  to Tt i ,y  to Tt 0 : x = vocalGroup ٨ y=VocalGroup ٨ x.filename =y.insertFile  CloneNode(x.id, idnew, Tt0) where idnew =∑ node  Tt0 ٨ RemoveNode(x, x.id) ٨ RemoveArc(ParentNode(x).id, x.id) ٨ z=root(Tt i ) ٨ Remove Node(z,z.id) ٨ AddArc(y.id, idnew) Multi-model transformations Element 1 Element 2 Element 3 Horizontal box Horizontal box Vertical box Element 1 Element 2 Line Break Element 3  x  Tt: x=bix ٨ ((rigthSibling(x)!=table ٧ rigthSibling(x)!=bix ٧ rigthSibling(x)!=cell ٧ rigthSibling(x)!=box) ٨ rigthSibling(x)!=NULL)  CloneNode (rightSibling(x).id, idnew) ٨   RemoveNode (rightSibling(x), rightSibling(x).id) ٨ RemoveArc (ParentNode(rightSibling(x)), rightSibling(x)) ٨ AddArc(x.id, idnew) where idnew =∑ node  Tt Layout Analysis
  149. RE of Web Pages: ReversiXML
        • Written in PHP5
        • On-line RE
        • About 12,000 LOC
        • Set of libraries
  150. RE of Phone Interfaces
    • Major working hypotheses
      • Static RE of WML 1.1 and voiceXML 2.0 up to the CUI in USIXML 1.4.6
      • Recovery of the layout and hierarchy/temporal ordering.
      • Dialog-Navigation analyzed (but not scripts)
      • No retargeting operations
    • Similar method to HTML applied for WML & VoiceXML reverse engineering
      • Description of languages meta-models and derivation rules.
  151. RE of WML Phone Interfaces
    • Example
      • Prototype using XSLT + XPATH
      • Local process allowing no design alternatives
    WML <p> Yahoo! ID: <input name=&quot;login&quot; value=&quot;&quot; format=&quot;*m&quot; />Password: <input type=&quot;password&quot; name= &quot;passwd&quot; value=&quot;&quot; format=&quot;*m&quot; /> <anchor title=&quot;OK&quot;>Submit <go method=&quot;post&quot; href=&quot;/raw?&quot;> USIXML <textComponent id= &quot;textComponent_1&quot; glueHorizontal =&quot;left“ isVisible=&quot;true&quot; isEnabled= &quot;true&quot; defaultContent=&quot;Yahoo! ID: <textComponent id= &quot;textComponent_2&quot; glueHorizontal= &quot;left&quot; defaultContent=&quot;&quot; isEditable= &quot;true&quot; isVisible=&quot;true&quot;/> [Cui,2005]
  152. Results of the Evaluation
    • Case studies illustrating various HTML RE techniques
    • Study of reengineering of the Sedan-Bouillon Website thanks to two FE tools: Teresa and QtkXML
  153. Results of the Evaluation
    • With Teresa
      • RE applied without retargeting
      • USIXML CUI model used as input in Teresa that performs translations for another context of use
      • Produces the Web site designed for Pocket PC (XHTML)
  154. Results of the Evaluation
    • With QtkiXML
    Original UI Reengineered UI
  155. What do we have so far? Concrete user Interface S Final user Interface S Task and Domain S Abstract user Interface S S=Source context of use UsiXML unsupported model UsiXML supported model User S Let’s go for adaptation [Calvary et al.,2003] Reification Abstraction Platform S Environment S
  156. What do we want to get? Final user Interface T Concrete user Interface T Task and Domain T Abstract user Interface T T=Target context of use Concrete user Interface S Final user Interface S Task and Domain S Abstract user Interface S S=Source context of use UsiXML unsupported model UsiXML supported model User S User T [Calvary et al.,2003] Environment T Reification Abstraction Reflexion Translation Platform S Environment S Platform T
  157. Rules for Graceful Degradation
    • Constitution of an original corpus of rules
    • Typology of rules, following the CAMELEON framework:
      • (Rules at the Final User Interface level)
      • Rules at the Concrete User Interface level
      • Rules at the Abstract User Interface level
      • Rules at the Tasks & Concepts level
    • Structured description of these rules
    [Florins,2006]
  158. Rules at the Concrete UI level
    • Transformation of graphical objects
      • Resizing rules
      • Modification rules
      • Substitution rules
      • Removal rules
    • Transformation of graphical relationships
      • Reorientation rules
      • Moving rules
    [Florins,2006]
    • Transformation of graphical objects
      • Resizing rules
      • Modification rules
      • Substitution rules
      • Removal rules
    • Transformation of graphical relationships
      • Reorientation rules
      • Moving rules
    Rules at the Concrete UI level [Florins,2006]
  160. Rules at the Concrete UI level
    • Transformation of graphical objects
      • Resizing rules
      • Modification rules
      • Substitution rules
      • Removal rules
    • Transformation of graphical relationships
      • Reorientation rules
      • Moving rules
    [Florins,2006]
  161. Rules at the Concrete UI level
    • Transformation of graphical objects
      • Resizing rules
      • Modification rules
      • Substitution rules
      • Removal rules
    • Transformation of graphical relationships
      • Reorientation rules
      • Moving rules
    [Florins,2006]
  162. Rules at the Concrete UI level
    • Transformation of graphical objects
      • Resizing rules
      • Modification rules
      • Substitution rules
      • Removal rules
    • Transformation of graphical relationships
      • Reorientation rules
      • Moving rules
    [Florins,2006]
  163. Rules at the Concrete UI level
    • Transformation of graphical objects
      • Resizing rules
      • Modification rules
      • Substitution rules
      • Removal rules
    • Transformation of graphical relationships
      • Reorientation rules
      • Moving rules
    [Florins,2006]
  164. Rules at the Abstract UI level
    • Spitting rules
    • Consist in breaking the initial UI into chunks
    • + adding transitions
    [Florins,2006]
  165. Rules at the Abstract UI level
    • Important because:
      • Difficult and significant step: generates important changes into the very structure of the UI
      • Appreciated by users
    • Supporting algorithms developed during the thesis.
    • Originality: involve UI description at several abstraction levels
      • Can be rely on the sole CUI level
      • Can exploit information from the AUI / task models.
    [Florins,2006]
  166. Rules at the Abstract UI level Source interface (in the graphical editor GrafiXML) (b) Execution of the splitting rule (a) box box box
    • Application of the rule using CUI level information
    [Florins,2006]
  167. Rules at the Abstract UI level
    • Application of the rule using task level information
    [Florins,2006]
  168. Rules at the Tasks&Concepts level
    • Task deletion
    • Information summarization
    • …
  169. GD Plug-in (4)
    • Sections of rules
    [Florins et al.,2006]
  170. GD Plug-in (4)
    • Sections of rules
  171. GD Plug-in (4)
    • Sections of rules
  172. GD Plug-in (5)
    • Rules selection / parameters
  173. GD Plug-in (6)
    • Results
  174. Multi-platform for Emergency [Amouh et al.,2005]
  175. Multi-platform for Emergency
    • Three platforms
      • Pocket PC
      • Desktop PC
      • Wall Screens
  176. Multi-platform for Emergency
    • Model and method
      • Design the reference screen first
      • Refine the others screens later
        • By applying graceful degradation
        • By applying transformation techniques
  177. Graceful degradation [Florins & Vanderdonckt,2004]
  178. Transformation rules
  179. Transformation rules Add all >> Add > << Remove all < Remove >> > << < > < Group box Item 1 Item 2 Item 3 Item 4 Item 5 Item 6 Item 7 Item 8 Item 1 Item 2 Item 3 Item 4 Item 5 Item 6 Item 7 Item 1 Item 1 Item 2 Item 3 Item 4 Item 5 Item 6 Item 7
  180. Transformation rules
  181. Considering the context of use in designing user interfaces
    • Context of use = triple
      • User
      • Platform
      • Environment
    • Each time one of these facets change, the context changes
    Generation for multiple platforms [Plomp & Keranen,2004]
  182. Context model
  183. Adapted from [Palanque,2002] Target Applications, Domains Notations and tools User Interface Interaction Technique 2006 TODAY 2007 No Interaction Technique Automated, batch systems Nothing Character UIs Business applications Screen definitions Graphical User Interfaces Information systems Entity-relationship Attribute model State-transition diagrams Multi-platform User Interfaces Web, Java applications Task model, context model, UML ,… Virtual Reality User Interfaces 3D Applications Scene model
  184. Virtualisation of UIs
    • Going from 2D to 3D
      • Mapping widgets in 2D to 3D
    [Vanderdonckt et al., 2004]
  185. Developed by Donatien Grolaux
    • Problem: how to design a UI that takes care of multiple displays?
    • Solution: Migration = DEMIPLAT
    • DistriXML = software architecture for migrating UIs from one platform to another at run-time
    DistriXML [Grolaux & Vanderdonckt,2005] Pencil Palette Painting Painting tool
  186. Why take care of multiple displays? [Czerwinsky,2005] Effects of Display Size on Task Times 0 20 40 60 80 100 120 140 160 DISPLAY Average Task Time (Seconds) Small Large
  187. Why take care of multiple displays? The tasks were easy to perform 0 1 2 3 4 5 Small Large Display Size Average Rating (1=Disagree, 5=Agree) [Czerwinsky,2005]
  188. Why take care of multiple displays? [Czerwinsky,2005]
  189. Demonstration DistriXML [Grolaux & Vanderdonckt,2005]
  190. Demonstration using two displays from two different computers DistriXML
  191. Example using a Pocket PC DistriXML
  192. UI Migration: DEMIPLAT
    • De tach
    DistriXML
  193. UI Migration: DEMIPLAT
    • De tach - Mi grate
    DistriXML
  194. UI Migration: DEMIPLAT
    • De tach - Mi grate - Pl astify
    DistriXML
  195. UI Migration: DEMIPLAT
    • De tach - Mi grate - Pl astify - At tach
    DistriXML
  196. This is not a floating toolbar Process DistriXML
  197. This is migration ! Computer B Process Process Computer A DistriXML
  198. User model
    • User population = hierarchy of user stereotypes
    • User stereotype = set of parameters
      • Context independent
        • Age, gender, motor skills, general preference,…
      • Context dependent
        • Preference for certain contents
        • Interaction history
        • Can be initiated by the corresponding context independent parameter
        • Can overwrite this parameter
  199. Why user preference is important?
    • End users tend to manipulate more efficiently the user interface they prefer
      • When end users notice significant differences in their own performance and when they consider performance as important, then preference and performance are highly correlated.
      • Preference is more important than performance
    [Johnsgard,1995] If information density is low If information density is high Simple choice Multiple choice Input with exte n- sion Predefined sele c- tio n
  200. Example of preference-based design
    • Goal = to input the temperature of a human body
      • Solution n°1: edit box
      • Solution n°2: list box
      • Solution n°3: drop down list
      • Solution n°4: field with scroll bar
      • Solution n°5: thermometer
  201. Users
    • Properties of the user also constrain interaction.
      • Expertise
        • Does the user understand this interaction technique?
      • Privileges
        • What tasks is the user allowed to perform?
      • Preferences
        • What tasks is the user likely to perform?
  202. Modeling a range of designs
    • UI modeling languages must model a range of design possibilities to account for different contexts
    • This could be accomplished by modeling preferences instead of decisions
  203. A Spectrum of Preference Relations
    • We describe a spectrum of preference relations
      • Less flexible relations are easier to implement but less powerful
      • More flexible relations can model highly adaptive UIs, but are difficult to implement
    [Eisenstein,2001]
  204. The Spectrum
    • Concrete Preferences
      • Bindings
      • Simple Preference
      • Ordered Preference
    • Abstract Preference
    • Design guideline
  205. Modeling preferences in UsiXML
    • UsiXML = User Interface eXtensible Markup Language (http://www.usixml.org)
    User1 Domain2 Domain3 Presen- tation4 Presen- tation5 User2
  206. Bindings
    • Bindings are simple one-to-one mappings
      • Task A is implemented by interactor X
      • A is the condition
      • X is the target
    • Bindings are our point of departure,
      • Most current modeling formalisms support only bindings
  207. Binding example
    • User U1 prefers presentation element P1
    <PREFERENCE ID=&quot;G1&quot;> <CONDITIONS> <RELATION_STATEMENT DEFINITION=&quot;condition&quot; REFERENCE=&quot;U1&quot;/> </CONDITIONS> <TARGETS> <RELATION_STATEMENT DEFINITION=&quot;target&quot; REFERENCE=&quot;P1&quot;/> </TARGETS> </PREFERENCE>
  208. Simple Preferences
    • Simple Preferences are many-to-one mappings
      • For user U, task A is implemented using interactor X
      • Conditions: U, A
      • Target: X
  209. Simple preference example
    • User U1 prefers presentation element P1 for representing domain model DM1
    <PREFERENCE ID=&quot;G1&quot;> <CONDITIONS> <RELATION_STATEMENT DEFINITION=&quot;condition&quot; REFERENCE=&quot;U1&quot;/> <RELATION_STATEMENT DEFINITION=&quot;condition&quot; REFERENCE=&quot;DM1&quot;/> </CONDITIONS> <TARGETS> <RELATION_STATEMENT DEFINITION=&quot;target&quot; REFERENCE=&quot;P1&quot;/> </TARGETS> </PREFERENCE>
  210. Bindings
    • Bindings are simple one-to-one mappings
      • Task A is implemented by interactor X
      • A is the condition
      • X is the target
    • Bindings are our point of departure,
      • Most current modeling formalisms support only bindings
  211. Binding example
    • User U1 prefers presentation element P1
    <PREFERENCE ID=&quot;G1&quot;> <CONDITIONS> <RELATION_STATEMENT DEFINITION=&quot;condition&quot; REFERENCE=&quot;U1&quot;/> </CONDITIONS> <TARGETS> <RELATION_STATEMENT DEFINITION=&quot;target&quot; REFERENCE=&quot;P1&quot;/> </TARGETS> </PREFERENCE>
  212. Ordered Preferences
    • An ordered preference is a type of many-to-many mapping
      • For user U, task A should be implemented by one of the following interactors: X1, X2, or X3
        • Multiple targets, in order of preference: X1, X2, X3
        • Targets can be assigned “priorities”
          • This specifies the “level of preference”
          • E.g.: X1 = 100, X2 = 95, X3 = 20
            • Means choose either X1 or X2, it doesn’t matter much
          • E.g.: X1 = 100, X2 = 35, X3 = 35
            • Try hard to choose X1
  213. Ordered preference example
    • User U1 prefers presentation element P1 to presentation element P2, for representing domain model DM1
    <PREFERENCE ID=&quot;G1&quot;> <CONDITIONS> <RELATION_STATEMENT DEFINITION=&quot;condition&quot; REFERENCE=&quot;U1&quot;/> <RELATION_STATEMENT DEFINITION=&quot;condition&quot; REFERENCE=&quot;DM1&quot;/> </CONDITIONS> <TARGETS> <RELATION_STATEMENT DEFINITION=&quot;target&quot; REFERENCE=&quot;P1&quot;> < ATTRIBUTE_STATEMENT DEFINITION=&quot;priority&quot;> 100 </ATTRIBUTE_STATEMENT> </RELATION_STATEMENT> <RELATION_STATEMENT DEFINITION=&quot;target&quot; REFERENCE=&quot;P2&quot;> <ATTRIBUTE_STATEMENT DEFINITION=&quot;priority&quot;> 50 </ATTRIBUTE_STATEMENT> </RELATION_STATEMENT> </TARGETS> </PREFERENCE>
  214. Summary of Concrete Preferences
    • All of the previous are concrete preferences
      • They model the target directly
      • Can be combined with a model of the contextual constraints:
        • Example: For task A,
          • interactors X1, X2, and X3 are preferred, in order
          • But X1 cannot be implemented on current device
          • Therefore, X2 is used
  215. Abstract Preferences
    • Abstract preferences do not model the target directly
    • They model characteristics of the target
    • Abstract preferences have criteria for making mappings
      • Preferential, stating which is preferred
      • Logical, stating which is allowed
  216. Example of Abstract Preference
    • User U prefers the interactor
      • Occupying the least screen space
        • This is a preferential criteria
        • The behavior is to minimize the value of the criteria
      • But not less than 20 pixels wide
        • This is a logical criteria
  217. Abstract preference example
    • User U1 prefers the presentation element that occupies the least screen space
    <PREFERENCE ID=&quot;G1&quot;> <CONDITIONS> <RELATION_STATEMENT DEFINITION=&quot;condition&quot; REFERENCE=&quot;U1&quot;/> </CONDITIONS> <TARGETS> <RELATION_STATEMENT DEFINITION=&quot;target&quot; REFERENCE=&quot;P1&quot;/> <RELATION_STATEMENT DEFINITION=&quot;target&quot; REFERENCE=&quot;P2&quot;/> </TARGETS> <CRITERIA> <RELATION_STATEMENT DEFINITION=&quot;criteria&quot; REFERENCE=&quot;screen space&quot;> <ATTRIBUTE_STATEMENT DEFINITION=&quot;criteria type&quot;> preferential </ATTRIBUTE_STATEMENT> <ATTRIBUTE_STATEMENT DEFINITION=&quot;criteria behavior&quot;> minimum </ATTRIBUTE_S TATEMENT> </RELATION_STATEMENT> </CRITERIA> </PREFERENCE>
  218. Multiple Preferential Criteria
    • Preferential criteria may have “priorities”
      • This specifies their relative importance
      • E.g.: “Prefer presentation elements that require few clicks and are small, but the criteria of having few clicks is most important”
      • Here, the “number of clicks” criteria has a higher priority than the “size” criteria
  219. Design Guidelines
    • Abstract preferences consider criteria that relate to the targets
    • Design guidelines consider criteria of any object in the UI model
    • Frequently take the form of If-Then statements
    • Example: “If the experience level of User U is less than “ expert”, then map to interactor X1 ”
  220. Chaining Abstract Preferences
    • Design guidelines can have other preference relations as their targets:
      • “ If User U is not an expert, then select the navigational structure with the least cognitive complexity.”
      • In this case, the target of the design guideline is an abstract preference
        • The abstract preference has a single, preferential criteria:
          • Minimize cognitive complexity
  221. Chaining Design Guidelines
    • Design guidelines can also target other design guidelines
    • This can create complex logical formulae:
      • “ If User U is an expert, and the platform is a PDA, then choose one of X1, X2, or X3, whichever is smallest”
    • Such formulae can model highly adaptive design phenomena
  222. How to argue for preference?
    • Use the QOC notation
    Question? [McLean & Belotti,1998] Criteria 1 Criteria 2 Criteria j Criteria m Option 1 Option 2 Option i Option n = negatively affects = positively affects
  223. Preference example Problem Solution 1 Solution 2 (RE1) (RE2) suggests suggests avoids contradicts respects = for = against (A1) = drop down list requires less space than radio buttons (A2) = some possible values become obsolete when they are infrequently used (A3) = drop down list shows better then current value than the possible values (A4) = radio buttons are faster and easier to manipulate than drop down list (A5) = radio buttons are recommended in Microsoft Windows style guide (A6) = radio buttons do not allow users to change the values drop down list radio buttons (A4) is contradicted by (A2) and (A3) : the widget should be more used for output than input. (A6) is contradicted by (A3) : it is better to present all possible values than only one at a time (A5) is an autority argument , and can fall in front of (A1)+(A2)+(A3) [Vanderdonckt,1997] (A1) (A2) (A3) (A4) (A5) (A6) Standard compatibility (e.g. Windows) Cognitive respect Minimal actions Dialog representativity Prompting Clarity contradicts
  224. Theory or argumentation [Toulmin,1975] [Perelman, 1978] Design problem Design option parameter (parameter, value) Is described by Is solved by Accepted solution Set of admissible solutions Solution 1 Solution 2 Solution n Counter-argument (rebuttal, counter-claim) Argument (ground, typically data) can be rejected because of Can be accepted thanks to Is justified by Design claim Warrant Is reinforced by Qualifier satisfied unsatisfied Is related to corresponds to Is justified by Is qualified by
  225. Mixed reality
    • Mixed reality = real world + digital world
  226. Mixed reality for games [Alterface,2006]
  227. Mixed reality in surgery
    • No representation for
      • Tasks in real or virtual world
      • Tasks independent of user interaction e.g. tracking object of interest
      • Various contexts of use in Mixed Reality Systems i.e. changing point-of-view
    Adaptive IS composition based on focus of interest [Trevisan et al.,2004]
  228. Composing the MIS Real CIO Patient head Virtual CIOs Anatomic organs Virtual CIOs Annotations Primary object of interest
  229. Annotation placement strategy
    • No overlapping annotations for readability
    • Highest priority of virtual object (3)
    • Medium priority of real object (2)
    • Low priority of background (1)
    • Annotations placed from peripheral to central zones
    • Other objects as part of Background
  230. Context-aware Multimodal Annotation Rendering (a) Background context (b) Background/Real Object context (c) Virtual Object context
    • - what information to display
    • annotation positions
    • modality switch
    User focus controls adaptive presentation management
  231. Context model in mixed reality
    • A context model describes all the entities that may influence carrying out the interactive task of user with the intended UI
      • Environment model - s uch attributes may be physical (e.g., lighting conditions), psychological (e.g., level of stress), and organization (e.g., location and role definition in the organization chart).
      • Platform model – couple of software/hardware, such atributes may be screen resolution/size
      • User model – stereotypes, sub-stereotypes (profiles)
  232. Plasticity analysis and usability evaluation Plasticity Domains in the MIS Plasticity thresholds and context boundary values Inter-context continuity Perceptive, cognitive, functional intra-context continuity and inter-context continuity Intra-context / inter-context frequency and frequency switch Refined Metrics for Plastic MR systems
  233. What do we have now? Final user Interface T Concrete user Interface T Task and Domain T Abstract user Interface T T=Target context of use Concrete user Interface S Final user Interface S Task and Domain S Abstract user Interface S S=Source context of use UsiXML unsupported model UsiXML supported model User S User T Environment T Reification Abstraction Reflexion Translation Platform S Environment S Platform T
  234. Conclusion
    • User model is described by parameters
      • Context independent
        • Age, gender, motor skills, general preference,…
      • Context dependent
        • Preference for certain contents
        • Interaction history
        • Can be initiated by the corresponding context independent parameter
        • Can overwrite this parameter
    • Preference specification
      • Is stored in user model
      • Can be processed by « critiquing tools » based on argumentation, QOC notation
  235. Evaluation [Skezely,1996] 0 2000 4000 6000 8000 10000 12000 14000 16000 Word KB KB Parser Query Network ops. UI states Presentation Actions Update Interactions Application logic User interface Generated code Models
  236. Coverage
    • How to improve our MDA-based method?
      • High ceiling
      • Low threshold
      • Wide walls
    Capabilities Resources (time, experience,…) 100% 50% Ceiling Threshold First generation Second generation MDA CASE tools Interface builders and integrated environments Resource win for applications supported by MDA-compliant tools Resource win for applications supported by first-generation
  237. What’s the User Interface language?
    • Today
      • Programming languages: Visual Basic, Java, C++,…
      • Markup languages: HTML, XUL,…
    • Tomorrow
      • Multimodal interfaces: XHTML + VoiceXML = X+V
      • Vectorial interfaces: SVG, LZX
      • Virtual reality interfaces: VRML97, XVRML
      • 3D user interfaces: X3D, MEL
      • Tabletop interface: OpenGL
      • Many new modalities are arriving
    • Conclusion
      • It is impossible to evolve/survive without any modelling approach, even for user interfaces
      • It is required to progress also on the design process
  238. Conclusion: the big picture of MDA UsiXML model: Abstract user interface UsiXML model: Concrete user interface Rendering Final user interface UsiXML models: task, domain Generative programming Graph transformations Graph transformations Derivation rules IdealXML ReversiXML FlashiXML QtkXML JaviXML VisualiXML TransformiXML GrafiXML VisiXML SketchiXML FormiXML KnowiXML MethodiXML
  239. The Invisible UI All Applications Adapted from [Palanque,2002] Target Applications, Domains Notations and tools User Interface Interaction Technique No more programming: only models 2020 2006 TODAY 2008 2007 No Interaction Technique Automated, batch systems Nothing Character UIs Business applications Screen definitions Graphical User Interfaces Information systems Entity-relationship Attribute model State-transition diagrams Multi-platform User Interfaces Web, Java applications Task model, context model, UML ,… Virtual Reality User Interfaces 3D Applications Scene model Mixed Reality User Interfaces Command and control systems, games World models Tangible UIs Embodied UIs Physical models Embarked syst
  240. Thank you very much for your attention For more information and downloading, http://www.isys.ucl.ac.be/bchi http://www.usixml.org User Interface eXtensible Markup Language http://www.similar.cc European network on Multimodal UIs Special thanks to all members of the team!

+ Jean VanderdoncktJean Vanderdonckt, 3 years ago

custom

3576 views, 6 favs, 2 embeds more stats

This presentation contains the slides of the Doctor more

More info about this document

© All Rights Reserved

Go to text version

  • Total Views 3576
    • 3573 on SlideShare
    • 3 from embeds
  • Comments 1
  • Favorites 6
  • Downloads 190
Most viewed embeds
  • 2 views on http://jisi.dreamblog.jp
  • 1 views on http://www.brijj.com

more

All embeds
  • 2 views on http://jisi.dreamblog.jp
  • 1 views on http://www.brijj.com

less

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate. If needed, use the feedback form to let us know more details.

Cancel
File a copyright complaint
Having problems? Go to our helpdesk?

Categories

Tags

more
-->
Search
RSS Feed
What's new?

© 2009 SlideShare Inc. All Rights Reserved