To the end of our possibilities with Adaptive User Interfaces

HITLAML ‘23 Keynote (Luxembourg, 5 September 2023) 1
To the end of our possibilities with
Adaptive User Interfaces
Jean Vanderdonckt
Université catholique de Louvain,
HITLAML ‘23 Keynote (Luxembourg, 5 September 2023)

Send me an
invitation
Prof. Jean Vanderdonckt
Full-time professor
Université catholique de Louvain
LouRIM
Place des Doyens, 1
B-1348 Louvain-la-Neuve, Belgium
Jean.Vanderdonckt@uclouvain.be
IFIP Fellow
ACM Distinguished Scientist
LinkedIn Scholar DBLP Orcid ACM DL IEEE DL Springer Research
Gate
Books Channel SlideShare

• What is an Intelligent User Interface (IUI)?
• An IUI should be knowledge-based and modular, infer
and evaluate the user's goals, and adapt its behavior to
users and their tasks [Sullivan & Tyler]
• IUIs are human-machine interfaces that aim to improve
the efficiency, effectiveness, and naturalness of human-
machine interaction by representing, reasoning, and
acting on models of the user, domain, task, discourse,
and media (e.g., graphics, natural language, gesture)
[Maybury]
• An IUI is any interface that results from the application of AI
techniques to HCI problems

• What is « intelligent » is disputable
• Focus on adaptation
• How to adapt what according to which user’s parameters
• Focus on knowledge
• What type of knowledge and how to exploit it
• What the user wants to do / what the user does not want to
do

• IUI is interdisciplinary by nature
Software
engineering
(SE)
Human-
Computer
Interaction
(HCI)
Artificial
Intelligence
(AI)
Social Sciences
(e.g., psychology,
ergonomics)
IUI = HCI  AI & EICS = HCI  SE
= Engineering Interactive Computing Systems
= Intelligent User Interfaces
= Adaptive User Interfaces

• A simple IUI software architecture
User interface
Control
Semantic core
User interface
Control
Intelligent
semantic core
Intelligent
User interface
Control
Semantic core
Knowledge
base
Interactive system Intelligent system System with IUI

• Terminology: User Interface Adaptation
• Biology: adaptation = the process of change by which an organism or
species becomes better suited to its environment
• HCI: UI adaptation = the process of modifying a user interface to
better suit it to the context of use C = (U,P,E)
• User: preferences, characteristics, traits, tasks, language, culture, …
• Platform, device, sensor:
• Graphical: screen resolution, orientation, viewport, color range, contrast,…
• Interactional: all input/output capabilities
• Environment
• Location: position, motion, acceleration,…
• Physical: light, noise, volume, network (online vs offline, bandwidth),
perturbation,…
• Organizational: changing structure, physical setup
• Psychological: stress, relationships,…
• But who is controlling this process?

• Environment: SLIME example
• Organizational: physical setup

Source: https://dl.acm.org/doi/10.1145/3457147

One left,
two right
(57.3%)
Two left,
one right
(39.8%)
All at
once
(2.9%)

• User  Adaptable user interface  Customization (manual)
Source: https://pptproductivity.com/blog/customize-powerpoint-ribbon-tabs

• User  Adaptable user interface  Customization (computer)
Source: https://dl.acm.org/doi/10.1145/989863.989936
Generative programming
Feature diagram
 Control
 Subjective satisfaction
 Time

• System  Adaptive UI  Personalization
Source: https://doi.org/10.1016/j.apergo.2013.04.017
 Error rate
 Acceptance
= Situation awareness
= Cognitive load

• System  Adaptive UI  Personalization ( Recommendation)
Source: https://doi.org/10.1016/j.apergo.2013.04.017
Onlinedeals.com
Home Headphones Camera & photo Sign in
Add to cart Add to cart Add to cart
Mpow 059 Bluetooth
Headphones - red
Beats Solo3 Wireless On-Ear
Headphones - Matte Black
Headphones compatible with
all smartphones
£89 £129 £49
 Fixation points
 Playfulness
 Customer experience
= Perception

Adaptable UI Adaptive UI
System
- Code size and complexity
- Processing time
- End user data access
Moderate
Low
None
High to very high
High to very high
High
End user
- Conformance to user’s needs
- Efficiency
- Level of control
- Cognitive load
- Predictability
Low
Low
Moderate
Very low
High
High
Moderate
None
Moderate
Very low
• How to compare adaptability vs. adaptivity?
Need for Mixed-initiative adaptation [Horvitz]

• Mixed-initiative adaptation
• The system proposes, the end user disposes

• How to characterize the evolution of UI adaptation?
• By level of mixed-initiative
1 The UI offers no assistance: end users make all decisions
2 UI offers a complete set of decision/action alternatives, or
3 Narrows the selection down to a few, or
4 Suggests one alternative, and
5 Executes that suggestion if the end user approves, or
6 Allows the end user a restricted time to veto before automatic execution, or
7 Executes automatically, then necessarily informs end users, and
8 Informs the end user only if asked, or
9 Informs the end user only if the UI decides to
10 The UI decides everything and acts autonomously
Adaptability
Adaptivity
Mixed
initiative
Adaptation mechanism
Adaptation
automation
Adaptation
control
Source: https://doi.org/10.1007/s10270-021-00909-7

• How to characterize the evolution of UI adaptation?
• By generation of programming language/technique
• 1GL: machine-dependent language
• 2GL: low-level language
• 3GL: high-level language (e.g., C++ like in MiniAba)
• 4GL: very-high level language (e.g., PHP, Python)
• 5GL: natural language (e.g., Mercury, Prolog, CSP)
• 6GL: extreme abstraction from implementation (e.g., no
code, visual development)

• Example #1: USE-IT (2GL, AAL=3)
Source: https://doi.org/10.1016/S0953-5438(97)00007-6
Menu.input_device=keyboard
Menu.inputTechnique=indirectPick2D
Menu.output_device=braille&speechSynthesiser
Menu.outTechnique=tactileTechnique
Menu.on_BrailleLines=2
Menu.on_BrailleCells=80
Menu.interim_feedback=speech
Menu.topology=horizontal
Menu.access_policy=byKeyboard
UI Development
Toolkit
Adaptivity
module
Ask { adapt {menu, selectionSet.size(15)}}
Adaptivity
decisions
Adaptation ( [menu.organization = SingleLevel, …])
Adaptation ( [menu.outTechnique = On_Screen.Popup, …]

• Example #2: RBUIS (3GL, AAL=10!)

• Example #2: RBUIS (3GL, AAL=10!):
deactivate a widget
1: function ApplyAdaptedUI(UIXML){
//Loop around the UI widgets
2: $(UIXML).find("Control").each(function () {
//Get the name and visibility attributes
3: var technicalName=$(this).attr('TechnicalName');
4: var isVisible = $(this).attr('Visible');
//Hide the invisible elements
5: if(isVisible == 'false'){
6: var element = GetElement(technicalName);
//Hide the element if it exists
7: if (typeof (element)!= 'undefined')
8: {element.style.visibility = 'collapse';}}
9: }); }

• Example #3: SEGUIA (3GL, AAL=5)
• Widget selection: how to select the most usable widgets
depending on the user’s context (i.e., user, platform, and
environment) and the user’s task
Source: https://doi.org/10.1109/UIDIS.1999.791464

• Widget selection: how to select the most usable widgets
depending on the user’s context (i.e., user, platform, and
environment) and the user’s task
Selection can be:
Manual (AAL=2)
Automated (AAL=10)
Semi-automated (AAL=5)

• Example #3: SEGUIA (3GL, AAL=6):
Shortcomings

• Example #4: Walkaware (4GL, AAL=7)
• Combines
• Walk awareness = route planning
• Weather awareness = weather forecasts
• Adapt to context of use
• User
• Platform configuration
• Location and environment

• Example #4: Walkaware (4GL, AAL=7): Weather

• Example #4: Walkaware (4GL, AAL=7): Walk

• Example #4: Walkaware (4GL, AAL=7): Both

• Example #4: Walkaware (4GL, AAL=7):
Need for higher-order rules
First order: direct
commands
Second order: govern first-
order adaptation rules
Third order: adaptation
strategies to promote or
demote sets of second-order
rules
if (platform = ‘mobile’)
then replace a radio box
by a drop-down list
prefer R1, than R2 if (user = ‘expert’) then
reverse the order of
“prefer R1, than R2”
if (platform = ‘mobile’)
and (entries are limited)
then
replace a radio box with
an edit field with codes

• Example #5: SURF (3GL, AAL=4):
Source: https://dl.acm.org/doi/10.1145/359784.360122 and
https://dl.acm.org/doi/10.1145/325737.325787

• Example #6: WiSEL (5GL, AAL=5):

• Example #6: WiSEL (5GL, AAL=5)
• Scoring function
• Principle: for each design option, display the widget having the
highest score computed by the scoring function
• Recalculate for each user, context of use, and input
Score (Widget)= P*SC+D*SU+ T*SG + f(w,SA)
where
P=the number of times when the widget is selected without being displayed by default
(WSelected = WDefault)
Score of change (SC)= promotion of a widget after being selected by a user or a designer
Score of unchanged (SU)= interest accorded to the system choice in term of rewarding a
well-behaved recommendation within reinforcement learning
D=the number of times where the selected widget is the displayed one (WSelected ≠ WDefault)
F(w,SA): determine if the selected widget match the designer recommendation
T=the total number of widget selection

• Benefit: some convergence over time
• Shortcoming: Who can play?
• The end user, but not all users are equal
• The designer, but not all designers are equal
• The system administrator, but not always appropriate
• Therefore, the scoring function should be parametrizable

• Example #7: AIDE (3GL, AAL=10)
• GUI layout as an optimization
Patient
Civil status
Sex
Date of day :
Male
Female
Unmarried
Married
Windowed
Divorced
Name :
Firstname :
Birthdate :
Complete Address :
Phone number :
Organization code :
Identification number :
Affiliation type :
Medecine man :
Single room
Two beds room
Four beds room
Regimen :
Ok
Cancel
Admission
Service :
Room type
Start
Patient
Civil status
Sex
Date of day :
Male
Female
Unmarried
Married
Windowed
Divorced
Name :
Firstname :
Birthdate :
Complete Address :
Phone number :
Organization code :
Affiliation type :
Medecine man :
Single room
Two beds room
Four beds room
Regimen :
Ok
Cancel
Admission
Service :
Room type
Start
Patient
Civil status :
Sex :
Date of day :
Female Male
Unmarried Married
Widowed Divorced
Name :
Firstname :
Birthdate :
Complete Address :
Phone number :
Organization code :
Affiliation type :
Medecine man :
Two beds Four beds
Regimen :
Ok
Cancel
Service :
Room type :
Organization
Medical care
Single
cost

• GUI auto-completion as a state-space search
Right auto-completion Bottom auto-completion

OIC1
OIC1 OIC2
OIC1 OIC2 OIC3
OIC1 OIC2 OIC3 OIC4
OIC1 OIC2 OIC3 OIC4 OIC5
OIC1 OIC2 OIC3 OIC4
OIC5
OIC1 OIC2 OIC3
OIC4
OIC1 OIC2 OIC3
OIC4 OIC5
OIC1 OIC2
OIC3
OIC1 OIC2
OIC3
OIC4
OIC1 OIC2
OIC3
OIC4
OIC1 OIC2 OIC3
OIC4
OIC5
OIC1 OIC2
OIC3
OIC4 OIC5
OIC1 OIC2
OIC3
OIC4
OIC5
OIC1 OIC2
OIC3
OIC4 OIC5
OIC1 OIC2
OIC3
OIC4
OIC5
OIC1
OIC2
OIC1
OIC2 OIC3
OIC1
OIC2
OIC3
OIC1
OIC2
OIC3
OIC4
OIC1
OIC2
OIC3
OIC4
OIC1
OIC2
OIC3
OIC4
OIC5
OIC1
OIC2
OIC3
OIC4 OIC5
OIC1
OIC2
OIC3
OIC4
OIC5
OIC5
OIC1
OIC2
OIC3
OIC4
OIC1
OIC2 OIC3
OIC4
OIC1
OIC2 OIC3
OIC4
OIC1
OIC2 OIC3
OIC4 OIC5
OIC5
OIC1
OIC2 OIC3
OIC4
OIC5
OIC1
OIC2 OIC3
OIC4
OIC1
OIC2 OIC3
OIC4
OIC5
(1)
(1)
(1)
(1)
(1)
(1)
(2)
(1)
(1)
(1)
(1)
(1)
(1)
(1)
(1)
(2)
(2)
(2)
(2)
(2)
(2)
(2)
(2)
(2)
(2)
(1)
(2)
(2)
2.2.2.2
2.2.2.1
2.2.1.2
2.2.1.1
2.1.2.2
2.1.2.1
2.1.1.1
2.1.1.2
1.2.2.2
1.2.2.1
1.1.2.1
1.2.2.2
1.1.2.2
1.1.2.1
1.1.1.1
1.1.1.2

• Example #9: FWL (5GL, AAL=10)
• GUI auto-completion as a CSP with fuzzy constraints

• Example #10: Sketch (6GL, AAL=6)
• GUI auto-completion with Graph Neural Network
(GNN), a Transformer model, and kNN

• Example #11: Instigator (6GL, AAL=6)
• GUI layout by GenerativeAI

• Some lessons to go
• Mixed-initiative: choose the right level
• Balancing who can contribute to what (by parameter
calibration) is key
Automatization
Degree
of invol-
vement
User
System

• Some lessons to go
• Cover adaptation life cycle where appropriate
U, S or a T maintains up-to-date
a series of goals to ensure user
interface adaptation
Adaptation request
Detection of
adaptation need
Notification for
adaptation request
Specification by
demonstration
Specification by
demonstration
Specification by
definition
This stage specifies which
entity will ensure a smooth
transition between the UI
before and after adaptation
This stage specifies which
entity will produce
meaningful information in
order to facilitate the
understanding of the
adaptation to the other
entities
This stage specifies the
entity responsible for
evaluating the quality of the
adaptation performed
Goals for user
Interface adaptation
Meaning of an
input adaptation
Meaning of an
output adaptation
pecification
of adaptation
Shape of an
input adaptation
Shape of an
output adaptation
pplication of
adaptation
INterpretation
of evaluation
nitiative for
adaptation
Evaluation
of adaptation
Interactive
System
Transition with
adaptation
Goals for user
Interface adaptation
Meaning of an
input adaptation
Meaning of an
input adaptation
Meaning of an
output adaptation
pecification
of adaptation
Shape of an
input adaptation
Shape of an
output adaptation
pplication of
adaptation
INterpretation
of evaluation
nitiative for
adaptation
Evaluation
of adaptation
Interactive
System
Interactive
System
Transition with
adaptation
Context-aware adaptive visualizations for critical decision makin
EU “Awareness Inside” SYMBIOTIK Project led by Prof. Luis Lei

Thank you very much for your attention
Any question?
49

• Reference paper on adaptive user interfaces
Source: https://link.springer.com/article/10.1007/s10270-021-00909-7

User System
User
Adaptation
engine
System
User
Adaptation
Manager
Adaptation
engine
System
(a)
(b)
(c)

Perception
Decision
•Learning,
Prediction,
Adaptation
Action
Perception
Decision
•Learning,
Predictiion,
Adaptation
Action
End user System
User interface

Context of use
Software system
Actor
External Sources
Information
Data Knowledge Wisdom
Experts
Context Model
User Model
Platform Model
Environment Model
Intelligent User Interface
Task & Domain
Abstract UI
Concrete UI
Final UI
Platform
Intention
Semantic Core
Context Probe
Third Parties
Intelligent UI Adaptor
Adaptation
Parameters
Adaptation Manager
Adaptation
Engine
Adaptation Machine Learning
Adaptation
Explainer
Adaptation
Transitioner
Adaptation
Manager UI

To the end of our possibilities with Adaptive User Interfaces

Recommended

Recommended

More Related Content

Similar to To the end of our possibilities with Adaptive User Interfaces

Similar to To the end of our possibilities with Adaptive User Interfaces (20)

More from Jean Vanderdonckt

More from Jean Vanderdonckt (20)

Recently uploaded

Recently uploaded (12)

To the end of our possibilities with Adaptive User Interfaces