The document describes a replication study that measured the user experience of different adaptive user interfaces using EEG brain activity data. The study aimed to analyze 20 different adaptive graphical menus to compare their influence on cognitive load, engagement, memorization, and attraction. 40 master's students participated in the within-subjects study, using each of the 20 menus in random order. EEG data was analyzed to measure the dependent variables, and questionnaires assessed perceived cognitive load and attraction. Statistical tests found significant differences between menus for all dependent variables, suggesting menu type influences user experience metrics.

1. Measuring the User Experience
of Adaptive User Interfaces
using EEG: A Replication Study
Daniel Gaspar-Figueiredo <dagasfi@epsa.upv.es>
Jean Vanderdonckt <jean.vanderdonckt@uclouvain.be>
Silvia Abrahao <sabrahao@dsic.upv.es>
Emilio Insfran <einsfran@dsic.upv.es>
EASE2023

2. Overview
Motivation
Baseline Experiment
Replication Study
Data Analysis
Study Results
2
Motivation
Baseline Experiment
Replication Study
Data Analysis
Study Results

3. • Adaptive User Interfaces (AUI) – UI that can be changed considering the Context of use to
enhance the User eXperience (UX)
• Type of changes:
• Layout
• Colours
• Modalities
• Element interactions
• Element styling
• …
Motivation
3
…
User Interface
Context of
use
User
Platform
Environment
• Too wide spectrum of possibilities
Motivation Baseline Experiment Replication Study Data Analysis Study Results

4. Motivation
4
Different UI designs produce different UX
Measure UX through physiological
data such as brain activity (EEG)
Motivation Baseline Experiment Replication Study Data Analysis Study Results

5. Overview
Motivation
Baseline Experiment
Replication Study
Data Analysis
Study Results
5
Motivation
Baseline Experiment
Replication Study
Data Analysis
Study Results

6. Baseline Experiment: Summary
6
• Analyse a set of adaptive graphical menus for the purpose of comparing them with respect to
their UX produced in terms of cognitive load, engagement, memorisation, and attraction from
the point of view of both researchers and user interface designer in the context of end-users of
applications with graphical menus
• 20 graphical adaptive menus were compared (mail manager and web browser applications)
• 40 participants aged from 18 to 63 years old (mean 42)
• Cognitive load, engagement, memorisation, and attraction were computed using EEG analysis
• RQ: Do graphical adaptive menus have a different influence on the UX?
Motivation Baseline Experiment Replication Study Data Analysis Study Results

7. Baseline Experiment: Summary
7
Statistically significant differences were found between the menus regarding the Cognitive load,
engagement, memorisation, and attraction
May be due to diversity of users
• Multiple backgrounds
• Very wide range of ages
Motivation Baseline Experiment Replication Study Data Analysis Study Results

8. Overview
Motivation
Baseline Experiment
Replication Study
Data Analysis
Study Results
8
Motivation
Baseline Experiment
Replication Study
Data Analysis
Study Results

9. Replication Study: Goal
9
• Analyse a set of adaptive graphical menus
• For the purpose of comparing them
• with respect to their user experience produced in terms of cognitive load, engagement,
memorisation, and attraction
• From the point of view of both researchers and user interface designers
• In the context of and end-users' group with a background in computer science
RQ1: Do the 20 graphical adaptive menus have a different influence on the UX?
RQ2: Does the UX measured using EEG signals correlate with the subjective ratings obtained using traditional
questionnaires?
Goal
Experimental
objects
Variables Hypotheses
Participants Study design Instrumentation
Motivation Baseline Experiment Replication Study Data Analysis Study Results

10. Replication Study: Experimental objects
10
Vanderdonckt, J., Bouzit, S., Calvary, G., & Chêne, D. (2019). Exploring a
design space of graphical adaptive menus: normal vs. small screens. ACM
Transactions on Interactive Intelligent Systems (TiiS), 10(1), 1-40.
Design space for graphical adaptive menus
(8 variables):
• Position
• Orientation
• Size
• Shape
• Value
• Color
• Texture
• Motion
Goal
Experimental
objects
Variables Hypotheses
Participants Study design Instrumentation
Motivation Baseline Experiment Replication Study Data Analysis Study Results

11. • We selected 20 different graphical adaptive menus from 
Replication Study: Experimental objects
11
Vanderdonckt, J., Bouzit, S., Calvary, G., & Chêne, D. (2019). Exploring a design
space of graphical adaptive menus: normal vs. small screens. ACM Transactions
on Interactive Intelligent Systems (TiiS), 10(1), 1-40.
Goal
Experimental
objects
Variables Hypotheses
Participants Study design Instrumentation
Motivation Baseline Experiment Replication Study Data Analysis Study Results
ItemA
MenuItem1
MenuItem2
MenuItem3
MenuItem4
MenuItem5
MenuItem6
MenuItem7
MenuItem1
MenuItem2
MenuItem3
MenuItem4
MenuItem5
MenuItem6
MenuItem7
ItemA
MenuItem1
MenuItem2
MenuItem3
MenuItem4
MenuItem5
MenuItem6
MenuItem7
ItemA ItemA
MenuItem1
MenuItem2
MenuItem3
MenuItem4
MenuItem5
MenuItem6
MenuItem7
MenuItem1
MenuItem2
MenuItem3
MenuItem4
MenuItem5
MenuItem6
MenuItem7
ItemA
MenuItem1
MenuItem2
MenuItem3
MenuItem4
MenuItem5
MenuItem6
MenuItem7
ItemA
MenuItem1
MenuItem2
MenuItem3
MenuItem4
MenuItem5
MenuItem6
MenuItem7
ItemA
MenuItem3
MenuItem6
MenuItem1
MenuItem2
MenuItem4
MenuItem5
MenuItem7
ItemA
Static menu Bolding
menu
Underlying
menu
Glowing
menu
Adaptive
activation-
area menu
Boxing
menu
Morphing
menu
Ephemeral
menu
Highlighting
menu
Split menu
without
replication
Greyscaling
menu
MenuItem1
MenuItem2
MenuItem3
MenuItem4
MenuItem5
MenuItem6
MenuItem7
ItemA

12. Replication Study: Experimental objects
12
MenuItem2 MenuItem4
MenuItem3
MenuItem6
MenuItem1
MenuItem8
MenuItem7
MenuItem5 ItemA
Leaf menu
MenuItem1
MenuItem2
MenuItem3
MenuItem4
MenuItem5
MenuItem6
MenuItem7
ItemA
Italicizing
menu
Blinking
menu
Pulsing
menu
Out-of-context
menu
Temporal menu Pink menu
MenuItem1
MenuItem2
MenuItem3
MenuItem4
MenuItem5
MenuItem6
MenuItem7
ItemA
Twisting
menu
Rotating
menu
• We represented these menus in two different contexts
Goal
Experimental
objects
Variables Hypotheses
Participants Study design Instrumentation
Motivation Baseline Experiment Replication Study Data Analysis Study Results

13. Goal
Experimental
objects
Variables Hypotheses
Participants Study design Instrumentation
Replication Study: Experimental objects
New tab
New window
Zoom
Favorites
New InPrivate window
History
Downloads
Apps
Extensions
Collections
Print
Web capture
Share
Find on page
Read aloud
More tools
Settings
Help and feedback
Close browser
>
>
>
Browser configuration
13
• Context A: Thunderbird Mail manager
• Context B: Microsoft Edge configuration menu
Motivation Baseline Experiment Replication Study Data Analysis Study Results
Mensaje
New Message
Reply
Forward
Forward As
Reply to All
Edit As New Message
Open Message
Open in Conversation
Attachements
Tag
Mark
Archive
Move to
Copy to
Move Again
Create Filter From Message…
Ignore Thread
Ignore Subthread
Watch Thread
>
>
>
>
>
>
Message

14. Replication Study: Participants
14
• Convenience sampling
• 40 students (5 female, 35 male, and 0 non-binary) enrolled in a Master’s degree in Computer
Science at the UPV
• A common background in computer science
• Aged from 20 to 40 years old (mean = 24)
Goal
Experimental
objects
Variables Hypotheses
Participants Study design Instrumentation
Motivation Baseline Experiment Replication Study Data Analysis Study Results

15. Replication Study: Variables
15
• Independent Variables:
• (20) Menu types
• Dependent Variables (Emotion-based):
• Attraction: positive/pleasant reaction to a negative/unpleasant reaction response to a situation
• Memorisation: intensity of cognitive processes related to the formation of future memories during an
experience.
Goal
Experimental
objects
Variables Hypotheses
Participants Study design Instrumentation
Motivation Baseline Experiment Replication Study Data Analysis Study Results
• Cognitive load: The use of cognitive resources to carry
out a task or visualize a stimulus
• Engagement: degree of involvement or connection
between the participant and the stimulus or task

16. Replication Study: Variables
16
• Performance-based Variables (Dependent):
• Completion-Time: Measures in seconds the time elapsed between the initial state of the task and the final
state, when the task is correctly completed
Goal
Experimental
objects
Variables Hypotheses
Participants Study design Instrumentation
Motivation Baseline Experiment Replication Study Data Analysis Study Results
• Perception-based Variables (Dependent):
• Perceived Attraction: Subjective perception of attraction in response to
stimuli or a situation
• Perceived Cognitive Load: Subjective perception of cognitive load or
mental effort required during the task

17. Replication Study: Hypotheses
17
• 𝐻𝑛11: There are no significant differences in the users’ attraction when using different graphical adaptive menus
• 𝐻𝑛12: There are no significant differences in the users’ memorisation when using different graphical adaptive menus
• 𝐻𝑛13: There are no significant differences in the users’ cognitive load when using different graphical adaptive menus
• 𝐻𝑛14: There are no significant differences in the users’ engagement when using different graphical adaptive menus
• 𝐻𝑛15: There are no significant differences in the users’ completion-time when using different graphical adaptive menus
• 𝐻𝑛21: There is no correlation between the Cognitive load (EEG) and the Perceived Cognitive Load (Questionnaire)
• 𝐻𝑛22: There is no correlation between the Attraction (EEG) and the Perceived Attraction (Questionnaire)
Goal
Experimental
objects
Variables Hypotheses
Participants Study design Instrumentation
Motivation Baseline Experiment Replication Study Data Analysis Study Results

18. Replication Study: Study Design
18
• Within Subjects design: Each participant used the 20 different menu types
• Each menu type was randomly assigned to a specific domain (Thunderbird’s mail manager or Web Browser settings)
• Each participant used the menus in a different order (also randomly assigned)
Subject 1 Menu_1 Menu_2 Menu_19 Menu_20
…
Subject 2 Menu_8 Menu_5 Menu_2 Menu_1
…
Subject 3 Menu_11 Menu_13 Menu_8 Menu_6
…
Subject n Menu_18 Menu_2 Menu_13 Menu_19
…
… … … … … …
Goal
Experimental
objects
Variables Hypotheses
Participants Study design Instrumentation
Motivation Baseline Experiment Replication Study Data Analysis Study Results

19. Replication Study: Instrumentation
19
• Interactive PowerPoint presentation representing the 20 different menu types
• EEG Headset to obtain brain activity
• UEQ-S and NASA-TLX questionnaires
Sensors are placed at
positions AF7, Fp1, Fp2, AF8,
F3, F4, P3, P4, PO7, O1, O2
and PO8 (10-10 standard)
Goal
Experimental
objects
Variables Hypotheses
Participants Study design Instrumentation
Motivation Baseline Experiment Replication Study Data Analysis Study Results

20. Replication Study: Instrumentation
20
Data obtained: Raw EEG Data. Brain activity
Goal
Experimental
objects
Variables Hypotheses
Participants Study design Instrumentation
Motivation Baseline Experiment Replication Study Data Analysis Study Results

21. Replication Study: Instrumentation
21
Attraction
Memorization
Cognitive load
Engagement
Raw EEG Data
Delta, 0-4 Hz Theta, 4-8 Hz
Alpha, 8-13 Hz Beta, 13-30 Hz
Goal
Experimental
objects
Variables Hypotheses
Participants Study design Instrumentation
Motivation Baseline Experiment Replication Study Data Analysis Study Results

22. Replication Study: Instrumentation
22
Subject n Menu_A Menu_B Menu_C Menu_D
…
Goal
Experimental
objects
Variables Hypotheses
Participants Study design Instrumentation
• Completion time: the time is measured from the beginning to the end of the interaction for each menu.
Motivation Baseline Experiment Replication Study Data Analysis Study Results

23. Replication Study: Instrumentation
23
Goal
Experimental
objects
Variables Hypotheses
Participants Study design Instrumentation
Motivation Baseline Experiment Replication Study Data Analysis Study Results
UEQ-Short
Nasa-TLX
Perceived Attraction: UEQ-Short Evaluation
Perceived Cognitive Load: NASA-TLX evaluation

24. Overview
Motivation
Baseline Experiment
Replication Study
Data Analysis
Study Results
24
Motivation
Baseline Experiment
Replication Study
Data Analysis
Study Results

25. Data Analysis
• Mean distance of the Dynamic Time Warping (DTW) for
each participant (40), menu (20), and EEG metric (4)
1. Group the time series by menu and metric
2. Compute the distance matrix
3. Calculate the mean distance for each
participant/menu
25
Motivation Baseline Experiment Replication Study Data Analysis Study Results
Cognitive load – Distance matrix for Blinking menu
Cognitive load per
participant - Blinking menu
1
2
3
40
D=3
D=10
Participant ID
Participant ID
Participant 3
Participant 2
Participant 1
Participant 2

26. Data Analysis
1. Descriptive Study of dependent variables
2. Kruskal-Wallis or ANOVA (hypotheses testing)
3. Pearson or Spearman (correlation)
4. Cliff′s δ (Statistical significances)
26
Motivation Baseline Experiment Replication Study Data Analysis Study Results

27. Overview
Motivation
Baseline Experiment
Replication Study
Data Analysis
Study Results
27
Motivation
Baseline Experiment
Replication Study
Data Analysis
Study Results

28. Study Results: Descriptive (emotion-based variables)
28
Motivation Baseline Experiment Replication Study Data Analysis Study Results

29. Study Results: Descriptive (emotion-based variables)
29
Motivation Baseline Experiment Replication Study Data Analysis Study Results
MenuItem1
MenuItem2
MenuItem3
MenuItem4
MenuItem5
MenuItem6
MenuItem7
ItemA
Out of context
Temporal
Coloured

30. Study Results: Descriptive (emotion-based variables)
30
Motivation Baseline Experiment Replication Study Data Analysis Study Results
MenuItem2 MenuItem4
MenuItem3
MenuItem6
MenuItem1
MenuItem8
MenuItem7
MenuItem5 ItemA
MenuItem1
MenuItem2
MenuItem3
MenuItem4
MenuItem5
MenuItem6
MenuItem7
ItemA
MenuItem1
MenuItem2
MenuItem3
MenuItem4
MenuItem5
MenuItem6
MenuItem7
ItemA
Italicizing
Underlying Leaf

31. Study Results: Descriptive (emotion-based variables)
31
Motivation Baseline Experiment Replication Study Data Analysis Study Results
MenuItem2 MenuItem4
MenuItem3
MenuItem6
MenuItem1
MenuItem8
MenuItem7
MenuItem5 ItemA
MenuItem1
MenuItem2
MenuItem3
MenuItem4
MenuItem5
MenuItem6
MenuItem7
ItemA
MenuItem1
MenuItem2
MenuItem3
MenuItem4
MenuItem5
MenuItem6
MenuItem7
ItemA
Italicizing
Underlying Leaf
MenuItem1
MenuItem2
MenuItem3
MenuItem4
MenuItem5
MenuItem6
MenuItem7
ItemA
Out of context
Temporal
Coloured

32. Study Results: Descriptive (perception-based variables)
32
Menu
Attraction
(EEG)
[-100, 100]
Attraction
(Perceived)
[-3,3]
Cognitive Load
(EEG)
[0, 100]
Cognitive Load
(Perceived)
[0, 100]
Blinking 11,336 1,306 29.420 29.983
Split without
replication
0.755 0.281 30.672 33.758
Out of Context
disapp
-1.159 -0.725 30.189 32.275
Morphing 4.225 -0.018 35.321 41.758
Italizing -1.002 -1.037 33.351 43.816
Leaf 3.331 1.020 35.444 66.933
Motivation Baseline Experiment Replication Study Data Analysis Study Results
MenuItem2 MenuItem4
MenuItem3
MenuItem6
MenuItem1
MenuItem8
MenuItem7
MenuItem5 ItemA
MenuItem1
MenuItem2
MenuItem3
MenuItem4
MenuItem5
MenuItem6
MenuItem7
ItemA
MenuItem3
MenuItem6
MenuItem1
MenuItem2
MenuItem4
MenuItem5
MenuItem7
ItemA
MenuItem1
MenuItem2
MenuItem3
MenuItem4
MenuItem5
MenuItem6
MenuItem7
ItemA

33. Study Results: Hypotheses testing
33
Hypotheses Method p-value Menu Difference Cliff’s Delta Effect Size
𝑯𝒏𝟏𝟏 Kruskal-Wallis .00 YES 0,302 Medium
𝑯𝒏𝟏𝟐 Kruskal-Wallis .00 YES 0,667 Large
𝑯𝒏𝟏𝟑 Kruskal-Wallis .00 YES 0,585 Large
𝑯𝒏𝟏𝟒 Kruskal-Wallis .00 YES 0,277 Medium
𝑯𝒏𝟏𝟓 Kruskal-Wallis .00 YES 0,575 Large
Hypotheses Normal Distr. Method Correlation p-value
𝑯𝒏𝟐𝟏 YES Pearson 0.819 .045
𝑯𝒏𝟐𝟐 YES Pearson 0.817 .046
Motivation Baseline Experiment Replication Study Data Analysis Study Results

34. Conclusions
• Baseline experiment results are confirmed with a more homogeneous group of users
• Effect size shows that these results could be generalised to other contexts
• Objective assessment of UX (cognitive load and attraction) through EEG signals
• Trends:
• Different font styles or structures  uncertainty on UX
• Different colour and temporal variance  predictable user reaction
• Menus that are preferred by the users do not necessarily perform well
• Future work:
• Experiments with more UI elements
• Provide recommendations for UI developers on the dimensions more relevant for
UX
• How to adapt UI to better fit with user needs in different contexts
34
Motivation Baseline Experiment Replication Study Data Analysis Study Results

35. Measuring the User Experience of Adaptive User
Interfaces using EEG: A Replication Study
Thanks for your attention
Daniel Gaspar-Figueiredo <dagasfi@epsa.upv.es>
Jean Vanderdonckt <jean.vanderdonckt@uclouvain.be>
Silvia Abrahao <sabrahao@dsic.upv.es>
Emilio Insfran <einsfran@dsic.upv.es>
EASE2023

36. Study Results: Descriptive (performance-based variables)
38
Motivation Baseline Experiment Replication Study Data Analysis Study Results