A human-computer interaction research talk about how we measure mental workload, and how people might reflect on this type of personal data in the future. The research is carried out at the University of Nottingham in the School of Computer Science. Based on a study published at CHI2022, led by Serena Midha. Presented to Bell Labs

1. Brain Data as
Cognitive
Personal Informatics
Max L. Wilson
@gingdottwit

3. Dr Max L. Wilson https://brain-data-uon.gitlab.io/
3
https://news.ubc.ca/
https://medium.com/
https://choosemuse.com
Consumer NeuroTech is here
And its not great yet…

4. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Cognitive Activity is not just
tracked from the head
4

5. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
So far
• Neurotech focuses on classifying some form of cognitive state
• And either warns you when you hit that state
- “you are stressed right now”
• Or helps you to stay in that state
- “try and meditate for longer”
• They also have low accuracy.
5

6. Dr Max L. Wilson https://brain-data-uon.gitlab.io/
Neurotech at work (thanks DALL-E)
6

7. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
But people want track change
• https://myfeel.co
7

8. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Apps already estimating “stress”
• Welltory ingests lots of data sources, inc neurotech
• Garmin shows you “Stress” over a day
8

9. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
“Stress” - 3 types get mixed up
• Stress as a mental health chronic condition
• Also - Post Traumatic Stress Disorder
• Feeling stressed about things
• The body being under stress (physiological data)
9

10. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
“Stress” - 3 types get mixed up
• Stress as a mental health chronic condition
• Also - Post Traumatic Stress Disorder
• Feeling stressed about things
• The body being under stress (physiological data)
10
Actually
Measurable with
Physiological
Data
What people
want to track

11. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
“Stress” - 3 types get mixed up
• Stress as a mental health chronic condition
• Also - Post Traumatic Stress Disorder
• Feeling stressed about things
• The body being under stress (physiological data)
11
Can be good!
Bad

12. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Good Stress
Probably == “High Mental Workload”
• Going on a run would show as high stress in Garmin
• Some people thrive on high pressure workloads and enjoy them
• And they do not necessarily feel stressed about it, and dont necessarily get chronic stress
• You need to be in high mental workload to achieve a hard task
• And then feel a sense of achievement
• Gaming can lead to high mental workload and feeling relaxed
• Many papers use the same tasks to induce stress and mental workload
• and both can be estimated by the same physiological indicators
12

13. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
We’re using “Mental Workload”
• Mental Workload is a kind of emotionally agnostic term
• Compared to ‘stress’, as a loaded term
• A super well established theoretical basis scientifically
- Wickens, attention, multiple resource theory, NASA TLX, etc
• Becomes a good ‘parallel’ for physical activity and representative of
general cognitive activity
13

14. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
So what are HCI Questions?
• State classification is largely a machine learning problem now
• The HCI Qs are for when it becomes a form of
cognitive personal informatics
• What amount of mental workload are people aiming for?
• What should neurotech aim to help us track longitudinally?
• How should apps visualise this to us?
• How will society handle the inclusion of this data?
14

15. Lived Experiences of Mental Workload
in Everyday Life
Serena Midha, Max L. Wilson, Sarah Sharples
CHI2022

16. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Research Questions
16
RQ1
How are experiences
of MWL perceived?
RQ2
What impact does
MWL have on our
lives and work, and
vice versa?
RQ3
What goals should we
be setting in terms of
MWL?

17. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Method Overview
17
19
participants
Subjective
MWL
tracking
1-2 hour
interview
Interpretive
Phenomen-
ological
Analysis
Mon-Fri Following week

18. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Mental Workload Ratings - App
• Custom app - every 30mins during work and every 1hr outside of work
• What has you overall mental workload level been since the last rating?
• Presented back to participants
for the interview
• Daily evening diary about sleep
stress, mood etc. Also for discussion
18

19. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Semi-Structured Interview
• Typically between 1-2hrs, on zoom or teams (during covid)
• Personal Experiences throughout the week
• How people understood and conceptualised their own mental workload
• What people wanted to know about their cognitive activity
• Ethical/Legal/Moral concerns they had about future tracking
19

20. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
19 Participants (from 20)
• Office workers, with android phones, no clinical mental health history
• Originally 10 researchers and 10 industry
- 1 researcher excluded after interview highlighted misunderstanding
• Received £75 remuneration (for week of ratings + interview)
- additional £25 given for good data return during the week
- all given £100 in the end, including excluded participant
20

21. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Results
21
CHI’22 - Mental Workload Concept - 4 themes
FAccT’22 - Neuroethics - 3 themes

22. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
4 Themes about Mental Workload
22
1 General Perceptions of MWL
Changing Perceptions of MWL
The MWL Cycle
The Cycle Can’t Always be Facilitated
2
3
4

23. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
1. General Perceptions
23
Positive Negative
High
Low
Medium
Fulfilled, enjoyment,
stimulated, less distracted
Pressure, stress,
overwhelmed
Relaxed, enjoyment,
manageable
Bored, distracted,
unsatisfied, unproductive
Comfort, enjoyment and in
control - the ‘sweet spot’ -

24. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
2. Things that change perceptions
24
Pressure
Outcome
Enjoyment
Location
!

25. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
3. The Mental Workload Cycle
25
Low MWL
Medium MWL
High MWL
Burnout, Fatigue, Resentment,
Reduced work quality,
Poor mental and physical health
Decreased productivity,
Decreased satisfaction,
Decreased enjoyment
Reduced excitement

26. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Each Level Serves a Purpose
26
High
Work quality and
quantity
Sense of
achievement
Difficult tasks
Medium
Productivity
Best of both
Low
Rest
and recovery
Preparation
Reward

27. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
4. The Cycle can’t always be Facilitated
27
1 Life factors - medical, exercise
Internal factors - thoughts, effort
External factors - circumstances, task completion
2
3

28. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
3 Themes about Ethical Concerns
28
1 Fear of the data
Negative effects of the data on the self
The spectrum of sharing
2
3

29. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
1. Fear of the Data
29
Fear of
inaccurate
judgement
Fear of
personal
judgement
Fear
of
consequences
Work
Personal
Work
Exploitation
Misinterpretation
Validity
Complex
Context

30. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
2. Negative Effects on the Self
30
Being
controlled by
the data
Data
exacerbating
negative
states

31. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
3. The Spectrum of Sharing
31
What
concerns?
Controlled sharing for
positive change
It depends
on the risk
An
absolute no

32. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Discussion
32

33. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
My own main lessons
• There’s a big difference compared to physical activity goals
• Cognitively sedentary people vs Overworked tired people
• The “ideal MWL cycle” could be very varied, if not very individual
• Hard to parameterise for people
• We wonder what will become the norms for longitudinal tracking
• like reach 10k steps or 30mins exercise a day
33

34. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
My own main lessons
• We probably need to move away
from the high == bad == red metaphor
• There’s lots of good high mental workload
• Getting stuck in a level seems to be bad
or not being able to apply the right level
when needed
34

35. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
My own main lessons
• There’s a very large gap between
- motivating behavioural goals / changes
- and understanding this mental workload cycle
• Most people talk about wanting to reduce “stress”
- so its part perception of what is good and bad
- and part understanding the relationship between what we can
measure and what people want to achieve
• A massive research gap in cognitive personal informatics
35

36. Building a community focus on
Cognitive Personal Informatics

37. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
CHI2022 SIG Discussion
37

38. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
CHI2022 SIG Discussion
38

39. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
CHI2022 SIG Discussion
39

40. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
The Community Aims
1. To identify the size of the interested community
2. To collectively identify and explicate the RQs and Topics of interest
3. To establish a research agenda
4. To plan future workshops and other events
40

41. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
The CHI2022 SIG Discussion
41

42. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Upcoming plans
• A CHI2023 Workshop
• An IJHCS Special Issue
• A Medium blog series
42

43. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Our Slack
43
https://cog-pers-informatics.slack.com

44. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Questions?
44
https://brain-data-uon.gitlab.io