A human-computer interaction research talk about how we measure mental workload, and how people might reflect on this type of personal data in the future. The research is carried out at the University of Nottingham in the School of Computer Science, involving functional Near Infrared Spectroscopy (fNIRS)

1. Physiological indicators
of task demand, fatigue,
and cognition
during Work Tasks
Max L. Wilson
@gingdottwit

2. Context

3. Brain-based HCI
What brain data can tell
us about HCI
Max L. Wilson
@gingdottwit

4. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
https://brain-physio-uon.gitlab.io/
4

5. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Most important criteria for us
• We can run a ‘normal’ user study.
• As much ecological validity in
- the environment they do the study
- natural user behaviour in the study
- as normal/natural a task as possible
• And tell if theres a cognitive difference between UIs or Tasks
5

6. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
What we need…
Reliable Brain Measurements
of HCI
6

7. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
What we need…
Reliable Brain Measurements
of HCI
We can trust the data from uncontrolled conditions
(not limiting interaction or body position)
7

8. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Our fNIRS - fully portable
• This is a fully portable fNIRS
- bluetooth connection
- Most commercially user friendly
8
Artinis: Octomon

9. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
9

10. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Showing that fNIRS works for HCI
• CHI2014: fNIRS + Think Aloud
• CHI2015: fNIRS vs HCI Artefacts
• CHI2015: 3 UIs create
different Mental Workload
• Horia Maior & Matthew Pike
• Working with Prof. Sarah Sharples
10

11. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
DigiTOP Grant
• Digital Toolkit for optimisation of operators and technology
in manufacturing partnerships (DigiTOP)
- £1.9M / 3 years
- PI: Prof Sarah Sharples (Nottingham)
• Global shift towards digital manufacturing techniques
• Physical Workload » Mental Workload
• Traditional Human Factors » Cognitive Ergonomics
11
http://digitop.ac.uk

12. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Facial Thermography vs fNIRS
12
Adrian Marinescu
See Computerphile on YouTube
And many other sensors, including Posture Analysis

13. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Mental Workload
13
Sharples, Sarah and Megaw Ted (2015). Definition and mesurement of human workload.
In Wilson John R and Sharples Sarah, editors, Evaluation of human work. CRC Press.

14. Physiological indicators of
task demand, fatigue, and cognition
during Work Tasks
EPSRC
DigiTOP Project
https://doi.org/10.1016/j.ijhcs.2020.102522

15. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
15
Investigate the relationship between
perceptual load, fatigue
(as inferred from time on task), and
physiological response
during tasks requiring
sustained attention

16. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
The task
16
Simulated quality control inspection task
- that required sustained attention
- and would vary MWL and induce fatigue over time
- with the aim of creating opportunities for task-unrelated thoughts.

17. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Task Unrelated Thoughts / Mindwandering
• Occurs more as you become practiced at a repetitive task (underload)
- associated with mistakes
• Hard to know ground truth - even self-awareness is hard
• Subjective response requests, by nature, draw you out of them
• Shown as related to activity in the Mid-Temporal Gyrus and Pre-frontal
Cortex
17

18. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Variables
• Two factor within-subjects approach
• Independent variables: perceptual load, fatigue
• Dependent subjective variables: NASA TLX and 9-point
Fatigue rating
• Dependent objective variables: fNIRS measures (HbO, HbR in
the PFC and MTG areas), heart rate, breathing rate, facial skin
temperature, task performance, mean response time
18

19. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Conditions
• Low Perceptual Demand High Perceptual Demand
• Adapted from Forster and Lavie (2009)
19

20. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Hypotheses
• H1 Perceived MWL will be higher in the high task demand condition than in the low.
• H2 Frequency of TUTs will decrease in the higher demand task condition.
• H3 Self-reported fatigue rating will increase over time.
• H4 Heart rate will increase in the high task demand condition as compared to the low.
• H5 Breathing rate will increase in the high task demand condition as compared to the low.
• H6 Hemodynamic response will vary between the low versus high task demand condition.
• H7 Nose temperature will decrease in the high task demand condition as compared to the low.
• H8 Heart rate will decrease as time progresses.
• H9 Breathing rate will vary as time progresses.
• H10 Hemodynamic response will vary as an individual progresses through the 10 task blocks.
• H11 Nose temperature will vary as as an individual progresses through the 10 task blocks.
20

21. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Hypotheses
21
Measure Perceptual load Fatigue
Subjective
MWL Increase with.
Fewer TUTs
Increase over time
Heart Rate Increase with Decrease over time
Breathing Rate Increase with Vary over time
Cerebral hemodynamics in PFC Vary with Vary over time
Cerebral hemodynamics in MTG Vary with Vary over time
Nose temperature Decrease with Vary over time

22. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
• Thermal Camera fNIRS Chest Strap
FAU temperature blood oxygenation breathing / heart rate
Sensors
22

23. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Facial Thermography
23

24. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Facial Thermography Data
24
Area (red)
Nose tip (blue)

25. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
fNIRS impact needs a baseline
25

26. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Results - Task Performance
• T-tests showed that
• Error rate increased significantly ( t(34)=-6.1, p<0.001 )
• Response team increased significantly ( t(34)=-21.02, p<0.001 )
• TUTs reduced in High Perceptual load ( χ2(1)=10.30, p = 0.0013 )
26

27. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Results
27
Measure Perceptual load Fatigue
Subjective
Significant Increase, particularly in
Mental Demand, temporal
demand, and effort
Significant increase over time
(e.g. last block compared to first
block)
Heart Rate No significant effect
HR difference from baseline
decreased as time progressed
Breathing Rate
BR difference from baseline
increased in high demand
conditions
BR difference from baseline
increased as time progressed
Cerebral hemodynamics in the
PFC
No significant effect No significant effect
Cerebral hemodynamics in the
MTG
No significant effect No significant effect
Nose temperature
Temperatures relative to baseline
decreased in high demand
conditions
No significant effect

28. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Breathing rate over block
28

29. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Breathing Rate by Perceptual Demand
29

30. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Results
30
Measure Perceptual load Fatigue
Subjective
Significant Increase, particularly in
Mental Demand, temporal
demand, and effort
Significant increase over time
(e.g. last block compared to first
block)
Heart Rate No significant effect
HR difference from baseline
decreased as time progressed
Breathing Rate
BR difference from baseline
increased in high demand
conditions
BR difference from baseline
increased as time progressed
Cerebral hemodynamics in the
PFC
No significant effect No significant effect
Cerebral hemodynamics in the
MTG
No significant effect No significant effect
Nose temperature
Temperatures relative to baseline
decreased in high demand
conditions
No significant effect

31. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
fNIRS Data
31

32. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Results
32
Measure Perceptual load Fatigue
Subjective
Significant Increase, particularly in
Mental Demand, temporal
demand, and effort
Significant increase over time
(e.g. last block compared to first
block)
Heart Rate No significant effect
HR difference from baseline
decreased as time progressed
Breathing Rate
BR difference from baseline
increased in high demand
conditions
BR difference from baseline
increased as time progressed
Cerebral hemodynamics in the
PFC
No significant effect No significant effect
Cerebral hemodynamics in the
MTG
No significant effect No significant effect
Nose temperature
Temperatures relative to baseline
decreased in high demand
conditions
No significant effect

33. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Nose Temperature by Perceptual Load
33

34. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Hypotheses
34
Measure Perceptual load Fatigue
Subjective
MWL Increase with.
Fewer TUTs
Increase over time
Heart Rate Increase with Decrease over time
Breathing Rate Increase with Vary over time
Cerebral hemodynamics in PFC Vary with Vary over time
Cerebral hemodynamics in MTG Vary with Vary over time
Nose temperature Decrease with Vary over time
✅ ✅
✅
✅ ✅
✅
"
"
"
"
"
"

35. Back to the Broader Context

36. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
fNIRS >> Mental Workload
• A few studies have not found significant results in PFC, despite e.g.
significant NASA TLX results.
• 1) Sensitivity?
• Studies have shown that fNIRS cant detect difference between 1-back and 2-back tests
• But can between e.g. 1-back and 3-back
• 2) Measurement Location?
• Mental Workload is presumed to be related to Working Memory, PFC
• Perhaps experienced mental workload goes beyond PFC
• 3) Study Design?
• Breaking out of limited study design is our goal though
36

37. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Maybe Machine Learning Can do it?
37

38. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Instead - Alerts at “moments”
38

39. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Train Signal Operators
39

40. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
• Self Reported Workload (IWS)
• Task Demand
• EDA
• fNIRS
• Heart Rate / HRV
40

41. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
• Self Reported Workload (IWS)
• Task Demand
• EDA
• fNIRS
• Heart Rate / HRV
41

42. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
• Self Reported Workload (IWS)
• Task Demand
• EDA
• fNIRS
• Heart Rate / HRV
42

43. Even Bigger Questions

44. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Brain Data as Personal Data
• What do people think if their employer was looking at this kind of data
• What do people think it means about them?
44

45. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
45

46. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
46

47. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Brain Data as Personal Data
47
Discussion of
MWL
Phase 1
Discussion of
High and Low
MWL
Descriptions
Phase 2
Interpreting
drawings from
interviews
Phase 3
Producing
metaphors based on
descriptions of
someones daily
activities
Phase 4
Phase 5
Prototype
Discussion
Suitable
representation?
Analyse and reflect
on individuals day?

48. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Questions?
48