A human-computer interaction research talk about how we measure mental workload, and how people might reflect on this type of personal data in the future. The research is carried out at the University of Nottingham in the School of Computer Science, involving functional Near Infrared Spectroscopy (fNIRS)

1. Brain-based HCI
What brain data can tell
us about HCI
Max L. Wilson
@gingdottwit

2. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Talk Topic Breakdown
Brain-based HCI
3

3. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Brain-based HCI
• As opposed to Brain-Computer Interaction (BCI)
• Even to passive BCI to some extent
4
Zander & Kothe 2010

4. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Brain-based HCI
• e.g. studying the effect of interruptions using brain-data
• Or indeed the design of them
• Ultimately, we (my team) are interested in studying HCI using brain
data, rather than creating systems that use brain data
• Except for some exceptions
5
Pike (PhD Thesis), 2017

5. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
To-Say List
• Research Background
• Workload Alerts Example
• Studying work with fNIRS
6

6. Research Background

7. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
8

8. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
9

9. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
User Interfaces
10
Great Complexity
and Confusion
With Great Power
Comes

10. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
11
Total Mental Capacity
Simple UI
EasyTask

11. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
12
Total Mental Capacity
Complicated UI
EasyTask

12. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
13
Total Mental Capacity
Complicated UI
DifficultTask

13. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
14
Total Mental Capacity
Better UI
DifficultTask

14. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Mental Workload
15
Sharples, Sarah and Megaw Ted (2015). Definition and mesurement of human workload.
In Wilson John R and Sharples Sarah, editors, Evaluation of human work. CRC Press.

15. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Most important criteria for us
• We can run a ‘normal’ user study.
• As much ecological validity in
- the environment they do the study
- natural user behaviour in the study
- as normal/natural a task as possible
• And tell if theres a cognitive difference between UIs or Tasks
16

16. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
What we need…
Reliable Brain Measurements
of HCI
17

17. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
What we need…
Reliable Brain Measurements
of HCI
We can trust the data from uncontrolled conditions
(not limiting interaction or body position)
18

18. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
What we need…
Reliable Brain Measurements
of HCI
Human-Computer Interaction
(doing work tasks, using software, etc)
19

19. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Which Brain Scanner?
20
M. Pike (2017) Exploring the use of Brain-Sensing Technologies for Natural Interactions. PhD Thesis. University of Nottingham.

20. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Our fNIRS - fully portable
• This is a fully portable fNIRS
- bluetooth connection
- Most commercially user friendly
21
Artinis: Octomon

21. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
fNIRS
• Began in 1977
Frans Jobsis
• 700-900nm sees through tissue/bone
• But is absorbed by hemoglobin
- oxygenated and deoxygenated
• Measure either side of cross-over
- (isobestic point)
• Modified Beer-Lambert Law used to
calculate relative concentration of each
22
By Adrian Curtin - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=20510064

22. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Change in oxygenated hemoglobin
• Px
23

23. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
24

24. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Comparing responses
25

25. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Showing that fNIRS works for HCI
• CHI2014: fNIRS + Think Aloud
• CHI2015: fNIRS vs HCI Artefacts
• CHI2015: 3 UIs create
different Mental Workload
• Horia Maior & Matthew Pike
• Working with Prof. Sarah Sharples
26

26. Workload Alerts
ToCHI Article
Presented at CHI2018

27. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
28

28. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Horia on Computerphile
29
https://www.youtube.com/watch?v=TQ0sL1ZGnQ4

29. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
When do you need to know MWL?
30

30. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
How to measure MWL?
• Performance based
• Subjective techniques
• NASA-TLX
• ISA - Instantaneous self-assessment of workload technique
• Physiological approach
• Indirect: Skin response, heart based measurements, etc.
• Direct: brain measurements: e.g. fNIRS
31

31. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
NASA Task Load Index (TLX)
• Using a 21-point subjective
questionnaire
• Captures 6 subscales:
• Mental Demand
• Physical Demand
• Temporal Demand
• Performance
• Effort
• Frustration
• Developed for demanding jobs
• Physically
• Mentally
32

32. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Instantaneous Self Assessment (ISA)
• ‘Continuous’ subjective technique
(could be an app while performing a study task)
• On Regular intervals during a study
(e.g. every 30 sec)
• Popular in analysis critical industry tasks
• BUT - it interrupts those tasks
33

33. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Workload Alerts: Aim
• What if we provide feedback of workload to people during tasks?
• Assess workload objectively using fNIRS
• Help people manage their workload during tasks
• Facilitate a form of Reflection-in-Action during tasks
(similar effect as ISA?)
34

34. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Related - Dynamic Adjustment, fNIRS
35

35. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Workload Alerts System
36

36. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Workload Alerts System
37
VS

37. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Task: Airport Madness 4
38

38. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Hypotheses (Complex)
39

39. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Hypotheses Simplified
• H1: High Demand »» High MWL
• H2: Performance w/ Feedback > Performance w/ ISA?
• H3: Perception of [H2]
• H4: MWL Feedback »» Taking Action
40

40. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Participants & Conditions
• 32 subjects
• Counterbalanced
• Average age: 25.3
• 4 Conditions * 7 min
41

41. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Dependent Variables
• Demand
• Number of airplanes on the screen every 30 seconds
• Task Performance
• Performance outcomes (number of planes landed, take-offs)
• Perceived performance
• five point rating scale (1 - poor, 5 - excellent)
• Subjective ratings
• ISA - 30 sec interval
• Objective measures
• fNIRS
42

42. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Config & Monitoring
• Tracking MWL
• Config using study task
• Thus also practice round for participants
• 3 variations of demand
• Rest
• Low-Normal Load (3-5 aeroplanes)
• Normal-High Load (>7 aeroplanes)
• State tracking
• Using a rolling average of 30 seconds
• Monitoring the most sensitive channel
43

43. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Choice of Feedback
• Noticeable
• Two states of interest:
• Low workload
• High Workload
• Hue Bulbs (Red and White Colours were used for feedback)
• Two Pronged approach:
• Phase1: Red for High Workload and White for Low workload
• Phase2: Reversed Phase1
44

44. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Choice of Feedback
45

45. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
H1: High Demand »» High MWL
46

46. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
H1: High Demand »» High MWL
47

47. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
H2: Perf. w/ Feedback > Perf. w/ ISA?
48

48. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
H2: Perf. w/ Feedback > Perf. w/ ISA?
49

49. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
H3: Perception of [H2]
50

50. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
H3: Perception of [H2]
51

51. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
H4: MWL Feedback »» Taking Action
52

52. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Takeaways
• ISA increases demand »» impact on performance
• Conversely, no evidence of negative impact of MWL Feedback
• But - we didn’t manage to see change in behaviour.
• RED IS SCARY - not a good feedback mechanism
53

53. Studying Work
And tasks, and
software, etc

54. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Studying Work
• What we’ve done: is put specific tasks in a controlled experiment
• In practice ‘work’ is messy, and we are far from understanding it with
physiological data
• BUT - we are not far from characterising types of cognitive change with
physiological data
55

55. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Studying Work
• RQ1) how are different cognitive changes affecting physiological
measurements of mental workload (in work?)?
• RQ2) how much does mental workload change in the daily work of
information workers? (what does two hours of work look like in fNIRS
data)
• RQ3) what are the cognitive effects of high mental workload periods on
work over time?
• RQ4) how do people reflect on this form of cognitive personal data?
56

56. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
fNIRS affected by Stress?
57
Norah Alsuraykh
Co-supervised by Paul Tennant and Sarah Sharples

57. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
How Stress & MRL are Connected
• Montreal Imaging Stress Task (MIST)
• Maths equally hard - but adds stress
• By time pressure and negative feedback
• Measures:
• Performance, Stress (SSSQ), MWL (NASA TLX)
• fNIRS, EDA, Heart Rate, Skin Temp
58
Alsuraykh et al 2019, Pervasive Health, https://doi.org/10.1145/3329189.3329235

58. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
How Stress & MRL are Connected
59
Alsuraykh et al 2019, Pervasive Health, https://doi.org/10.1145/3329189.3329235

59. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
How Stress & MRL are Connected
60
Alsuraykh et al 2019, Pervasive Health, https://doi.org/10.1145/3329189.3329235

60. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
What does work look like?
• Creative Arts
• Safety Critical (in rail)?
• Information Workers
• Manufacturing Industry
61

61. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
fNIRS during Creative Practice
62
Caroline Locke and Debra Swann

62. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
fNIRS during Creative Practice
63
Caroline Locke and Debra Swann

63. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Train Signal Operators
64

64. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
• Self Reported Workload (IWS)
• Task Demand
• EDA
• fNIRS
• Heart Rate / HRV
65

65. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
• Self Reported Workload (IWS)
• Task Demand
• EDA
• fNIRS
• Heart Rate / HRV
66

66. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
• Self Reported Workload (IWS)
• Task Demand
• EDA
• fNIRS
• Heart Rate / HRV
67

67. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Students studying
68
Discussion of
MWL
Phase 1
Discussion of
High and Low
MWL
Descriptions
Phase 2
Interpreting
drawings from
interviews
Phase 3
Producing
metaphors based on
descriptions of
someones daily
activities
Phase 4
Phase 5
Prototype
Discussion
Suitable
representation?
Analyse and reflect
on individuals day?

68. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Students studying
69

69. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Brain Data as Personal Data
• Has an MSc in Brain Imaging
• Industry Sponsor: Brain+ (Copenhagen)
• Proposed Plan: study diary-like brain data
• Perhaps: in relation to brain training data (Brain+)
70
Serena Midha

70. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Studying Information Workers
71
Easy Medium Hard (Dual)
Reading
Reading
easy text
Reading a
Journal article
Reading an
article +
Finding a word
Writing
Writing an
Email
Describing
research area
Describing
research area +
saying blah

71. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Studying Information Workers
72
Reading Writing Reading (fNIRS)
NASA TLX

72. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Studying Information Workers
• Interruptions interesting!
• Depending on if it stopped
the task, or happened in
parallel!
• People accommodated
interruptions in low MWL
73

73. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
• Next stop - 3-part study
Studying Information Workers
74
Gloria Mark
style 5 day Diary-
based
Lifestyle Study
Qualitative data
from rich
Contextual Inquiry
Interviews
2hr High density
fNIRS data
collection

74. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
DigiTOP Grant
• Digital Toolkit for optimisation of operators and technology
in manufacturing partnerships (DigiTOP)
- £1.9M / 3 years
- PI: Prof Sarah Sharples (Nottingham)
• Global shift towards digital manufacturing techniques
• Physical Workload » Mental Workload
• Traditional Human Factors » Cognitive Ergonomics
75
http://digitop.ac.uk

75. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Facial Thermography vs fNIRS
76
Adrian Marinescu
See Computerphile on YouTube
And many other sensors, including Posture Analysis

76. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Current Study
• Aim: identify overload and underload
• Aim: identify periods of mind-wandering
• 40 participants in Sustained Attention task
• Low demand High Demand
•
77

77. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Example fNIRS data
78

78. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Blink rate during mind-wandering
79

79. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
fNIRS by fatigue
80
PFC MTG

80. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Fatigue biggest differentiator
81
(for visual attention task)

81. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Studying Work
• RQ1) how are different cognitive changes affecting physiological
measurements of mental workload (in work?)?
• RQ2) how much does mental workload change in the daily work of
information workers? (what does two hours of work look like in fNIRS
data)
• RQ3) what are the cognitive effects of high mental workload periods on
work over time?
• RQ4) how do people reflect on this form of cognitive personal data?
82

82. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Questions?
83