A human-computer interaction research talk about how we measure mental workload, and how people might reflect on this type of personal data in the future. The research is carried out at the University of Nottingham in the School of Computer Science, involving functional Near Infrared Spectroscopy (fNIRS)

1. Mental Workload Alerts
Reliable Brain Measurements
of HCI using fNIRS
Max L. Wilson
@gingdottwit

2. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
2

3. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Horia on Computerphile
3
https://www.youtube.com/watch?v=TQ0sL1ZGnQ4

4. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
To-Say List
• Context
• Workload Alerts
• Future Directions
4

5. Context

6. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
6

7. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
7

8. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
User Interfaces
8
Great Complexity
and Confusion
With Great Power
Comes

9. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
9
Total Mental Capacity
Simple UI
EasyTask

10. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
10
Total Mental Capacity
Complicated UI
EasyTask

11. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
11
Total Mental Capacity
Complicated UI
DifficultTask

12. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
12
Total Mental Capacity
Better UI
DifficultTask

13. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Mental Workload
13
Sharples, Sarah and Megaw Ted (2015). De nition and mesurement of human workload.
In Wilson John R and Sharples Sarah, editors, Evaluation of human work. CRC Press.

14. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Most important criteria for us
• We can run a ‘normal’ user study.
• As much ecological validity in
- the environment they do the study
- natural user behaviour in the study
- as normal/natural a task as possible
• And tell if theres a cognitive difference between UIs or Tasks
14

15. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Which Brain Scanner?
15
M. Pike (2017) Exploring the use of Brain-Sensing Technologies for Natural Interactions. PhD Thesis. University of Nottingham.

16. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
fNIRS
16
functional Near InfraRed Spectroscopy

17. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Showing that fNIRS works for HCI
• CHI2014: fNIRS + Think Aloud
• CHI2015: fNIRS vs HCI Artefacts
• CHI2015: 3 UIs create
different Mental Workload
• Horia Maior & Matthew Pike
• Working with Prof. Sarah Sharples
17

18. Workload Alerts
ToCHI Article
Presented at CHI2018

19. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
When do you need to know MWL?
19

20. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
How to measure MWL?
• Performance based
• Subjective techniques
• NASA-TLX
• ISA - Instantaneous self-assessment of workload technique
• Physiological approach
• Indirect: Skin response, heart based measurements, etc.
• Direct: brain measurements: e.g. fNIRS
20

21. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
NASA Task Load Index (TLX)
• Using a 21-point subjective
questionnaire
• Captures 6 subscales:
• Mental Demand
• Physical Demand
• Temporal Demand
• Performance
• Effort
• Frustration
• Developed for demanding jobs
• Physically
• Mentally
21

22. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Instantaneous Self Assessment (ISA)
• ‘Continuous’ subjective technique
(could be an app while performing a study task)
• On Regular intervals during a study
(e.g. every 30 sec)
• Popular in analysis critical industry tasks
• BUT - it interrupts those tasks
22

23. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Workload Alerts: Aim
• What if we provide feedback of workload to people during tasks?
• Assess workload objectively using fNIRS
• Help people manage their workload during tasks
• Facilitate a form of Reflection-in-Action during tasks
(similar effect as ISA?)
23

24. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Related - Dynamic Adjustment, fNIRS
24

25. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Workload Alerts System
25

26. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Workload Alerts System
26
VS

27. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Task: Airport Madness 4
27

28. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Hypotheses (Complex)
28

29. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Hypotheses Simplified
• H1: High Demand »» High MWL
• H2: Performance w/ Feedback > Performance w/ ISA?
• H3: Perception of [H2]
• H4: MWL Feedback »» Taking Action
29

30. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Participants & Conditions
• 32 subjects
• Counterbalanced
• Average age: 25.3
• 4 Conditions * 7 min
30

31. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Dependent Variables
• Demand
• Number of airplanes on the screen every 30 seconds
• Task Performance
• Performance outcomes (number of planes landed, take-offs)
• Perceived performance
• five point rating scale (1 - poor, 5 - excellent)
• Subjective ratings
• ISA - 30 sec interval
• Objective measures
• fNIRS
31

32. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Config & Monitoring
• Tracking MWL
• Config using study task
• Thus also practice round for participants
• 3 variations of demand
• Rest
• Low-Normal Load (3-5 aeroplanes)
• Normal-High Load (>7 aeroplanes)
• State tracking
• Using a rolling average of 30 seconds
• Monitoring the most sensitive channel
32

33. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Choice of Feedback
• Noticeable
• Two states of interest:
• Low workload
• High Workload
• Hue Bulbs (Red and White Colours were used for feedback)
• Two Pronged approach:
• Phase1: Red for High Workload and White for Low workload
• Phase2: Reversed Phase1
33

34. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Choice of Feedback
34

35. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
H1: High Demand »» High MWL
35

36. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
H1: High Demand »» High MWL
36

37. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
H2: Perf. w/ Feedback > Perf. w/ ISA?
37

38. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
H2: Perf. w/ Feedback > Perf. w/ ISA?
38

39. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
H3: Perception of [H2]
39

40. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
H3: Perception of [H2]
40

41. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
H4: MWL Feedback »» Taking Action
41

42. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
H4: MWL Feedback »» Taking Action
42

43. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Takeaways
• ISA increases demand »» impact on performance
• Conversely, no evidence of negative impact of MWL Feedback
• But - we didn’t manage to see change in behaviour.
• RED IS SCARY - not a good feedback mechanism
43

44. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Future Directions from this work
• More advanced monitoring techniques
• Best ways of giving feedback
• Can we get people to reflect-in-action, and change behaviour
44

45. Future Directions

46. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
fNIRS affected by Stress?
46
Norah Alsuraykh
Co-supervised by Paul Tennant and Sarah Sharples

47. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
New fNIRS - fully portable!
• This is a fully portable fNIRS
- bluetooth connection
- Most commercially user friendly
47
Artinis: Octomon

48. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
fNIRS during Creative Practice
48
Caroline Locke and Debra Swann

49. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
fNIRS during Creative Practice
49
Caroline Locke and Debra Swann

50. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Brain Controlled Film - The MOMENT
50
http://www.imdb.com/title/tt7853742/
Written & Directed by Richard Ramchurn

51. Brain Data as Personal Data
A Fitbit for the brain?

52. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Brain Data as Personal Data
• Can he measure fNIRS in the wild?
• A series of studies planned
- in increasingly mobile contexts
- additional physiological data
• Bringing in some machine learning
- mentored by Michel Valstar
52
Horia Maior
EPSRC Prize
(2 years postdoc)

53. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Brain Data as Personal Data
53
Discussion of
MWL
Phase 1
Discussion of
High and Low
MWL
Descriptions
Phase 2
Interpreting
drawings from
interviews
Phase 3
Producing
metaphors based on
descriptions of
someones daily
activities
Phase 4
Phase 5
Prototype
Discussion
Suitable
representation?
Analyse and reflect
on individuals day?

54. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Brain Data as Personal Data
54
Discussion of
MWL
Phase 1
Discussion of
High and Low
MWL
Descriptions
Phase 2
Interpreting
drawings from
interviews
Phase 3
Producing
metaphors based on
descriptions of
someones daily
activities
Phase 4
Phase 5
Prototype
Discussion
Suitable
representation?
Analyse and reflect
on individuals day?

55. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Brain Data as Personal Data
• Has an MSc in Brain Imaging
• Industry Sponsor: Brain+ (Copenhagen)
• Proposed Plan: study diary-like brain data
• Perhaps: in relation to brain training data (Brain+)
55
Serena Midha
New PhD Student

56. New EPSRC Grant
EP/R032718/1

57. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
DigiTOP Grant
• Digital Toolkit for optimisation of operators and technology
in manufacturing partnerships (DigiTOP)
- £1.9M / 3 years
- PI: Prof Sarah Sharples (Nottingham)
• Global shift towards digital manufacturing techniques
• Physical Workload » Mental Workload
• Traditional Human Factors » Cognitive Ergonomics
• Aim: Theory + Standards + Toolkit for ‘Industrie 4.0’
57

58. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Facial Thermography vs fNIRS
58
Adrian Marinescu
See Computerphile on YouTube
And many other sensors, including Posture Analysis

59. New Horizon 2020 Grant
Fast Track to Innovation Fund

60. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Alzheimer’s Protection Shield
• Pre-symptomatic Alzheimer’s disease detection & prevention
- H2020 Fast Track to Innovation Fund (€2.4M)
• Brain+, Aarhus, Oxford, Nottingham
- Mixed Reality Lab & MindTech
• Overall - develop ‘Brain Coaching’ app from Brain+
• Nottingham Aims:
- Longitudinally (5 points) study non-patient adults
- Patient Engagements with Alzeimer’s patient groups
- Develop early evidence for future NHS trials
60

61. Dr Max L. Wilson http://cs.nott.ac.uk/~pszmw
Questions?
61