A human-computer interaction research talk about how we measure mental workload, and how people might reflect on this type of personal data in the future. The research is carried out at the University of Nottingham in the School of Computer Science, involving functional Near Infrared Spectroscopy (fNIRS)

1. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Measuring & Reflecting
on Mental Workload
Max L.Wilson
Uni Birmingham, May 2017

2. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/

3. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
With Great UI Power
Comes
Great Complexity
and Much Confusion

4. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Increasing Cognitive Costs
Total Mental Capacity
Simple UI
EasyTask

5. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Increasing Cognitive Costs
Total Mental Capacity
Simple UI
HardTask

6. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Increasing Cognitive Costs
Total Mental Capacity
Complex UI
HardTask

7. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Mental Workload
the limited resource model [22] describing the relat
between the demands of a task, the resources alloca
e task and the impact on performance.
ure 3: Resources available vs task demands
act on performance [22]
Megaw,T. (2005)The definition and measurement of mental workload. Evaluation
of human work, 525-551.

8. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Cognitive LoadTheory
Total Mental Capacity
Simple UI
EasyTask
Intrinsic Load
Extrinsic Load
Germane Load

9. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Shneiderman & Plaisant
Total Mental Capacity
Simple UI
EasyTask
Semantic Load
Syntactic Load
Spare Load

10. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Cognitive LoadTheory
(on Duel CodingTheory)
https://dywestblog.wordpress.com/designing-multimedia-instruction-for-
professional-development-in-an-efl-environment-a-cognitive-perspective-final/

11. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Research Phases
• 1 & 2) Establish a way to measure mental workload reliably
(CHI2014 and CHI2015)
• 3) Demonstrate it can be used to separate out User Interfaces
(CHI2016)
• 4) Study how people respond to this data
(In Submission)
• 5) Design for giving people this data in life
(In Progress)

12. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Brain HCITeam
• Co-Investigators
Horia Maior Matt Pike Richard Ramchurn Norah Alsuraykh
Sarah Sharples Steve Benford Sarah Martindale

13. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Most Important to MyTeam
• We can run a ‘normal’ user study.
• As much ecological validity in
- the environment they do the study
- natural user behaviour in the study
- as normal/natural an IIR task as possible
• And tell whether theres a cognitive difference between UIs

14. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Measuring Mental Workload
Nasa-TLX fMRI
EEG
fNIRS

15. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
functional Near Infra-Red Spectroscopy

16. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Prior work on fNIRS in HCI
, partici-
been re-
wers, we
nd found
ed meas-
act types
F(4,36)=
periment
nology in
derations,
arized in
t several
measure-
physical
ith accu-
Overall,
se typing
uld avoid
heartbeat
There are
educe the
adaptive
ering [5,
analysis
n be used
ously on
hods are
ctical for when using fNIRS. By using the guidelines described
above, researchers can have access to the user’s cognitive
state in realistic HCI laboratory conditions. This is impor-
Table 1. Summary of considerations. Legend: 9 in-
dicates acceptable, C indicates to correct, and 8 in-
dicates to avoid or control.
Considerations
Result
Reference
Correction
Methods
Forehead movement 8 Exp 4
Major head movement 8 Exp 3 Use chin rest
Minor head movement C
Exp 3,
[20]
Filter
Respiration and Heart-
beat
C [4, 20] Filter
Mouse Clicking 9 Exp 2
Collect signal during
a clicking only task
Typing 9 Exp 1
Ambient Light C [3] Wear isolating cap
Hemodynamic
Response
9 [1]
Expect 6-8s
response
Ambient Noise C [22]
Minimize external
noise
Eye Movement and
Blinking
9 [16]
Solovey et al, UIST2009
Showed that fNIRS was
much more reliable for
use in normal user
study conditions than
other brain scanners

17. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Peck et al, CHI2013
n-back tests with pie chart vs bar chart

18. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
1) RepliCHI & Extend
• Repeated Solovey et al (2009) but with added Spatial Task
& added Verbal Artefact
• 15 participants
CHI2015 Paper

19. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
• Instead of remembering a 7 digit number
• Repeated 8 times for each artefact+task condition
1) RepliCHI & Extend
CHI2015 Paper

20. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
1) RepliCHI & Extend
• Confirmed Solovey’s results
• Also - typing affects verbal
- and head movement affected spatial less
CHI2015 Paper

21. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
2) UnderstandingVerbal
• Can we really haveThink Aloud + fNIRS in normal studies?
- how far can this be taken?
• Conditions
- silent
- blah blah blah
- passiveThink Aloud
- invasiveThink Aloud
CHI2014 Paper

22. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
2) UnderstandingVerbal
CHI2014 Paper
Task involves more reasoning
Verbalising is during task, not in memory gap period

23. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
2) UnderstandingVerbal
• Performance wasn't significant
- Speed of solving was slower
for InvasiveTA (expected)
• NASATLX showed diff
between silent & blah blah blah
• fNIRS showed a difference ———>
- blah vs everything else
• High performers slowed down by
InvasiveThink Aloud
CHI2014 Paper

24. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
3) Evaluating User Interfaces
• Personal Info
• Accident Info
• Summary
description
CHI2016

25. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
3) Evaluating User Interfaces
CHI2016

26. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
• fNIRS showed significant difference
• NASATLX too - but people preferred 3! Why?
3) Evaluating User Interfaces
CHI2016

27. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
4) Responding to MWL Data
• If you make people aware of their Mental Workload levels
- Do they change their behaviours to manage it?
• Needs an even more real task
- one that they can adapt to manage Mental Workload
• And a way of showing people their Mental Workload
- so they can reflect-in-action
TOCHI (under revisions)

28. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw

29. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
4) Responding to MWL Data
TOCHI (under revisions)

30. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
• Instantaneous Self Assessment
- is like short NASATLX - but during task (every 30s)
4) Responding to MWL Data
TOCHI (under revisions)

31. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
4) Responding to MWL Data
TOCHI (under revisions)

32. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
4) Responding to MWL Data
TOCHI (under revisions)

33. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
4) Responding to MWL Data
TOCHI (under revisions)
“So whenever it went red, it was kind of stressful ... and I felt like why are
you showing me red if I am stressed? ... It was NOTICEABLE!”

34. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw

35. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
4) Responding to MWL Data
TOCHI (under revisions)

36. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
5) Reflecting on MWL Data
• Diary Study - 12 People, Hourly submissions over 3 days
- whats been hard and whats been easy
- objective score & description
• Followed by semi-structured interviews
- sketching exercises
In Progress - Phase 1

37. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
• Various Metaphors
Used
• Mostly broken into
- continuous views
- discrete views
5) Reflecting on MWL Data

38. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
• Produced 2 primary design implications (+ 3 others)
5) Reflecting on MWL Data
Design Probes used for Focus Groups

39. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Discussion of
MWL
Phase 1
Discussion of
High and Low
MWL
Descriptions
Phase 2
Interpreting
drawings from
interviews
Phase 3
Producing
metaphors based on
descriptions of
someones daily
activities
Phase 4
Phase 5
Prototype
Discussion
Suitable
representation?
Analyse and reflect
on individuals day?
5) Reflecting on MWL Data
Focus Groups

40. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
5) Reflecting on MWL Data
Issue 1:What MWL is Desirable?

41. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
5) Reflecting on MWL Data
Issue 2: How do you feel about it?

42. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Future Work 1
• New student focused on
• Stress vs Workload
- or maybe Affect vs Workload (TBC)
New Student: Norah Alsuraykh

43. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Future Work 2
• Can you tell MWL from a wristband?
-TJ can. so far up to ~75% (low, med, high)
New Student:TJ Broodryk

44. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Research Phases
• 1 & 2) Establish a way to measure mental workload reliably
(CHI2014 and CHI2015)
• 3) Demonstrate it can be used to separate out User Interfaces
(CHI2016)
• 4) Study how people respond to this data
(In Submission)
• 5) Design for giving people this data in life
(In Progress)

45. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Brain Controlled Film
CC2015, CHI2016 + work in submission

46. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Taxonomy of Indirect Control

47. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Social Brain Controlled Film

48. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Any Questions?
https://braincontrolledmovie.wordpress.com/
New Brain Controlled Film - Coming 2018