Measuring Mental Workload in Interactive Information Retrieval User Studies with fNIRS - a review of how we've used fNIRS in our user studies at the University of Nottingham Mixed Reality Lab

1. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Measuring Mental Workload in IIR
User Studies with fNIRS
Max L.Wilson
University of Nottingham, UK
@gingdottwit
My Brain Team
Norah Alsuraykh, Horia Maior, Matthew Pike, Richard Ramchurn

2. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/

3. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
With Great Power
Comes
Great Complexity
and Confusion

4. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Increasing Cognitive Costs
Total Mental Capacity
Simple UI
EasyTask

5. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Increasing Cognitive Costs
Total Mental Capacity
Simple UI
HardTask

6. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Increasing Cognitive Costs
Total Mental Capacity
Complex UI
HardTask

7. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Mental Workload
the limited resource model [22] describing the relat
between the demands of a task, the resources alloca
e task and the impact on performance.
re 3: Resources available vs task demands
act on performance [22]
Megaw,T. (2005)The definition and measurement of mental workload. Evaluation
of human work, 525-551.

8. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Most Important to My Lab
• We can run a ‘normal’ user study.
• As much ecological validity in
- the environment they do the study
- natural user behaviour in the study
- as normal/natural an IIR task as possible
• And tell whether theres a cognitive difference between UIs

9. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
5 Challenges for NeuroIIR
• 1: Physical Constraints
- its hard to minimise physical constraints
• 2: DesigningTasks
- its hard to design tasks that extract cognitive differences
• 3: ConfoundingVariables
- its hard to stop other artefacts affecting measurements
• 4: Protocol Compatibility
- its hard to integrate neurophysiological sensors into ‘normal studies’
• 5: Data Analysis
- chunking lots of continuous data into task differences is hard

10. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
functional Near Infra-Red Spectroscopy

11. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
4Years of HCI Papers
• CHI2014 - Pike et al. Measuring the effect of Think Aloud
Protocols on Workload using fNIRS.
• CHI2015 - Maior et al. Examining the Reliability of Using fNIRS in
Realistic HCI Settings for Spatial and Verbal Tasks.
• CHI2016 - Lukanov et al. Using fNIRS in Usability Testing:
Understanding the Effect of Web Form Layout on Mental
Workload.
• TOCHI - Maior et al. Workload Alerts - Using Physiological
Measures of Mental Workload to Provide Feedback during
Tasks (under review)

12. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Our Approaches
• fNIRS is actually very good for this.
• Research shows ‘most’ normal computer usage is fine
- although large upper body movements show in our data
• Mostly limited by 1.5-2m cables
- Artinis now sell a portable fNIRS
Challenge 1: Physical Constraints

13. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Our Approaches
• We do use a lot of n-back psych tests
- but mostly for calibration
• We’ve managed to do ‘normal’ tasks
• For CHI2016, we had people fill
in insurance claim forms.
- and saw Cognitive differences
• ForTOCHI, we had people doing
AirTraffic Control taks
Challenge 2: DesigningTasks
By Pedros.lol - Own work, CC BY-SA 4.0,
https://commons.wikimedia.org/w/
index.php?curid=39241201

14. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Our Approaches
• We’re just learning about this.
• We’ve noticed that anxiety/stress is affecting results
• Norah is starting a PhD on these
- we’ve started using empatica in parallel with fNIRS
Challenge 3: ConfoundingVariables

15. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Our Approaches
• We’ve recently noticed that anxiety/stress is affecting results
• Norah is starting a PhD on this confounding variable
- we’ve started
using empatica
in parallel
with fNIRS
• Other
confounding
variables?
Challenge 3: ConfoundingVariables

16. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Our Approaches
• Part of ‘normal user studies’ is using other protocols
- likeThink Aloud (see our CHI2014 paper)
• Because fNIRS is not very restrictive, and tolerant of artefacts
- its essentially fine to design a study, and ‘add fNIRS’
• Just involves adding setup times
- we use Rest, 1-back, and 3-back to calibrate
Challenge 4: Protocol Compatibility

17. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Our Approaches
• We collect data with COBI (from fNIRS supplier) » CSV files
- 16 channels, so we target sensitive regions for the task
- We chunk those channels into task periods (& shift by BOLD)
• Using fNIRSoft
- We use low-pass filtering etc to remove noise
- and Correlation Based Signal Improvement (CBSI)
(comparison between HbO and Hb signals)
- and produce some visualisations
• Then do multivariate ANOVA type statistics (channels x conditions)
Challenge 5: Data Analysis

18. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Limitations of fNIRS
• Its temporal resolution is slower than others (2hz)
- and affected by BOLD response
• Its not good for measuring other locations/cognitive responses
- although full-scalp fNIRS exist
• Doesn't provide you imaging or location identification like MRI

19. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Questions?