Presentation at the annual Stress-nl meeting

1. Discover theworld at Leiden UniversityDiscover theworld at Leiden University
Unobtrusive sensors for
measuring well-being at
work
Wessel Kraaij
Saskia Koldijk (UMCU), Anne Marie Brouwer (TNO), Koen Hogenelst (TNO), Mark Neerincx (TNO, TU Delft
1

2. Discover theworld at Leiden University
Challenges / Outline
• How can we measure stress related phenomena at work at high resolution for a
longer period?
• Cortisol measurements (hair, saliva) have disadvantages
• Galvanic skin response sometimes generates artifacts
• Challenges: non-intrusive measurement of subjective affect, combining modalities
• Experiment 1: Controlled study (Koldijk et al), rich sensor data, low resolution of
labels
• Experiment 2: Pseudo real life study (Brouwer et al), selected modalities, high
frequency sampling of affect (arousal/valence)
2

3. Project focus: help alter bad habits of knowledge workers and
improve creativity, effectiveness
SWELL, one of the COMMIT/ (FES)
projects
2011-2018
www.swell-project.net

4. 4
Affective
computing
Privacy respecting
data processing
Causal inference
Real time
classification
Combination of
unobtrusive sensors

5. 5
Personalized
context sensitive
recommendation
Explainable AI
Learning to coach
from user feedback
Persuasive
technology

6. Controlled office work study (Koldijk)
▪ Promising results with unobtrusive stress monitoring
▪ Methodology for designing stress interventions, grounded in
theory and operationalized by sensor technology and taking
into account privacy concerns
▪ Several prototype m-health apps
Koldijk, S., Bernard, J., Ruppert, T., Kohlhammer, J., Neerincx, M.A., & Kraaij, W. (2015). Visual Analytics of Work Behavior Data -
Insights on Individual Differences. In: Proceedings of EuroVis 2015
Koldijk, S., Neerincx, M.A., Kraaij, W., Detecting work stress in offices by combining unobtrusive sensors , (2016) IEEE Transactions
on Affective Computing
Koldijk, S., Kraaij, W. & Neerincx, M.A. (2016). Deriving Requirements for Pervasive Well-Being Technology From Work Stress and
Intervention Theory: Framework and Case Study. JMIR Mhealth Uhealth 2016;4(3):e79.

7. Research on: Sensing & Interpretation
Perceived
Stress (VAS)
Emotion
(SAM)
Mental Effort
(RSME)
Task Load
(NASA-TLX)
Controlled
variable:
induced stress
Outcomes

8. How could stress or related aspects be
measured with (unobtrusive) sensors?
Unobtrusive
measurements

9. The SWELL Knowledge work dataset for
stress and user modeling research [koldijk,2014]
9
▪ Controlled experiment with 25 subjects
▪ Three blocks: neutral, stressor 1, stressor 2
▪ Sensor measurements and questionnaires
Koldijk, S., Sappelli, M., Verberne, S., Neerincx, M.A., & Kraaij, W. (2014). The SWELL
Knowledge Work Dataset for Stress and User Modeling Research. In: Proceedings of the
16th ACM International Conference on Multimodal Interaction (ICMI 2014)
Kraaij, Prof.dr.ir. W. (Radboud University & TNO); Koldijk, MSc. S. (TNO & Radboud
University); Sappelli, MSc M. (TNO & Radboud University) (2014): The SWELL Knowledge
Work Dataset for Stress and User Modeling Research. DANS.
http://dx.doi.org/10.17026/dans-x55-69zp
Sappelli, M., Verberne, S., Koldijk, S., & Kraaij, W. (2014). Collecting a dataset of
information behaviour in context. In: Proceedings of the 4th Workshop on Context-
awareness in Retrieval and Recommendation (CARR @ ECIR 2014) (Amsterdam, The
Netherlands, 13-16 April 2014).
Feature values are
averaged per
minute

10. Identifying the working condition
▪ Conclusion: Sensors do record different behaviour in
stressor conditions
10 fold cross validation

11. Modality comparison
▪ Conclusion: Posture is the strongest feature modality, Facial helps

12. Predicting subjective mental state/effort
▪ Sensor data seems to be
most powerful for
predicting ‘mental effort
(RMSE)’
▪ Model tree regression
best performer
(correlation 0.83)
▪ Facial expression features
are best predictors,
followed by posture.

13. How important are individual differences?
▪ Identifying working condition:
▪ Full population based SVM : 90% (majority baseline 62%)
▪ Adding participant ID: no change!
▪ Test on unseen user: (leave one out): average 59% (min 37.5,
max 88.34)
▪ => performance for new users might be very low
▪ Predicting mental state:
▪ Full population based regression model : 0.83
▪ Adding participant ID: 0.94
▪ Test on unseen user: (leave one out): 0.03
▪ => performance for new users might be very low
▪ Especially the second task is sensitive to individual
differences

14. Towards a subtype based analysis
▪ Idea: cluster subjects, train and evaluate classifier for
subtypes
▪ Hierarchical clustering (for determining k) and k-
mean
Modality Subtype1 Subtype2 Subtype3 General
Computer Writers(16)
0.17
Copy-pasters(9)
0.34 0.15
Facial Low expression (16)
0.79
Eyes wide &
mouth tight(3)
0.81
Tight eyes &
loose mouth (6)
0.87 0.81
Posture Sits still & moves
right arm (5)
0.76
Restless body &
calm wrist 6)
0.85
Average
movement (14)
0.69 0.59

15. MONITORING EMOTIONAL STATE DURING REAL
LIFE OFFICE WORK
17 individuals rated emotional state every 15 minutes during 10 whole working days
We found associations between currently experienced emotion and heart rate, facial expression,
keylogging and task switches, initial arousal (hi/lo) classifier at 60%
Strong adherence to the task in our approach, participants’ emotions seem to vary enough, and they are
able to indicate their emotion precise enough to be able to capture this
This opens the way for finegrained continuous monitoring in real life (application and research)
Aroused
Unpleasant
Pleasant
Calm
Anne-Marie Brouwer
Loïs van de Water
Maarten Hogervorst
Koen Hogenelst
Bart Joosten
Thymen Wabeke
Jan-Willem Streefkerk
Anne-Marie Brouwer, Lois van de Water, Maarten Hogervorst, Wessel Kraaij, Jan Maarten Schaagen, and Koen Hogenelst. Monitoring
mental state during real life office work. In Proceedings of Symbiotic 2017, Eindhoven, 2017.

16. Conclusions
▪ SWELL KW dataset multimodal dataset for research in user modeling and
affective computing
▪ We can distinguish stressful working conditions in a controlled setting using
non obtrusive sensors (posture best feature)
▪ Mental effort can best be estimated using facial expression data
▪ Individual differences play a significant role
▪ Data has been collected in a proximate ‘real life’ setting
▪ Affect grid (15min) judged as pleasant, easy and quick to use. Usually not very
interfering or bothering.
▪ Unobtrusive measures were indeed experienced as unobtrusive. Most
participants reported that the recordings did not make them behave
differently
▪ Next steps:
▪ Investigate individual models (for improved mapping of physiology to mental
state)
▪ Monitor workers across longer period (and use hair cortisol biomarkers as
ground truth) 16

17. Discover theworld at Leiden UniversityDiscover theworld at Leiden University
More info at
www.swell-project.net
17