This document describes MindScope, an algorithm-assisted stress management system for college students. It collects behavioral data from students' smartphones to predict their stress levels and provides explanations of the predictions. A study was conducted with 36 students who used MindScope for 25 days. It found that providing algorithmic explanations for stress predictions helped reduce students' perceived stress levels over time. It also supported their stress management practices and self-reflection. However, the study was limited by its short duration and lack of a control group.

1. 2022.09.22
Prediction for Retrospection: Integrating
Algorithmic Stress Prediction into Personal
Informatics Systems for College Students’
Mental Health
CHI '22
Taewan Kim, Haesoo Kim, Ha Yeon Lee, Hwarang Goh, Shakhboz Abdigapporov, Mingon Jeong, Hyunsung Cho,
Kyungsik Han, Youngtae Noh, Sung-Ju Lee, Hwajung Hong

2. 2
Personal Informatics System
• Help to gain insights from self-reflection on past behaviors
• Use passive sensing technology (smartphones and wearable devices) to
collect behavioral data
• Apply machine learning technologies to suggest proper actions from
insights and patterns
• Assist users’ self-reflection by incorporating algorithm and explainability

3. 3
MindScope
Algorithm-assisted stress management system
• Determine user stress
levels
• Explain how the stress
level was computed
Purpose Data Collection Process
• Social
• Location
• Activity
• Sleep
• Phone usage
1) Data collection phase:
get current stress level
as an input four times
a day (10 days)
2) Reflection phase:
report current stress
level determined by
ML model and
explanation of
predictors four times
a day(15 days)

4. 4
MindScope
Type 1: Predicted stress level
Type 2: Highlighted data category
Type 3: More detailed explanation
Level of explanability
How explanation of prediction affect users’ perception of the system
and self-reflection experience

5. 5
System Design
• What to explain
• How to explain
• Level of detail in
explanation
Iterative Design Process
• Pilot study with 30 college students

6. 6
System Design
• Data
• Accelerometer and GPS data, app usage, app type, noise levels
in the surrounding environment
• Commonly visited locations as context information
• Stress data (low, moderately high, high) through ecological
momentary assessment (EMA) at 4-hour intervals (11am, 3pm,
7pm, 11pm)
• Qualitative information in a form of hashtag
Modeling Phase

7. 7
System Design
• Personalized Stress Prediction Reports
• Type 1: Only stress level
• Type 2: Categories highlighted among phone usage, social activity,
movement, physical action and sleep
• Type 3: Granularized context such as major deviations from the norm
• Receive 4 reports a day
• Confirm whether the prediction was correct
• Answer whether the explanation was useful
• Answer issue with the explanation every three times of not useful
Prediction Phase

8. 8
System Design
Prediction Phase
• Personalized Stress Prediction Reports

9. 9
System Design
• Stress Interventions through microtasks
• Detect opportune moments to
perform the microtask
• To relieve stress
• Send a push notification to suggestion
the intervention’s execution
• Detect habitual phone usage
• Setup, new suggestion, something else
• Log completion
Prediction Phase

10. 10
Implementation Details
• Social activity
• Number of incoming and outgoing phone calls, call duration, missed call
• Audio loudness (every 20 minutes for 5 seconds)
• Silence threshold -65db
• Location
• 5-minute periodic check-up and 10 meters threshold
• Sleep
• Screen off durations from 6:00 pm to 10:00am
• Physical action
• Activity Recognition and Transition API
• Still, walking, running, riding a bicycle, on a vehicle
• App usage (when screen is unlocked)
Data Collection (29 data features)

11. 11
Implementation Details
Machine Learning
• gRPC server
• Data preprocessing
• Synchronization
• Normalization
• XGBoost
• To combine multiple decision
trees
• Shapley Additive explanations (SHAP)
• To measure feature importance

12. 12
Method
• 36 participants
• Perceived Stress Scale (PSS) – 10 items from 0 to 4
• Before / 10 days of modeling / after
• User Experience Questionnaire – 15 UX-related items with 7 Likert scale
• Application Usage Log – MindScope app
• Follow-up Interview
• Motivation, stress-management skills, effect of app to perception of
stress, acceptance of algorithm, usability
30-minute introductory 25-day field deployment 40-minute interview

13. 13
Analysis
• PSS scores
• One-way repeated measures ANOVA with Greenhouse-Geisser
correction
• Tukey’s HSD for post-hoc analysis
• Open coding with thematic analysis by 3 researchers
• How MindScope supported users’ stress management behavior
• How users perceive prediction results
Quantitative
Qualitative

14. 14
Result
• User engagement
• 78% of average response rate for the modeling phase
• 85% for the prediction phase
• 86% users used intervention feature at least once
• Stress level changes
• Majority of self-reported stress levels were low and number of low
increased slightly from modeling to prediction phase (56.3%, 61.5% resp.)
• Ratio of reports having high stress slightly decreased slightly (8.5%, 5.8%
resp.)
Descriptive Statistics

15. 15
Result
• Changes in prediction
• 63.89% of cases consistent with self-reported stress level
• 36.1% adjusted by the user
• 59.97% of perceived accuracy
• Retraining process in the prediction phase with confirmed stress
level raise to 68% of performance
Descriptive Statistics

16. 16
Result
• Stress significantly reduced first,
then persisted (A)
• It is better than no explanation,
but more doesn’t make it useful (B)
• The more accurate the algorithm is
perceived, the more useful it is
Statistical Finding

17. 17
Qualitative Findings
• The impact of algorithmic stress prediction and explainability on stress
management practices
• Identifying stress patterns through a data-driven approach
• Supporting reminiscence with data about the past
• Planning actionable stress intervention to offset identified stressors
• User perceptions toward stress prediction algorithms
• More data input, the better prediction accuracy
• Challenges in data collection
• Privacy concerns over personal data

18. 18
Qualitative Findings
• The level of explainability and self-reflection for managing stress
• Type 3 (62% preferred)
• Guide specific action to mitigate the stress
• Enable to reconstruct the past
• Should guarantee accuracy
• Type 2 (27%)
• Allow proactive reflection by investigating stressor
• Value more trustworthy
• Require less effort to analyze
• Type 1 (10%)
• Easy to analyze
• Explanability uncovered the reasoning process

19. 19
Discussion
• Prediction for retrospection: Exploiting algorithms to facilitate technology mediated
reflection
• Objective algorithm decisions can supplement subjective judgement
• Prediction with explanation is more useful and effective
• Increase in level of explainability makes more sensitive to the accuracy
• Explainability for supporting self-reflection in the algorithm-assised PI system
• Stress prediction visualization for promoting user initiative and behavior change
• Open-ended algorithmic stress prediction visualization for promoting self
reflection

20. 20
Discussion and Limitation
• Design suggestions for algorithm-assisted self-reflection in PI system
• Progressively improve explainability as the interaction with the system
increases and the model is trained sufficiently
• Co-performing relationship between a user and an algorithmic system is critical
to encourage active participation
• Short period of examination
• No control group
• Not strictly counterbalanced explanation types
Limitation

21. Thank you