1. iSenseStress: Assessing Stress
Through Human-Smartphone
Interaction Analysis
Matteo Ciman1, Katarzyna Wac2 and Ombretta Gaggi1
1 University of Padua
Padua, Italy
2 University of Geneva and University of Copenhagen
Geneva, Switzerland and Copenhagen, Denmark

2. PervasiveHealth 2015, Istanbul, Turkey /26
Stress Experience
• Stress is mental condition experienced every day
• Long exposure can lead to anxiety, depression etc. =>
increase of healthcare costs
• In 2013 American teens reported stress experienced at
unhealthy levels (and at increasing lower ages)
[http://www.apa.org/news/press/releases/stress/2013/teenstress.aspx]
2
• Early assessment of stress condition can help to
provide feedback to improve health state of individuals

3. PervasiveHealth 2015, Istanbul, Turkey /26
Stress Assessment - State of the Art
• Stress assessment using wearable and ubiquitous
devices can increase individuals’s acceptance without
interfering with their life
• MouStress: project for stress assessment considering
computer mouse movements or keyboard [1]
• Usage smartphone sensors (WiFi, GPS, Bluetooth,
calls, SMS) [2]
3
[1] D. Sun, P. Paredes, and J. Canny, “Moustress: Detecting stress from mouse motion,” in SIGCHI Conference on
Human Factors in Computing Systems, 2014, pp. 61–70.
[2] G. Bauer and P. Lukowicz, “Can smartphones detect stress-related changes in the behaviour of individuals?” in
PERCOM Workshops, 2012.

4. PervasiveHealth 2015, Istanbul, Turkey /26
The idea
4

5. PervasiveHealth 2015, Istanbul, Turkey /26
Our Approach
• No external devices used, just smartphone (less expensive,
more usable)
• No privacy-related information (i.e., calls, messages,
location etc.)
• Possible to run a phone background service all the day long
• Based on human-smartphone interaction analysis
• Limitation: an interaction with the smartphone is required to
make an assessment
5

6. PervasiveHealth 2015, Istanbul, Turkey /26
Human-Smartphone Interaction
6
Tap
Scroll
Swipe
Text
Writing
Double
Tap
Rotate
Zoom
Pinch
Long
press

7. PervasiveHealth 2015, Istanbul, Turkey /26
Tasks Definition
• Search Task:
• Scroll, swipe and tap
• Write Task:
• Tap, Text Writing
7
Tap Scroll Swipe
Tap Text Writing

8. PervasiveHealth 2015, Istanbul, Turkey /26
Search Task
• Find inside a 21x15 grid
the right icon (s)
• Scroll and Swipe to
inspect all the icons
• Tap to select the right
icon
8

9. PervasiveHealth 2015, Istanbul, Turkey /26
Search Task Features
• Tap {min, max, average} pressure / length / size
• Scroll, Swipe
• {min, max, average} speed / time length /
acceleration / pixels length / pressure
• Linearity
• D(interaction, center), D(interaction, top_left_screen)
9

10. PervasiveHealth 2015, Istanbul, Turkey /26
Write Task
• Paragraph writing as
text message
• Keyboard without
autocorrection or word
suggestion
• English as text
language
10

11. PervasiveHealth 2015, Istanbul, Turkey /26
Write Task Features
• Tap {min, max, average} pressure, length, size
• Tap movement and duration
• Writing:
• Speed
• # errors
• Back digits
11

12. PervasiveHealth 2015, Istanbul, Turkey /26
Protocol
12
Initial
Relax
(5’)
Relaxed
Tasks
(~30’)
Stressor
(5’-10’)
Stressed
Tasks
(~10’)
Self Assessment
Negative Valence
Low energy
Not Stressed
Positive Valence
High Energy
Stressed
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
ESM 1 ESM 2 ESM 3 ESM 4 ESM 5

13. PervasiveHealth 2015, Istanbul, Turkey /26
Protocol (II)
13
Initial
Relax
(5’)
Relaxed
Tasks
(~30’)
Stressor
(5’-10’)
Stressed
Tasks
(~10’)

14. PervasiveHealth 2015, Istanbul, Turkey /26
How to Stress People
• Most common used stressors for tests
• Mathematical problems
• Timing pressure
• Social evaluation
• Repetition
• Uncontrollability
14
We used these

15. PervasiveHealth 2015, Istanbul, Turkey /26
Stressor Task: Math
15
Large
prime number

16. PervasiveHealth 2015, Istanbul, Turkey /26
Stressor Task: Math (II)
16
Progress bar +
tic-tac sound
Random
decrease
Digits back to 0
every time
Each
wrong answer annoying
sound and going back

17. PervasiveHealth 2015, Istanbul, Turkey /26
Search Task - Stressed
17
Progress bar +
tic-tac sound
Sound +
vibration

18. PervasiveHealth 2015, Istanbul, Turkey /26
Write Task - Stressed
18
Progress bar +
tic-tac sound

19. PervasiveHealth 2015, Istanbul, Turkey /26
User Study
• 13 Participants (7M, 6F), average age 26,38 (± 2,53)
• Own phone, no constraints for the type to use (Android
OS)
• Different English literacy level
• Average protocol duration: 1 hour
• Cover story: New Google interface analysis
19

20. PervasiveHealth 2015, Istanbul, Turkey /26
Stress Induction Analysis
20
Initial
Relax
Relaxed
Tasks
Stressor Stressed
Tasks
ESM 1 ESM 2 ESM 3 ESM 4 ESM 5
TEST t(13) p-value
ESM 3 VS ESM 4 1.99 0,007 *
ESM 3 VS ESM 5 -2.84 0,009 *
ESM 4 VS ESM 5 -2.74 0.5
Participants were stressed
Different stress level at
the end of tasks
Kept stressed during
stress tasks

21. PervasiveHealth 2015, Istanbul, Turkey /26
Features Evaluation
• Statistical analysis for significance evaluation
• Stress prediction model using Decision Tree (DT), k-
Nearest Neighbourhood (kNN), Bayes Network (BN),
Support Vector Machine (SVM) and Neural Networks
(NN)
• User and global model (evaluated using 10-Fold cross
validation and leave-one-out)
21

22. PervasiveHealth 2015, Istanbul, Turkey /26
Search Task - Statistical Correlation
• Only weak correlation between our features
• Global Model
• Average swipe pressure (p-value = 0,09)
• Scroll distance from center (p-value = 0,065)
• Scroll distance from top left (p-value = 0,07)
22
• User model
• Scroll interaction length (strong correlation for 61% of users)
• Scroll delta (strong correlation for 40% of users)
• Scroll linearity (strong correlation for 45% of users)

23. PervasiveHealth 2015, Istanbul, Turkey /26
Search Task - Prediction Model
23
F-measure for Scroll interaction models
MODEL DT KNN SVM NN BN
USER
(AVERAGE)
0.79 0.80 0.81 0.80 0.77
GLOBAL
(AVERAGE)
0.73 0.71 0.78 0.74 0.67
F-measure for Swipe interaction models
MODEL DT KNN SVM NN BN
USER
(AVERAGE)
0.86 0.86 0.79 0.87 0.85
GLOBAL
(AVERAGE)
0.92 0.75 0.81 0.82 0.77

24. PervasiveHealth 2015, Istanbul, Turkey /26
Write Task - Statistical Correlation
• User Model
• Digits size (64% of users with strong correlation)
• Pressure/Size ratio (55% of users with strong correlation)
• Global Model
• Wrong Words / Total words ratio (p-value = 0,028)
• Digits time distance (p-value = 0,012)
• Digit duration (p-value = 0,08)
24

25. PervasiveHealth 2015, Istanbul, Turkey /26
Conclusions
• Stress assessment using data from non-intrusive
devices can increase people’ acceptance
• Human-smartphone interaction analysis can be
leveraged to assess stress state in users
• Scroll and Swipe: F-measure of stress prediction
between 79% and 85% for user models, and between
70% and 80% for global model.
• Text writing: several features showed strong
correlation
25

26. PervasiveHealth 2015, Istanbul, Turkey /26
Future works
• Real-time background service for stress assessment
• Behaviour suggestion implementation
• Stress assessment in the wild (ongoing study, 29
participants)
26
• My PhD Thesis :)

27. iSenseStress: Assessing Stress
Through Human-Smartphone
Interaction Analysis
Matteo Ciman, Katarzyna Wac and Ombretta Gaggi
{mciman,gaggi}@math.unipd.it
katarzyna.wac@unige.ch
wac@di.ku.dk