9th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW), 2021 Several obstacles have to be overcome in order to recognize emotions and affect in daily life. One of them is collecting a large amount of emotionally annotated data necessary to create data-greedy machine learning-based predictive models. Hence, we propose the Emognition system supporting the collection of rich emotional samples in everyday-life scenarios. The system utilizes smart-wearables to record physiological signals unobtrusively and smartphones to gather self-assessments. We have performed a two-week pilot study with 15 participants and devices available on the market to validate the system. The outcomes of the study, alongside the discussion and lessons learned, are provided.

1. 2021 9th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW)
A system for collecting emotionally annotated
physiological signals in daily life using wearables
Stanisław Saganowski*
, Maciej Behnke, Joanna Komoszyńska, Dominika Kunc, Bartosz Perz,
Przemysław Kazienko
Department of Computational Intelligence, Wrocław University of Science and Technology, Wrocław, Poland
*
stanislaw.saganowski@pwr.edu.pl
Abstract—Several obstacles have to be overcome in order to
recognize emotions and affect in daily life. One of them is
collecting a large amount of emotionally annotated data necessary
to create data-greedy machine learning-based predictive models.
Hence, we propose the Emognition system supporting the col-
lection of rich emotional samples in everyday-life scenarios. The
system utilizes smart-wearables to record physiological signals
unobtrusively and smartphones to gather self-assessments. We
have performed a two-week pilot study with 15 participants
and devices available on the market to validate the system.
The outcomes of the study, alongside the discussion and lessons
learned, are provided.
Index Terms—emotion recognition, Emognition, field studies,
self-assessment triggering, rich emotion labelling, data collection
I. INTRODUCTION
The fields of emotion recognition and emotions’ psy-
chophysiology are dominated by laboratory studies in which
emotions are elicited with standardized stimuli. To avoid bias
caused by the artificial stimuli and controlled environment, a
necessary step forward is to measure emotions in a real-life
environment. However, only a few emotion recognition studies
have been performed in the field so far [1], e.g., [2], [3].
Such studies are recently possible due to the development
of wearables and smartphones, which are now able to collect
and process physiological data. The only remaining question is
how to find the real-life moments in which individuals experi-
ence the emotions? The SOTA methods to assess participants’
momentary emotions in their daily living environments are
the experience sampling method (ESM) [4] and the ecolog-
ical momentary assessment (EMA) [5]. They provide high
ecological validity of the repeated in-the-moment experience
measurement in which participants receive the measurements’
notifications in a semi-random design. However, ESM and
EMA can be further improved with the recent developments
in affective computing, in which the measurement moments
can be detected by a pre-trained model [6] or tailored to
participants’ daily routines (e.g., events in the calendar).
We propose and validate the Emognition system for in-
the-field studies. It utilizes the scheduled quasi-random as-
This work was partially supported by National Science Centre, Poland,
project no. 2020/37/B/ST6/03806; by the statutory funds of the Department of
Computational Intelligence, Wroclaw University of Science and Technology;
by the European Regional Development Fund as a part of the 2014-2020 Smart
Growth Operational Programme, CLARIN - Common Language Resources
and Technology Infrastructure, project no. POIR.04.02.00-00C002/19.
sessments, machine learning-based triggers, and self-triggered
assessments. The system allows to perform in-the-field studies,
thus mitigating recall bias and the laboratory environment bias,
emphasizing a more natural context of the emotions along with
a wider spectrum of the emotions.
II. SYSTEM
The Emognition system comprises wearables, smartphones,
and a back-end server. The wearables such as smartwatch
Samsung Galaxy Watch 3 or chest strap Polar H10 record
physiological data (among others, BVP/ECG, TEMP, ACC)
and transfer the data to the connected smartphone. Any
smartwatch available on the market can be utilized as long as it
provides the raw physiological signals [7]. The smartphone is
mainly responsible for collecting self-assessments. The custom
application analyzes the physiological data in real-time and
triggers the self-assessment when emotion is detected. The
participant is informed about the self-assessment through the
standard system notification. From time to time, the sched-
uled self-assessment is also triggered. The rules of triggering
self-assessments (what triggers them, how often, how many)
can be adjusted and configured on the back-end server. The
smartphone sends the physiological data and self-assessments
to the back-end server when the WiFi connection is available.
The affect is measured in valence and arousal dimensions,
in line with a circumplex model [8]. Participants are asked,
”How much [emotion] are you experiencing at this moment?”
on a scale ranging from 0 (not at all) to 100 (extremely). We
use validated items [9] to measure positive high arousal affect
(PHA): excited, joyful, proud, amused; positive low arousal
affect (PLA): peaceful, relaxed, satisfied, calm; negative high
arousal affect (NHA): nervous, anxious, angry, stressed; and
negative low arousal affect (NLA): sad, bored, depressed,
powerless.
A video demonstrating the Emognition system is available
at youtu.be/F0cMw2rtYvE. See Fig. 1a for system screenshots.
III. PILOT STUDY
The study was approved by and performed in accordance
with guidelines and regulations of the Wroclaw Medical
University, Poland; approval no. 149/2020. We recruited 15
participants (7 female) between the ages of 20 and 28 years
978-1-6654-0021-3/21/$31.00 ©2021 IEEE
2021
9th
International
Conference
on
Affective
Computing
and
Intelligent
Interaction
Workshops
and
Demos
(ACIIW)
|
978-1-6654-0021-3/21/$31.00
©2021
IEEE
|
DOI:
10.1109/ACIIW52867.2021.9666272
Authorized licensed use limited to: Politechnika Wroclawska - CWINT. Downloaded on February 10,2022 at 17:47:48 UTC from IEEE Xplore. Restrictions apply.

2. TABLE I: The number of assessments in control groups.
Control
group
Scheduled
per day
Scheduled
filled out
Filled out /
all scheduled
Self-triggered
filled out
A 3 53 59% 26
B 6 61 35% 31
C 9 26 20% 39
(M=24.87, SD = 2.31). All participants provided written in-
formed consent and received a 50 PLN (c.a., $15) voucher for
an online store.
Participants visited the lab on day 1 of the study. They
received smartwatches, installed the Emognition application on
their phones, and were instructed in the use of the smartwatch,
the smartphone application, and in emotion reporting. The
assessments consisted of brief questionnaires using ESM at
quasi-random times and self-initiated reports during 14 con-
secutive days. Participants were randomized into three groups
that received 3, 6 or 9 scheduled questionnaires per day.
This resulted in 42, 84, or 126 reports for each participant.
Participants were asked to report on their emotions whenever
the self-assessment was available or on demand. After 14 days,
participants visited the lab for the second time, returned the
smartwatches, and provided feedback.
Reliability for all subscales was satisfactory, NHA Cron-
bach’s α = .79, NLA α = .72, PHA α = .79, and PLA α = .80.
Moreover, the items within each quadrant (PHA, PLA, NHA,
NLA), as well as within positive vs. negative dimensions were
correlated, see Fig. 1b.
IV. DISCUSSION
The study has been performed between June 7-21, 2021;
therefore, only a brief analysis of the outcomes is presented.
First and foremost, the system collecting emotional data in
everyday life scenarios is feasible. We were able to utilize
wearables and smartphones available on the market. However,
with some of them, we encountered technical problems. Sev-
eral Xiaomi smartphones could not connect with the Samsung
smartwatch, or the connection was unstable. As a result,
the physiological data were not obtained. Furthermore, the
smartphone was blocking the self-assessment notification in
some cases, so it was not displayed to the participant, and
the questionnaire was not filled out. We have removed seven
participants with such problems from the analysis. A possible
solution to these problems would be to provide participants
with a specific smartphone model tested beforehand and found
to be working with Samsung smartwatch correctly. This would,
however, increase the study cost. Another technical problem
to consider is battery draining. This concerns both smartwatch
and smartphone batteries. A potential solution is reducing
sampling, which would, however, result in lower data quality.
We found that participants fill out fewer questionnaires as
the study progresses, Fig. 1c, and at the same time, they are
able to fill out questionnaires more quickly, Fig. 1d. Moreover,
participants who had only three scheduled questionnaires per
day (control group A) filled out almost 60% of the ques-
tionnaires, while participants who had six and nine scheduled
(a) Emognition applications for smartwatch and smart-
phone.
(b) Correlation within emotional subscales.
(c) Percentage of assessments
filled out per day.
(d) Average time of filling out a
single assessment.
Fig. 1: (a) The Emognition system; (b) the correlation within
emotional subscales: PHA, PLA, NHA, NLA; (c) and (d)
the analysis of participants’ behaviour (interaction with the
application) throughout the study, moving average over 3 days
was applied.
assessments filled out only 35% and 20% of all questionnaires,
respectively (Tab. I). This suggests that participants are capable
of filling out only three to six scheduled assessments per day.
Further improvements in the system include integration with
other wearables – chest-strap Polar H10, which offers ECG
signal, and with Wear OS-based smartwatches that comprise a
large share of the market. Additionally, enhancements of the
pre-trained model triggering assessments will be performed.
Authorized licensed use limited to: Politechnika Wroclawska - CWINT. Downloaded on February 10,2022 at 17:47:48 UTC from IEEE Xplore. Restrictions apply.

3. REFERENCES
[1] S. Saganowski, A. Dutkowiak, A. Dziadek, M. Dzieżyc, J. Komoszyńska,
W. Michalska, A. Polak, M. Ujma, and P. Kazienko, “Emotion recognition
using wearables: A systematic literature review - work-in-progress,”
in 2020 IEEE International Conference on Pervasive Computing and
Communications Workshops (PerCom Workshops - EmotionAware), 2020,
pp. 1–6.
[2] N. T. Nguyen, N. V. Nguyen, M. H. T. Tran, and B. T. Nguyen, “A poten-
tial approach for emotion prediction using heart rate signals,” in 2017 9th
International Conference on Knowledge and Systems Engineering (KSE).
IEEE, 2017, pp. 221–226.
[3] P. Schmidt, R. Dürichen, A. Reiss, K. Van Laerhoven, and T. Plötz,
“Multi-target affect detection in the wild: An exploratory study,”
in Proceedings of the 23rd International Symposium on Wearable
Computers, ser. ISWC ’19. New York, NY, USA: Association
for Computing Machinery, 2019, p. 211–219. [Online]. Available:
https://doi.org/10.1145/3341163.3347741
[4] M. Csikszentmihalyi and R. Larson, “Validity and reliability of the
experience-sampling method,” in Flow and the foundations of positive
psychology. Springer, 2014, pp. 35–54.
[5] S. Shiffman, A. A. Stone, and M. R. Hufford, “Ecological momentary
assessment,” Annu. Rev. Clin. Psychol., vol. 4, pp. 1–32, 2008.
[6] M. Dzieżyc, J. Komoszyńska, S. Saganowski, M. Boruch, J. Dziwiński,
K. Jabłońska, D. Kunc, and P. Kazienko, “How to catch them all? en-
hanced data collection for emotion recognition in the field,” in 2021 IEEE
International Conference on Pervasive Computing and Communications
Workshops and other Affiliated Events (PerCom Workshops). IEEE, 2021,
pp. 348–351.
[7] S. Saganowski, P. Kazienko, M. Dzieżyc, P. Jakimów, J. Komoszyńska,
W. Michalska, A. Dutkowiak, A. Polak, A. Dziadek, and M. Ujma,
“Consumer wearables and affective computing for wellbeing support,”
in Proceedings of the 17th EAI International Conference on Mobile
and Ubiquitous Systems: Computing, Networking and Services, 2020.
[Online]. Available: https://arxiv.org/abs/2005.00093
[8] L. Feldman Barrett and J. A. Russell, “Independence and bipolarity in the
structure of current affect.” Journal of personality and social psychology,
vol. 74, no. 4, p. 967, 1998.
[9] H.-J. De Vuyst, E. Dejonckheere, K. Van der Gucht, and P. Kuppens,
“Does repeatedly reporting positive or negative emotions in daily life
have an impact on the level of emotional experiences and depressive
symptoms over time?” Plos one, vol. 14, no. 6, p. e0219121, 2019.
Authorized licensed use limited to: Politechnika Wroclawska - CWINT. Downloaded on February 10,2022 at 17:47:48 UTC from IEEE Xplore. Restrictions apply.