1.
Jan Wohlfahrt-Laymann, Hermie Hermens, Claudia Villalonga,
Miriam Vollenbroek-Hutten, Oresti Banos
EmotionAware (PerCom2018)
March 23, 2018
o.banoslegran@utwente.nl
@orestibanos
http://orestibanos.com/
23-Mar-18 1
Enabling remote assessment of
cognitive behaviour through
mobile experience sampling

2.
@orestibanos
http://orestibanos.com/
Context
 Circa 15% of adults over 60 suffer from a
cognitive disorder [who16] and 47.5 million people
are affected by dementia, with 7.7 million new
cases every year [who17]
2
[who16] World Health Organization (2016) Mental health and older adults.
http://www.who.int/mediacentre/factsheets/fs381/en/
[who17] World Health Organization (2017) Dementia. http://www.who.int/ mediacentre/factsheets/fs362/en/

3.
@orestibanos
http://orestibanos.com/
Measuring cognition
 “The guiding idea of cognitive science is that
mental operations involve processing information,
and hence that we can study how the mind works
by studying how information is processed” [ber14]
3
[ber14] Bermudez JL (2014) Cognitive Science: An Introduction to the Science of the Mind. Cambridge University Press.
Perception, Attention, Memory, Language skills
Visuospatial processing, Executive functions

4.
@orestibanos
http://orestibanos.com/
Measuring cognition: (traditional) clinical tests
4
Addenbrooke's
Cognitive
Examination
Montreal
Cognitive
Assessment
Saint Louis University
Mental Status Exam
Mini Mental State
Examination

5.
@orestibanos
http://orestibanos.com/
Measuring cognition: challenges and opportunities
5
� Tests administered quite occasionally
(normally during intake meetings and/or
after first symptoms are detected)
� Person needed to introduce the test and
collect the answers
� Presence of expert may bias the
responses of the patient
Addenbrooke's
Cognitive
Examination
Montreal
Cognitive
Assessment
Saint Louis University
Mental Status Exam
Mini Mental State
Examination
Cognitive functioning is
shown in our daily
behavior?!

6.
@orestibanos
http://orestibanos.com/
MobileCogniTracker: mobile ESM + passive sensing
6
(Cognitive)
Experience
Sampling
Methods
(Cognitive)
Passive
Sensing

7.
@orestibanos
http://orestibanos.com/
MobileCogniTracker: system overview
7
Platform
Sensing
Infrastructure
Accelerometer
Applications
Barometer
Bluetooth
Communication
ESM
Light
Gyroscope Magnetometer
Locations
Telephony Scheduler Temperature
Screen Text-2-Speech
Proximity Network Apps
+ PLUGINS/
PROBES!
Battery
AWARE
http://www.awareframework.com/
(Cognitive) Experience
Sampling Methods
(Cognitive) Passive Sensing
+

8.
@orestibanos
http://orestibanos.com/
MobileCogniTracker: system overview
8

9.
@orestibanos
http://orestibanos.com/
Cognitive ESMs (Examples)
9
Orientation Language

10.
@orestibanos
http://orestibanos.com/
Cognitive ESMs (Examples)
10
Language
Repetition
Complex tasks

11.
@orestibanos
http://orestibanos.com/
Evaluation
 Experimental Setup
 13 participants (65+) – no
cognitive impairment reported
 1 domain expert (psychiatrist)
 Google Pixel, Samsung Galaxy
S7, LG G5, Huawei P9
(arbitrarily provided)
 Realisation of MMSE
 Methods
 System usability scale (SUS)
 Semi-structured interview
11

12.
@orestibanos
http://orestibanos.com/
Evaluation
 Results (SUS)
12
1-I think that I would like to use this system frequently.
2-I found the system unnecessarily complex.
3-I thought the system was easy to use.
4-I think that I would need the support of a technical person to be
able to use this system.
5-I found the various functions in this system were well integrated.
6-I thought there was too much inconsistency in this system.
7-I would imagine that most people would learn to use this system
very quickly.
mean > 68 (good usability) [bro96]
[bro96] Brooke J (1996) SUS-a quick and dirty
usability scale. Usability Evaluation in Industry
189(194):4–7
8-I found the system very cumbersome to use.
9-I felt very confident using the system.
10-I needed to learn a lot of things before I
could get going with this system.

13.
@orestibanos
http://orestibanos.com/
Evaluation
 Results (semi-structured interview)
 Orientation, registration and recall tasks such as
questions on the participant’s current location or
date “do not make a big difference with respect to
the paper version, and do not pose in principle
additional cognitive load to the user.”
 Attention and calculation tasks are considered
“quite straightforward”
 Concerns regarding the feasibility of the language
tasks: “I have some concerns regarding the effect
that the keyboard autocorrect function can play in
the naming of objects. This function appears to
suggest different options that can influence the
decision of the patient.”
13
[bro96] Brooke J (1996) Sus-a quick and dirty usability
scale. Usability Evaluation in Industry 189(194):4–7

14.
@orestibanos
http://orestibanos.com/
Evaluation
 Results (semi-structured interview)
 Text-to-speech functionality was seen as a good
feature to facilitate the comprehension of the action
to be carried out. “Sometimes I have to read the
questions for my patients [...], thus having them on
audio seems to me very relevant. I cannot judge
though how patients will feel about a device telling
them to do things.”
 Speech-to-text capability was highly valued by the
expert: “I was somewhat intrigued with how the tool
could register the voicing of a given sentence. The
speech recognition functionality works surprisingly
good”.
14
[bro96] Brooke J (1996) Sus-a quick and dirty usability
scale. Usability Evaluation in Industry 189(194):4–7

15.
@orestibanos
http://orestibanos.com/
Contributions
 MobileCogniTracker: a digital cognitive
experience sampling tool
 An integration of MobileCogniTracker with
passive multimodal mobile sensing (AWARE)
 A realisation of MobileCogniTracker based on
the full implementation of the MMSE
 A (first) study of the usability and reliability of
MobileCogniTracker
15

16.
@orestibanos
http://orestibanos.com/
Limitations and Future work
 Users experienced some difficulties with drawing
and copying tasks (presumably worse for small
smartphones)
 Some built-in functions of smartphones could
influence the answers of the users (e.g., auto-
complete)
 Learning effect vs. fully randomized tests
 Involving cognitively impaired people
16

17.
@orestibanos
http://orestibanos.com/
Many thanks!
Questions?
17
This work has received funding from the European Union’s Horizon 2020
research and innovation programme under Grant Agreement #769553.
This result only reflects the author’s view and the EU is not responsible
for any use that may be made of the information it contains.