Nowadays it is widely accepted that games, and entertainment technologies in general, have very interesting features that, if used properly, can largely contribute to the effectiveness of treatments in different health domains. These games, also known as games-with-a-purpose, need to achieve a very difficult and interesting balance among science, health, engineering and entertainment. In this talk I will present the approach we follow at the NeuroRehabLab, where we combine games, Human Computer Interaction and clinical rehabilitation guidelines to develop interactive systems that are novel and effective tools for motor and cognitive rehabilitation, with special emphasis on stroke. I will discuss the effect of interface technology in motor-cognitive interference in task performance; a participatory design approach with health professionals to develop parameterized models for the training of Activities of Daily Living in a simulated environment; and how we automate the parameter selection process in these games by means of an adaptive approach. This strategy allows these systems to be used by patients of different cognitive and motor skills while still providing a personalized training.

1. Sergi Bermúdez i Badia
Inv. Assist. Prof., Universidade da Madeira
Marie Curie Research Fellow, Madeira – ITI
sergi.bermudez@uma.pt

2. 2050
*United Nations Population Fund 2012
• The world population is ageing dramatically
• In Europe ~20% of the population is over 60; it will be 30+% by 2050.2

3. • One of the main causes of permanent disability in older adults that consumes 2-4% of total healthcare costs worldwide.1
• The number of strokes will increase during the next 40 years.2
• It will be difficult to manage the sharp rise of social and healthcare costs.
1Donnan et al., Lancet 2008; 2Howard & Goff, Ann. N.Y. Acad. Sci 2013

4. Peripheral manipulation of the skeletal-motor system is expensive, boring, treats the sequels but not the source of the problem…
Dobkin, Nat Clin Pract Neurol 2008

5. 1.Treatment frequency and intensity correlate with recovery (Kwakkel et al., 2004; Sonoda, Saitoh, Nagai, Kawakita, & Kanada, 2004).  Low cost technology for extended deployment
2. Movement practice and repetition play a fundamental role in recovery (Karni et al., 1995).  Building requirements into game mechanics
3. Specificity of rehabilitation training with respect to the deficits and required functional outcomes has an impact on recovery (Krakauer 2006).  Accessible interface technology and task personalization

6. 6

7. VE can provide:
- Fully controlled environments
- Minimally supervised intensive training
- Task-specific movement reiteration
- Individualized training
- Feedback for reward and motivation

8. 3
1
6
2
5
4
1- Virtual Reality technology:
- 3D virtual environments
- Head Mounted Displays (Oculus Rift, Vuzix) 2- Patient monitoring: - Eye tracking (EyeTribe, Tobii)
- Autonomic System responses (Bitalino)
- Face tracking (Face API) 3- Accessible Rehabilitation:
- Movement kinematics (kinect, wii)
- BCI-EEG (g.MobiLab, OpenVibe, BCI2000, EMOTIV) 4- EMG driven Neuro-Robotic orthosis (mPower1000) 5- Augmented Reality (AR) and custom tracking through AnTS 6- We provide it as open and freely available software for research at http://neurorehabilitation.m-iti.org

9. 9

10. Training paradigm:
- Goal oriented and repetitive actions
- Bimanual training (non-paretic arm support)
- Parameterized (flying speed, turning speed, acceptance radius, distance between objects) Motivation:
- Embedded in a game
- Extensive visual and sound feedback
- Automatic computation of training parameters (Avoid failure and frustration) Quantify performance

11. 11
- Optimal performance at intermediate stress levels
- We use the Yerkes & Dodson law to optimize time to the average performance of the user
* Yerkes, R.M. and J.D. Dodson, 1908
* Yerkes, R.M. and J.D. Dodson, 1908

12. 12
- Flow is “a state of peak enjoyment, energetic focus, and creative concentration by people engaged in adult play”
- Balance user skills with challenge using our psychometric model
* Csikszentmihalyi, M. 1975
* Csikszentmihalyi, M. 1975

13. A study with 10 healthy participants has shown that the NTT captures precise quantitative kinematic information during a NTT training session, including:
• Range of Movement (ROM)
• Movement smoothness
• Arm coordination
• Arm contribution to task
Bermúdez i Badia, Stroke Research & Treat, 2012

14. s
t
a
d
s*t
s*a
s*d
t*a
t*d
a*d
s2
t2
a2
d2
Movement Smoothness
Range of Motion
Arm Displacement
Arm coordination
Kinematic measure = c0 + c1*speed + c2*turning + c3*acceptance + c4*distance
+ c5*speed*turning + c6*speed*acceptance + c7*speed*distance
+ c8*turning*acceptance + c9*turning*distance
+ c10*distance*acceptance + c11*speed2 + c12*turning2 + c13*acceptance2
+ c14*distance2
- Not all parameters contribute to all movement kinematic measures
- We have a quantitative way of adapting parameters depending on a higher level desired kinematic training

15. •Cognitive domain is not always considered.
•ADL’s are mostly cognitive and motor (dual-task).
RGS (Cameirão et al., 2011)
NTT
(Bermúdez i Badia & Cameirão, 2012)
What do we know about cognitive demands of these tasks?

16. 16

17. 17
• Cognitive rehabilitation is mostly
performed with “paper and
pencil tools”.
• Based on “tradition”, not
necessarily scientific evidence.
• Traditional tasks lack ecological
validity.
• There is no framework for how
to design and select tasks for
cognitive rehabilitation.

18. 18
* Taxonomy Revised (Anderson et al., 2001)
* Toulouse – Piéron task (Toulouse et al., 2004)

19. 19

20. 20

21. Creating ecologically valid Activities of Daily Living (ADLs)
•Attention  simulated supermarket, post office, bank, pharmacy
•Executive functions  elementary instructions, problem solving tasks
•Visuospatial orientation  navigation through simulated city
Vourvopoulos et al. International Conference on Advances in Computer Entertainment Technology (ACE’14).

22. 22

23. Inclusion criteria:
•Absence of hemi-spatial neglect;
•Sufficient cognitive capacity (MMSE) ≥ 15 (Folstein et al, 1975 English version of Warrior et al., 1994);
•Ability to be seated;
•Education ≥ 4th class or read and write;
•Motivation to participate in the study.
Participants:
•18 (9 experimental + 9 control), with ages between 34 – 90 years old
•10 female and 8 male
•Time after stroke: 3 months to 14 years
•Localization: 9 left hemisphere + 9 right hemisphere
•15 participants had no experience with computers
•with prof. Luísa Soares (UMa) Dra. Manuela Barros (SESARAM) Dr. Rafael Freitas (SESARAM) Dra. Teresa Gois (SESARAM)
23

24. 12 sessions of 20 min of cognitive
training (~ 1 month) on top of
conventional therapy with pre
and post assessment.
• Experimental group: VR based
simulation of ADL
• Control group: Cognitive
stimulation using traditional
tools (puzzles, paper and pencil
tasks, etc)
24

25. • Addenbrooke's cognitive examination - ACE-R - assesses: attention, memory, verbal fluency, language and visual- spatial orientation.
• Trail making test A and B - evaluates attention, sequencing of stimuli, visual search, information processing, eye-hand coordination, etc.
• Layout of pictures - ability to organize and sequence a story.
• Stroke Impact Scale - SIS – subjective evaluation of strength, hand function, memory, emotion, mobility, ADL’s, communication and social participation.
25

26. 26
* Bold indicates p < 0.05 matched-pairs Wilcoxon test
Experimental Group
Control Group
Pre
Post
Pre
Post
ACE-R - Total
Median
72
81 (+9)
66
69
IQR
15
19
27
20
MMSE
Median
23
29 (+6)
23
26
IQR
6
4
6
6
(ACE-R) Attention & Orientation
Median
15
18 (+3)
14
16
IQR
3
2
5
5
(ACE-R) Memory
Median
15
18 (+3)
18
18
IQR
5
7
9
9
(ACE-R) Verbal Fluency
Median
5
6
6
5 (-1)
IQR
4
4
4
3
(ACE-R) Language
Median
22
24 (+2)
19
21
IQR
2
5
6
8
(ACE-R) Visuo-spatial
Median
12
14
12
14
IQR
7
2
6
9
Layout of pictures
Median
2
4 (+2)
2
2
IQR
2
5
3
3
* Bold indicates p < 0.05 matched-pairs Wilcoxon test
* Bold indicates p < 0.05 matched-pairs Wilcoxon test

27. 27
* Bold indicates p < 0.05 matched-pairs Wilcoxon test
* Bold indicates p < 0.05 matched-pairs Wilcoxon test
* Bold indicates p < 0.05 matched- pairs Wilcoxon test
Experimental Group
Control Group
Pre
Post
Pre
Post
Memory
Median
62,5
71,88 (+9,38)
56,25
62,5 (+6.25)
IQR
37,5
33,48
37,52
32,82
Emotional Estability
Median
75
83,33 (+8.33)
58,33
66,67
IQR
29,15
12,44
27,78
27,78
Comunication
Median
75
85,71
67,86
67,86
IQR
30,35
32,15
37,51
39,29
ADLs
Median
50
56,25
43,75
45,337
IQR
42,71
37,5
38,54
33,35
Social Participation
Median
63,89
66,67 (+2.78)
36,11
50 (+13.18)
IQR
43,17
29,8
22,21
16,66
Total Recovery
Median
50
70 (+20)
40
60
IQR
15
25
15
30

28. •A close dialog between health practitioners – neuroscientists – technologists is necessary
•Our systems are not the end product, are the hypotheses
•Hypotheses need validation  impact assessment
•Game / training mechanics need to include:
•Game parameters need to be automatically personalized
•Challenge vs. skill needs to be quantified and well understood
•Stress levels need to be controlled to ensure maximal performance and consequent maximal learning
•Consider both motor and cognitive aspects
•Relation with Activities of Daily Living

29. 29
Sergi Bermúdez i Badia (Assist. Prof)
Mónica S. Cameirão (Assist. Prof)
Ana Lúcia dos Santos Faria (PhD student)
Athanasios Vourvopoulos (PhD student)
John Edison Muñoz (PhD student)
Teresa Paulino (Research assistant)
Afonso Gonçalves (PhD student)
Andreia Andrade, MSc student Psychology (join thesis with Luísa Soares) Júlio Alves, MSc student Informatics (join thesis with VisLab, IST-Lisbon) Davide Neves, MSc student Informatics Miguel Sousa, MSc student Telecommunications and Networks (join thesis with Luís Gomes) André Ferreira, MSc student Informatics Rúben Jardim, MSc student Informatics

30. University of Pittsburgh
Department of Occupational Therapy
Quality of Life Technologies Center http://www.cmu.edu/qolt
Carnegie Mellon University
Daniel P. Siewiorek Asim Smailagic
Scott Bleakley
Myomo Inc http://www.myomo.com
Steve Kelly Ela Lewis
SESARAM - www.sesaram.pt Serviço de Saúde da RAM
Dra. Manuela Barros Dr. Rafael Freitas Dra. Teresa Gois Dr. Jean-Claude Fernandes Dr. Gil Bebiano Dr. Rafael Macedo
30

31. For more information contact:
sergi.bermudez@uma.pt
or visit
http://neurorehabilitation.m-iti.org