Narratives and Sensor Driven Cognitive Behavior Training Game Platform

1. Narratives and Sensor Driven Cognitive
Behavior Training Game Platform

2. • Research Objective
• ADHD Facts
• ADHD Treatment Strategy
• Serious Game and its Effects
• Narrative Therapy
• Behavior Training Game Platform
• Experiments and Analysis
• Demo
• Review and Future Research
2
Contents

3. • Behavior Training Game Platform
• Brain-Computer Interface (BCI) with NeuroSky’s MindWave
• Skeleton and face tracking with Microsoft’s Kinect V2
• Narrative fairy tale contents with characters and contextual
instructions
• This Platform:
• Keeps track of user’s brainwaves and associated behaviors
responding to the given narrative contents or implicit contextual
instructions
• Works as a treatment framework for children with ADHD
• to improve the children’s sustained attention
• to control their hyperactive behavior
• to promote reading comprehension capability
• to give them an opportunity of self-examination
• to get them treated emotionally
• to encourage them to feel self-confidence
3
Research Objective

4. • ADHD (Attention-Deficit/Hyperactivity Disorder)
• 5% to 15% children in the U.S. have been diagnosed
• American Psychiatric Association (APA): Avg. 5%
• Centers for Disease Control and Prevention (CDC) : Avg. 11%
• Typical Symptoms include:
• Deficits in behavioral inhibition, sustained attention, and
resistance to distraction
• Impaired behavioral inhibition and self-control needed as a
critical foundation for the performance of any task
• Weak social skills
• Low level reading capabilities
4
ADHD Facts

5. 5
http://www.healthline.com/health/adhd/facts-statistics-infographic
ADHD by the Numbers

6. • ADD: Attention Deficit Disorder
• HD: Hyperactivity Disorder
• Mixed ADHD: Attention Deficit/Hyperactivity Disorder
6
Rief, S.F. 2005. How to reach and teach children with ADD/ADHD: practical techniques, strategies
and interventions, 2nd ed. San Francisco: Jossey-bass.
Types of ADHD

7. 7
https://www.pinterest.com/thebluebunyip/adhd/
Attention Deficit Disorder (ADD)

8. 8
http://www.theatlantic.com/health/archive/2012/02/a-western-diet-high-in-sugars-and-fat-could-
contribute-to-adhd/252853/
Hyperactivity Disorder (HD)

9. • Children might have co-existing conditions
• learning disabilities
• conduct disorders and difficulties: antisocial behavior, fighting,
and oppositional defiant disorder
• anxiety disorder
• depression
• sleep disorders, etc.
• Boys and girls display very different ADHD symptoms.
• Boys: more likely the attention disorder
• Girls: more internal and less obvious
9
ADHD & Co-Existing Conditions

10. • Develop relatively long-term sustained attention to successfully
complete the instruction (ADD)
• Keep high-level attention to figure out complicate and boring story
or discussion (ADD/HD)
• Encourage children to stay motionless for a while and behave
appropriately in a certain situation (HD)
10
ADHD Behavioral Treatment Strategy

11. 11
Game Intro

12. 12
Go Fishing

13. • Any form of interactive computer-based game software for one or
multiple players to be used on any platform and that has been developed
with the intention to be more than entertainment
• Primary purpose beyond pure entertainment
• To train or educate users
• To achieve a desired progress
13
Serious Games: Mechanisms and Effects, by Ute Ritterfield, Michael Cody, and Peter Vorderer, 2009
Serious Games

14. • Called also,
• immersive learning simulations
• digital game-based learning
• gaming simulations
• Players come to serious games with that expectation that they can impr
ove some specific aspect of learning
• Simply refer to any kind of online game for learning
14
Serious games: online games for learning, by Anne Derryberry, I’m Serious.net
Serious Games

15. • Indirect Narrative Experiences
• A fairy tale story with pain and frustration
• Narrative Experiences 
Catharsis, Empathy 
Change in perspective, Conscious of a kind, Self-examination,
Generalization 
Relief, Problem-solving, Therapy
• Converting the existing cognitive frame causing behavioral
disorder into a new frame
15
Narrative Therapy Mechanism

16. • Reading narratives requires complete understanding of the
components of a story and its development processing.
• Children with ADHD can be effectively treated with well-known,
structured narratives (fairy tales) to improve reading skill,
control hyperactive behavior, and treat emotional problem.
16
Designing Behavioral Treatment with Narratives

17. 17
Unity3D
Game Contents Manager
Contents
Kinect V2
MindWave
Logger Replayer
Screen Capture
MPEG Processor
Middleware
Event Logger Video Producer
Server
ZooKeeper
Kafka
Elasticsearch DB
Video File Server
Video File Storage
API
Experiments Dashboard
Users Experiments Analysis
Behavior Training Platform

18. • Monitoring Concentration with Brain-Computer Interface (BCI)
• Monitoring Behavior with Motion Sensors
• Interactive Narratives and Intervention Data
• User’s Responses
18
Interactive Data Collection

19. • eSense meter of Brainwaves
• Attention
• Meditation
• Saved to the list of Brainwaves
• After finishing game, the data sent to Data Sever
19
Brainwaves
MindWave
Interface
Collect and
Save
Brainwave
s
ElasticSearch Collect
Event Logs
Send to
Data Server
Brainwaves via MindWave

20. 20
00_Hip_center 10_Wrist_Right
01_Spine 11_Hand_Right
02_Shoulder_Center 12_Hip_Left
03_Head 13_Knee_Left
04_Shoulder_Left 14_Ankle_Left
05_Elbow_Left 15_Foot_Left
06_Wrist_Left 16_Hip_Right
07_Hand_Left 17_Knee_Right
08_Shoulder_Right 18_Ankle_Right
09_Elbow_Right 19_Foot_Right
Joints Twenty joints
Kinect Skeleton Structure

21. • Skeleton Tracking
• Face Tracking
21
Skeletal tracking
with Kinect:
recognizing 22
different
motions
Face tracking
with Kinect:
recognizing 6
different
motions
Motion and Face Tracking

22. 22
Kinect: Motion Recognition

23. 23
The fisherman goes fishing. The fisherman falls down due to the
player’s motion
Behavior Training Scheme using
Adaptive Intervention Approach

24. 24
Reading comprehension text with
BCI attention requirement
BCI attention requirement with the
quest of counting by 10
Behavior Training Scheme using
Adaptive Intervention Approach

25. • Three Characters
• Fisherman: the main character
• Genie: initially a threat, then a friend character
• Child: outside helper
• Design Strategy
• 6 Sequences
• Real-time events collection and intervention
• Reproduction of past games for post evaluation and self-
examination
• Near real-time analytics
25
Fisherman and Genie

26. 26
Reading Flow

27. 27
Sequence 1 Sequence 2 Sequence 3 Sequence 4 Sequence 5 Sequence 6
Reading
Comprehen
sion
Fishermanis
preparingfor
fishing.
Fishermanis
headingfor
fishingplace.
Fishermanis
fishingabottle.
Genieis
threatening
fishermanto
kill.
Fisherman is
escaping out
ofthelife-
threatening
situation
using wisdom.
Fisherman
andreader
(child)are
savingGenie
together.
Contextual
Instruction
Greeting Keep high-
level
attention
without
hyper action
Keep high-
level
attention
according to
instructions
Comprehend
complex
dialogue
Comprehend
complex
dialogue
according to
instructions
Consider
others’
situation
Processes

28. • Reading Skill Improvement: By figuring out complicate dialogue in
Sequence 4 and 5 with sustained high-level attention
• Behavior Treatment: By training high-level attention,
keeping motionless in each scene,
and behaving accordingly in a certain situation
• Emotional Treatment: By helping the child experience feeling of
sympathy for Genie especially in Sequence 6.
28
Expected Effects via Complicate Reading with
Intervention Scheme

29. • Visualization of the Child’s Experience with the Game for Post
Evaluation and Self-Examination
• The child recognizes how Cyberspace was affected by
his/her behavior.
• As an Object Watcher, self-examination may stimulate
changes in his/her behavior
29
Treatment Effects via Self-Examination
with Reproduction of the Game

30. • Windows 8.1, 64 bit
• Ram: 8GB
• USB 3.0
• CPU: Intel Core i5-4590m
• HDD: 500GB SATA3
• GPU: GeForce 705
30
H/W Configuration

31. 1) Motion Tracking Test
• To see if brainwaves and motion tracking data are successfully
collected while playing the narrative game
• The result shows the motion tracking capabilities recognize 22
different motions successfully
2) Game Tracking Test
• Skeletal and face tracking have been tested while playing the
game.
• At the same time, the tester’s brainwaves were measured and
collected.
31
Experiments and Analysis

32. 32
Body
Category
Motion
No. Motions
Tester 1
No. Motions
Tester 2
No. Motions
Tester 3
No. Motions
Tester 4
No. Motions
Tester 5
UpperBody
UBLeft 2 4 1 3 0
UBRight 2 3 0 3 0
UBFront 0 5 2 1 1
UBBack 0 2 1 0 0
UBStand 0 2 0 1 1
UBSit 0 2 0 1 1
Left Hand
LHLeft 3 2 1 3 0
LHRight 2 1 1 3 0
LHFront 0 1 1 0 1
LHBack 0 1 0 0 0
LHUp 2 3 3 4 1
LHDown 0 0 0 0 1
Right Hand
RHLeft 2 2 4 3 0
RHRight 1 1 1 3 0
RHFront 0 1 1 1 0
RHBack 0 1 0 0 0
RHUp 2 1 3 3 1
RHDown 0 0 0 0 1
Left Foot
LFLeft 1 1 1 0 0
LFRight 1 1 1 0 0
LFFront 1 1 2 5 1
LFBack 0 1 0 0 0
LFUp 0 1 0 0 0
Right Foot
RFLeft 1 1 0 0 0
RFRight 1 1 0 0 0
RFFront 1 1 2 5 1
RFBack 0 1 0 1 0
RFUp 0 1 1 0 1
Game Tracking Results

33. 33
Body
Category
Motion
No. Motions
Tester 1
No. Motions
Tester 2
No. Motions
Tester 3
No. Motions
Tester 4
No. Motions
Tester 5
Head
HDLeft 1 2 1 2 1
HDRight 1 2 0 2 1
HDUp 0 2 0 1 0
HDDown 1 2 0 1 0
HDLeftTilt 0 2 1 1 1
HDRightTilt 0 1 0 1 1
Brainwaves
Avg. eSense
Attention 75.8759 62.7829 51.4263 53.7829 29.0852
Meditation 67.2558 63.0921 61.2523 55.2282 34.1012
Clear Time
(sec)
Total 65 112 67 101 61
Scene1 28 40 26 51 24
Scene2 37 59 41 50 37
Total No. Motion 25 53 28 48 15
Game Tracking Results (cont.)

34. • User List
• Experiment List
• Players Interaction Log:
• Motion Sensing Data Log: Body and face tracking
• Brainwaves: Attention and meditation tracking
• Web App Dashboard Demo
34
Web App Dashboard (Analytics)

35. 35
Click “admin” User
User List

36. 36
Experiment List of
“admin” User
Click this event
Experiment List

37. 37
Kinect Motion Tracking and Interaction

38. 38
Attention Tracking
Meditation Tracking
Kinect Motion Tracking and Interaction

39. 39
Demo Fisherman and Genie

40. • The main contribution is the implementation of the behavior training platform
which combines BCI and motion sensing technology with fairy tale
narrative contents and treat children withADHD in terms of the behavioral
treatment and reading comprehension.
• We are expecting this training platform to help the children improve their
attention and control hyperactive behavior based on narrative therapy
strategy.
• In addition to the current narrative content, “Fisherman and Genie,” we hope
to provide diverse contents with wide-range of fairy tales.
• We would like to test this platform as the treatment framework for children
withADHD by collaborating with related fields and medical professionals.
40
Review and Future Research

41. Thank you !
41
Q&A
Copyright 2016 Taesuk Kihl, Kyungeun Park, Seungie Park, Min-Jae Kim, Juno Chang, Seok Kyoo Kim all rights reserved.