This document discusses developing a system called LeapPlay that uses hand gesture control of serious games to improve cognitive and motor skills in children with special needs. It aims to precisely track hand gestures to identify improvements and develop games suitable for skills development. The system utilizes real-time hand gesture tracking from a Leap Motion controller to link gestures to game actions. An evaluation with 6 users found that the LeapPlay application was able to improve skills through the specially designed therapy games controlled by natural gestures.

1. Towards Virtual Rehabilitation:
Hand Gesture Controlled Games to Improve
Cognitive and Motor Skills of Children with
Special Needs
Thiwanka A. Makumburage
2013/MCS/048
2016
Supervisor
Dr. G.D.S.P. Wimalaratne

2. Nearly 34,000 of children
in Sri Lanka has
difficulties with cognitive
and motor skills
Support tools in
existence are either
traditional or out of
date.
Healthcare professionals
are very hard to find and
their services are very
expensive to be afforded
What are Serious
Games?
Can we play games
using hand
gestures?
Will games
Improve human
skills?
Introduction

3. Aim
 The aim of the research on a specialized approach to help the
children with special needs on their rehabilitation exercise,
using serious games controlled by hand gestures, to improve
their cognitive functions and motor skills.

4. Objectives
 Track the hand gestures so precisely, that it can be used to identify the
improvements of the human body. Uniquely identifying fingers, joints of a
hand leads to better information input to the serious game.
 Identify the type of serious games that are suitable for the cognitive and
motor skills development of a child.
 Develop a process by utilizing real-time precision hand gesture tracking to
bridge the hand gestures with the game actions.
 Collection and analyze of the game play data, measure the improvements
of children’s cognitive and motor skills.

5. Research
 Literature review
 Identifying serious
games

6.  Creating taxonomy of serious games

7.  Evaluation of the related work

8.  Developing serious games for children with cerebral palsy:
case study and pilot trial [1]
Discusses a series of approaches used to develop appropriate serious games
for children with cerebral palsy. They have used two sample games shown
above, including data and feedback from one child with cerebral palsy who
evaluated the games for entertainment, engagement and replay value. This
research only focuses on improvements of the cognitive functions of the
cerebral palsy effected children.

9.  Help Me! A Serious Game for Rehabilitation of Children
Affected by Cerebral Visual Impairment [2]
The system integrates an eye tracker system to correctly measure the
performances of the child. The developed game was called HelpMe! And has
two goals: to train the ability to see and touch an object at the same time and
to train the problem-solving capabilities of the child.

10.  Azahar: Serious games, improving autonomy [3]
Azahar is a platform for free, customizable applications and Serious
Games designed to improve the quality of life and autonomy of
people with autism and/or intellectual disability

11.  Gesture based serious games for physical therapy [4]
This is a game developed by advancedmedialab.com in Saudi Arabia as a
replacement for physiotherapy exercises. It’s an online flying game that can
be played by a patient using hand and body gestures. The gestures used to
play the game are the ones required to perform physical therapy. Difficulty
level of the game can be changed automatically by the system or remotely by
the therapist according to the recovery behavior of each patient.

12.  Visited Sri Lankan therapy units
 National Hospital of Sri Lanka

13.  Lady Ridgeway Hospital for
Children

14.  Evaluation of hand gesture recognition and hand gesture
controllers.

16.  Contribution
 Taxonomy built by categorizing the serious games.
 Identify the type of serious games that are suitable for the cognitive
and motor skills development of a child.
 Develop a process by utilizing real-time precision hand gesture
tracking to bridge the hand gestures with the game actions.
 ‘Leap Play’ application, to improve the cognitive and motor skills of the
children with special needs.
 Children who participated for the evaluation of the research, got their
cognitive and motor skills improved.

17.  System Architecture

18.  Process

19.  Process Flow

20.  Algorithms
 Scoring
[5]
T = Theoretical minimum time spent to finish the level (in seconds)
Tx = Time spent to finish the level (in seconds)
N = Number of times repeated the level
M = Maximum marks can be achieved from the level

21.  Coordinate conversion

22.  Technology
 Java 8
 Leap Motion Java SDK (Version 2.2.3)
 JavaFX 2
 MySQL

23.  Evaluation
 Six users
 Multiple Scenarios
 Usability feedback
 Critical evaluation of the results

24.  Results

25.  Gesture recognition

26.  Evaluation Form

27.  Sample (User B) :

30.  Conclusion
 Using a Natural User Interface, LeapPlay application was able to
improve the skills of disabled children using the serious games
specially designed for therapy.
 This project introduces a new way of utilizing hand gestures
controls for the wellbeing of the society.
 The literature review on serious games and hand gesture
recognition lead to better understanding of the knowledge area.
 Visits to the National Hospital of Sri Lanka and Lady Ridgeway
Hospital for Children gave insight of the problem in Sri Lankan
context.

31.  Detailed comparison between hand gestures controllers lead to
the identification of the Leap Motion device as a good gesture
tracker .
 LeapPlay system was implemented according to the design with
the limitations mentioned in the design chapter.
 The project was evaluated successfully and the results were
positive towards achieving the goal.
 Finally this project will provide a strong case to support the fact
that serious games can contribute towards the virtual
rehabilitation.

32.  Lessons learned
 Identified multiple ways to precision hand gesture tracking.
 Fatigue is a major concern for long duration game play.
 Future work
 Integrate the system with haptic feedback gloves.

33.  Proof of Concept

34. References
[1] Henschke, M., Hobbs, D., & Wilkinson, B. (2012, November).
Developing serious games for children with cerebral palsy: case study and
pilot trial. In Proceedings of the 24th Australian Computer-Human
Interaction Conference (pp. 212-221). ACM
[2] Ciman, M., Gaggi, O., Nota, L., Pinello, L., Riparelli, N., & Sgaramella, T.
M. (2013). Help Me! A Serious Game for Rehabilitation of Children
Affected by CVI. In WEBIST (pp. 257-262).
[3] Azahar: Serious Games Improving Autonomy ,
http://seriousgamesmarket.blogspot.com/2015/09/azahar-serious-
games-improving-autonomy.html
[4] Serious Games for Physical Threapy , http://advancedmedialab.com/e-
health/products/gesture-based-serious-games-for-physical-therapy.html
[5] R.A.G.K. Ranathunga, L.A.V.N. Rajakaruna, S.A.T.N. Karunarathne, L.N.Y.
Abeywardena,D.P. Nawinna, T. Halloluwa (2014) 'A Gamified Learning
Tool for Sri Lankan Primary Schools ', PNCTM, Vol 3.

35. Thank You