Towards Virtual Rehabilitation - MCS (UCSC) Research Project 2016 - Presentation
Towards Virtual Rehabilitation:
Hand Gesture Controlled Games to Improve
Cognitive and Motor Skills of Children with
Thiwanka A. Makumburage
Dr. G.D.S.P. Wimalaratne
Nearly 34,000 of children
in Sri Lanka has
difficulties with cognitive
and motor skills
Support tools in
existence are either
traditional or out of
are very hard to find and
their services are very
expensive to be afforded
What are Serious
Can we play games
The aim of the research on a specialized approach to help the
children with special needs on their rehabilitation exercise,
using serious games controlled by hand gestures, to improve
their cognitive functions and motor skills.
Track the hand gestures so precisely, that it can be used to identify the
improvements of the human body. Uniquely identifying fingers, joints of a
hand leads to better information input to the serious game.
Identify the type of serious games that are suitable for the cognitive and
motor skills development of a child.
Develop a process by utilizing real-time precision hand gesture tracking to
bridge the hand gestures with the game actions.
Collection and analyze of the game play data, measure the improvements
of children’s cognitive and motor skills.
Developing serious games for children with cerebral palsy:
case study and pilot trial 
Discusses a series of approaches used to develop appropriate serious games
for children with cerebral palsy. They have used two sample games shown
above, including data and feedback from one child with cerebral palsy who
evaluated the games for entertainment, engagement and replay value. This
research only focuses on improvements of the cognitive functions of the
cerebral palsy effected children.
Help Me! A Serious Game for Rehabilitation of Children
Affected by Cerebral Visual Impairment 
The system integrates an eye tracker system to correctly measure the
performances of the child. The developed game was called HelpMe! And has
two goals: to train the ability to see and touch an object at the same time and
to train the problem-solving capabilities of the child.
Azahar: Serious games, improving autonomy 
Azahar is a platform for free, customizable applications and Serious
Games designed to improve the quality of life and autonomy of
people with autism and/or intellectual disability
Gesture based serious games for physical therapy 
This is a game developed by advancedmedialab.com in Saudi Arabia as a
replacement for physiotherapy exercises. It’s an online flying game that can
be played by a patient using hand and body gestures. The gestures used to
play the game are the ones required to perform physical therapy. Difficulty
level of the game can be changed automatically by the system or remotely by
the therapist according to the recovery behavior of each patient.
Visited Sri Lankan therapy units
National Hospital of Sri Lanka
Evaluation of hand gesture recognition and hand gesture
Taxonomy built by categorizing the serious games.
Identify the type of serious games that are suitable for the cognitive
and motor skills development of a child.
Develop a process by utilizing real-time precision hand gesture
tracking to bridge the hand gestures with the game actions.
‘Leap Play’ application, to improve the cognitive and motor skills of the
children with special needs.
Children who participated for the evaluation of the research, got their
cognitive and motor skills improved.
T = Theoretical minimum time spent to finish the level (in seconds)
Tx = Time spent to finish the level (in seconds)
N = Number of times repeated the level
M = Maximum marks can be achieved from the level
Using a Natural User Interface, LeapPlay application was able to
improve the skills of disabled children using the serious games
specially designed for therapy.
This project introduces a new way of utilizing hand gestures
controls for the wellbeing of the society.
The literature review on serious games and hand gesture
recognition lead to better understanding of the knowledge area.
Visits to the National Hospital of Sri Lanka and Lady Ridgeway
Hospital for Children gave insight of the problem in Sri Lankan
Detailed comparison between hand gestures controllers lead to
the identification of the Leap Motion device as a good gesture
LeapPlay system was implemented according to the design with
the limitations mentioned in the design chapter.
The project was evaluated successfully and the results were
positive towards achieving the goal.
Finally this project will provide a strong case to support the fact
that serious games can contribute towards the virtual
Identified multiple ways to precision hand gesture tracking.
Fatigue is a major concern for long duration game play.
Integrate the system with haptic feedback gloves.
 Henschke, M., Hobbs, D., & Wilkinson, B. (2012, November).
Developing serious games for children with cerebral palsy: case study and
pilot trial. In Proceedings of the 24th Australian Computer-Human
Interaction Conference (pp. 212-221). ACM
 Ciman, M., Gaggi, O., Nota, L., Pinello, L., Riparelli, N., & Sgaramella, T.
M. (2013). Help Me! A Serious Game for Rehabilitation of Children
Affected by CVI. In WEBIST (pp. 257-262).
 Azahar: Serious Games Improving Autonomy ,
 Serious Games for Physical Threapy , http://advancedmedialab.com/e-
 R.A.G.K. Ranathunga, L.A.V.N. Rajakaruna, S.A.T.N. Karunarathne, L.N.Y.
Abeywardena,D.P. Nawinna, T. Halloluwa (2014) 'A Gamified Learning
Tool for Sri Lankan Primary Schools ', PNCTM, Vol 3.