Track 4 - Gamification ecosystems Authors: Paula Escudeiro, Nuno Escudeiro, Marcelo Norberto and Jorge Lopes https://www.youtube.com/watch?v=BcaVupA8uZY&list=PLboNOuyyzZ87fUoPqXZ_EyOGbtVW5OaDB&index=1

1. VIRTUALSIGN
TRANSLATOR AS
A BASE FOR A
SERIOUS GAME

2. VIRTUAL SIGN
GILT
Games, Interaction and Learning Technologies

3. OBJETIVES
Implementation of a Portuguese Sign
Language(PSL) bidirectional
translating system.

4. PORTUGUESE SIGN LANGUAGE
• PSL is an official language used in learning,
leisure moments and in professional
environments.
• In PSL, the speaker should take into
consideration a series of parameters that
define the manual and non-manual
components.

5. LINGUISTIC ASPECTS- I
Manual Components:
– Hand configuration;
– Hand orientation;
– Articulation position;
– Hand movement;

6. LINGUISTIC ASPECTS- II
Non-manual component:
– Body movement
– Facial expression

7. VIRTUAL SIGN: PROJECT
Three main modules:
 PSL translator to text;
 Text translator to PSL;
 Didatic game;

8. PSL TO TEXT

9. PSL TO TEXT - HARDWARE
• Sensor Gloves(5DT) • Kinect
• Provides information
about the hand
configuration
• Hands orientation and
movement, body movement
and facial expression

10. PSL TO TEXT – SOFTWARE I
“Divide and conquer”
– Bom Dia!

11. PSL TO TEXT – SOFTWARE II
Sign in PSL
• The beginning of a gesture is defined by a
configuration of the dominant hand.
• The ending of the gesture is determined
by the return of the dominant hand to a
neutral position or a configuration
change(new gesture).
• 57 configurations (letters,numbers, ...)

12. PSL TO TEXT – SOFTWARE III
• The gloves provide the information of each
hand allowing the software to estimate the
current hand configuration. The Kinect
provides infromation about the hands
movement and orientation
• The comparison between series of points is
done by applying a sequence alignment
algorithm.

13. PSL TO TEXT – SOFTWARE IV
The resulting sequence can represent one or more
words in sign langauge. The sequency is classified
using the database words as atoms.

14. LGP TO TEXT - SOFTWARE V
• Result sequency analysis :

15. PSL TO TEXT– IN PROGRESS
Processing not manual aspects:
– Body movement: Data retrieved by the kinect;
– Facial expression: Facial expression classifier;

16. TEXT TO PSL

17. TEXT TO PSL - AVATAR
• 3D animations created on
Blender
• Can be reused in other
avatars with similar bone
structure;

18. TEXT TO PSL - TRANSLATOR
The translator checks the
text for orthographic
mistakes and then
translates it to PSL.

19. TEXT TO PSL– INTEGRATING PPT
• Powerpoint add in
connected to the
translator;
• Text in the slides is
translated by the avatar;

20. TEXT TO PSL – CHAT
• Supporting long distance
communication using
PSL

21. SERIOUS GAME

22. SERIOUS GAME – ASPECTS I
Structure:
• 3 PSL learning scenes;
• 3 PSL evaluation scenes;
• Multiple minigames;
• Integration of the VirtualSign translator;

23. SERIOUS GAME – ASPECTS II
 Large maps
 Interactive environment
 Intuitive gameplay
 Inventory system
 Checkpoint system

24. SERIOUS GAME – SCENE I
• Alphabet learning scene
• Interactive objects;

25. SERIOUS GAME – SCENE II
• Free roaming scene
• Teaches basic words
• The player chooses the
order of events;

26. SERIOUS GAME – SCENE III
• Futuristic scene;
• Teaches basic sentences
• Specific sentences trigger
minigames;

27. SERIOUS GAME – EVALUATION SCENES

28. SERIOUS GAME – INTEGRATING THE TRANSLATOR

29. SERIOUS GAME - MINIGAMES
• Memory • 2D Shooter

30. VIRTUAL SIGN
Thank you!