SIM U KIN project is a virtual soft skills educator that simulates presentation and interviewing processes and gives feedback on the weakness points which includes body language, facial expression and speech mistakes then recommends the correct behavior.

1. Kinect Virtual
Learning
Team members:
 Mohamed Hesham
 Mahmoud Soliman
 Ahmed Amr
 Yousef Ahmed
 Ahmed Nasser Date: 5/7/2015

2. Agenda
• Problem & Solution
• Presentation Application
• Interviewing Application
• Pattern Recognition techniques
• Speech Recognition
• Demo
• Questions ?

3. Problem
• Why Soft Skills ?
• Business & Education
• Chance !!
• Software !!

4. Existing Solutions
• Expensive Courses
• Online Materials

5. Idea & Solution
• Simulate the presentation & interview Processes
• train the presenter & interviewee on the correct behavior
 Body movements
 Speech recognition
 Facial Expression
• Feedback of weakness points

6. What is Kinect !

7. Message Delivery
• Body Language
• Speech
• Facial Expression

8. Project Overview

9. PresentationApplication

10. Detected Body Mistakes
References

11. Body Joints

12. Mistakes Categories

13. Dynamic Mistakes

14. Body Language
Body Language

15. Body Language
Body Language

16. Body Language
Body Language

17. Static Mistakes

18. Body Language
Body Language

19. Body Language

20. Body Language

21. Body Language

22. Body Language

23. PresentationApplication Flow

33. Interview Application

34. References
Detected Face Mistakes

35. Facial Techniques
Face Joints Face Orientation

36. Face Gestures

37. Facial Techniques
Face Joints Face Orientation

38. Face Gestures

39. Face Gestures

40. Face Gestures

41. Face Gestures

42. Face Gestures

43. Face Gestures

44. InterviewApplication
Flow

55. Pattern Recognition
Techniques

56. Classifiers Categories

57. Rule-Based Classifiers
• Conditions
 Threshold
• 12 Joints with XYZ Coordinates

58. Hidden Markov Model Classifiers
• Dataset
 50 sample for each mistake
• Models
 9 states
 200 Observations
• Phases
 Learning
 Decoding
 Evaluate

59. 80
82
84
86
88
90
92
94
96
Rule-Based HMM
Arm
Leg
Body center
Classifier Arm Leg Center Type
Rule-Based 92% 88% 90% Static
HMM 95% 85% 93% Static –
Dynamic
Results and Statistics

60. Classifiers Conclusion
• HMM is more accurate than Rule-based and support
Dynamic states
• Rule-based is complex to detect specific threshold for
different bodies

61. Gesture Detection Flow
• Not all features are relevant to all gestures.
 Each gestures has its own feature vector.
• More than one Gesture can happen at the same time.
 We group related gestures together under respective limbs.
• Gestures can be related to more than one limb
 We divide gestures into parts called “states”.
Problems :

62. Gesture Detection Flow
Righ
t
Arm
Hand Over Hand
Hand On Waist
Hand In Pocket
Left
Arm
Hand Over Hand
Hand On Waist
Hand In Pocket
Righ
t Leg
Cross Leg
Up Leg
Left
Leg
Cross
Leg
Up Leg
body
Leaning
left
Leaning
right

63. Gesture Detection Flow
Flow :
• Each Gesture has a certain condition on the detected states.
• Body consists of 5 Limbs.
• Each Limb has a most probable state.
• Each state has a classifier object that receives the feature vector.
• Each state has its own feature vector.

64. Gesture Detection Flow
Gesture types :
• Static Gestures : No movement involved
 Happens when any state from a group of states happen.
 Happens when all states from a group of states happen.
• Dynamic Gestures : Requires the body to move
 Happens when a sequence of states happen in a short period of time.

65. Speech Recognition

66. Speech Recognition
• Speech recognition process

67. Speech Recognition
• Presentation application (Filler Words)
 Fillers words and phrases people use to cover verbal gaps—
are word crutches. Presenters often use them out of fear.
• The most common fillers are:
 So, And, All right, Okay, Like, Now, Well, You know, Right,
Um and Uh.

68. English Test phases

69. First Phase "Put Question"

70. Second Phase "Paragraph Test"

71. Third Phase “Knowledge"

72. Conclusion
• HMM is more accurate than Rule-based Classifiers.
• Kinect is the best device to use due to infrared feature.
• Kinect V2 is better than Kinect V1 in Joint detection.
• Kinect V2 has face Joints property over Kinect V1.

73. Future Work
• Interviewing Enhancement
• Try other classifiers seeking better accuracy
• Provide the Grammar Builder with more alternatives

74. Sponsorship

75. Presentation Demo

76. Interview Demo