It is presentation on automatic user state recognition for hand gesture based low cost tv control system

1. Automatic User State
Recognition
for Hand Gesture Based Low-
Cost
Television Control System
Martina Thampan
S7c
28

2. CONTENTS
 INTRODUCTION
 HAND FREE CONTROL SYSTEM
 PROPOSED SYSTEM
 COMPONENTS OF AUSR SYSTEM
 USER STATE DEFINITION
 USER STATE INITIALIZATION
 USER STATE TRANSITION
 HAND GESTURE RECOGNITION
 EXPERIMENT RESULTS
 CONCLUSION
 REFERENCES
AUSR FOR HAND GESTURE BASED LOW COST TV CONTROL SYSTEM
2

3. INTRODUCTION
Innovation in TV display screen
TV controller remains unchanged
Fitt's Law
Hand Free control system
AUSR FOR HAND GESTURE BASED LOW COST TV CONTROL SYSTEM
3

4. HAND FREE CONTROL SYSTEM
Voice controlled devices
Hand gesture control system
AUSR FOR HAND GESTURE BASED LOW COST TV CONTROL SYSTEM
4

5. PROPOSED SYSTEM
 Low cost hand gesture in automatic manner
AUSR FOR HAND GESTURE BASED LOW COST TV CONTROL SYSTEM
5

6. PROPOSED SYSTEM(contd..)
Automatic user state recognition(AUSR)
Ultra sonic distance array and camera
module
AUSR FOR HAND GESTURE BASED LOW COST TV CONTROL SYSTEM
6

7. IMPLEMENTATION OF THE SYSTEM
AUSR FOR HAND GESTURE BASED LOW COST TV CONTROL SYSTEM
7

8. COMPONENTS OF AUSR SYSTEM
Hardware
TV
set-top box
sensors(RGB or depth cameras)
 ultrasonic distance array
AUSR FOR HAND GESTURE BASED LOW COST TV CONTROL SYSTEM
8

9. COMPONENTS OF AUSR(contd…)
Embedded software components
connects sensors with TV
Process the sensor data
Drive the TV control process
AUSR FOR HAND GESTURE BASED LOW COST TV CONTROL SYSTEM
9

10. USER STATE DEFINITION
Classification of user state
Absent
Other action
Controlling
Watching
AUSR FOR HAND GESTURE BASED LOW COST TV CONTROL SYSTEM
10

11. USER STATE DEFINITION (contd..)
AUSR FOR HAND GESTURE BASED LOW COST TV CONTROL SYSTEM
11

12. USER STATE INITIALIZATION
 Action detection
 Distance detection
 Movement detection
 Single sensor based & sensor array based
action decision.
AUSR FOR HAND GESTURE BASED LOW COST TV CONTROL SYSTEM
12

13. USER STATE INITIALIZATION (contd..)
 L(t)=C.T(t)/2
L(t)=distance between object and sensor
C=sonic speed in air(340 m/s).
T(t)=time between the sensor emits and
receives the ultrasonic waves.
 ∆Lt =|L(t+P)-L(t)|
AUSR FOR HAND GESTURE BASED LOW COST TV CONTROL SYSTEM
13

14. USER STATE INITIALIZATION (contd..)
 At = 0, ∆Lt <R (considering the threshold)
1,∆Lt ≥ R
AUSR FOR HAND GESTURE BASED LOW COST TV CONTROL SYSTEM
14

15. USER STATE INITIALIZATION (contd..)
Considering action sequence :
St=[∆ Lt-k+1 ,∆Lt-k+2 ,..., ∆Lt-1, ∆Lt ]
Based on threshold R for non intentional
movements
SRt =[At-k+1 ,At-k+2 ,..,At-1 , At ]
AUSR FOR HAND GESTURE BASED LOW COST TV CONTROL SYSTEM
15

16. USER STATE INITIALIZATION (contd..)
 Action status
A= 0, SUM(SRt )< Q
1, SUM(SRt )≥ Q
 SUM(SRt )=∑ i=0
K-1 At-i
 S=2.√2.r=2√2.D.tgθ
 Action decision will be according to
A= 0, max [A0 ,A1 ,A2 ,A3 ]=0
1, max [A0 ,A1 ,A2 ,A3 ]=1
AUSR FOR HAND GESTURE BASED LOW COST TV CONTROL SYSTEM
16

17. USER STATE INITIALIZATION (contd..)
 Presentation detection
 Face detection
Horizontal view angle α
α= arctg(W/2L)
 arctg- inverse trigonometric function
 W- width of tv screen
 L-distance between user eye and TV screen.
AUSR FOR HAND GESTURE BASED LOW COST TV CONTROL SYSTEM
17

18. USER STATE INITIALIZATION (contd..)
 Vertical view angle β
Β = arctg(H/2L)
H- height of TV screen
 Roll angle[φ]
 Lf ={F|-α≤Fh ≤α, -β ≤Fv ≤β and –φ ≤ Fr ≤φ }
 B= o, if the face is not detected
1, if the face is detected
AUSR FOR HAND GESTURE BASED LOW COST TV CONTROL SYSTEM
18

19. USER STATE INITIALIZATION
(contd..)
AUSR FOR HAND GESTURE BASED LOW COST TV CONTROL SYSTEM
19

20. USER STATE INITIALIZATION (contd..)
AUSR FOR HAND GESTURE BASED LOW COST TV CONTROL SYSTEM
20

21. USER STATE INITIALIZATION (contd..)
User state decision
decided from results of action detection
and presentation detection.
AUSR FOR HAND GESTURE BASED LOW COST TV CONTROL SYSTEM
21

22. USER STATE INITIALIZATION (contd..)
ALGORITHIM OF DECISION OF USER STATE
If (the intentional motion is detected)
If (the face watching TV is detected)
Set (A=1,B=1) - Controlling
Else
Set (A=1,B=0) - Other Action
Else if (the face watching TV is detected)
Set (A=0,B=1) - Watching
Else
Set (A=0,B=0) - Absent
AUSR FOR HAND GESTURE BASED LOW COST TV CONTROL SYSTEM
22

23. USER STATE INITIALIZATION (contd..)
AUSR FOR HAND GESTURE BASED LOW COST TV CONTROL SYSTEM
23

24. USER STATE TRANSITION
If (the intentional motion is detected)
If (A=1 and B=1)
Detect hand gestures
Else if (A=0 and B=0)
Set (A=1,B=0)
Else if (A=1 and B=0)
If (the face watching TV is detected)
Set (A=1,B=1)
Else
If (the face watching TV is detected)
Set (A=1,B=1)
Else if (the intentional motion is not detected for a long time)
If (A=1,B=0)
Set (A=0,B=0)
Else if(A=1,B=1)
Set (A=0,B=1)
AUSR FOR HAND GESTURE BASED LOW COST TV CONTROL SYSTEM
24

25. USER STATE TRANSITION(contd..)
AUSR FOR HAND GESTURE BASED LOW COST TV CONTROL SYSTEM
25

26. HAND GESTURE RECOGNITION
 Hand gesture recognition module & camera
activated when user state is controlling.
 Gesture continuously detected in Tf
 Tf is the threshold for restrict the time period
permitted for the user to do hand gestures
AUSR FOR HAND GESTURE BASED LOW COST TV CONTROL SYSTEM
26

27. EXPERIMENTAL RESULTS
 User study
 Out of 6 people 2young,2middle age and 2 old users
are considered
AUSR FOR HAND GESTURE BASED LOW COST TV CONTROL SYSTEM
27

28. EXPERIMENTAL RESULTS (contd..)
 Automatic user state recognition
AUSR FOR HAND GESTURE BASED LOW COST TV CONTROL SYSTEM
28

29. EXPERIMENTAL RESULTS (contd..)
 Computational cost and power consumption
AUSR FOR HAND GESTURE BASED LOW COST TV CONTROL SYSTEM
29

30. CONCLUSIONS
 TV user’s states are defined according to finite state
machine.
 Camera system and gesture recognition is activated or
closed by recognised user state.
 Reduction in computational cost and power consumption
AUSR FOR HAND GESTURE BASED LOW COST TV CONTROL SYSTEM
30

31. REFERENCES
 [1]Shiguo Lian, Member, IEEE, Wei Hu, Kai Wang,” Automatic User State
Recognition for Hand Gesture Based Low-Cost Television Control System”,
IEEE Transactions on Consumer Electronics, Vol. 60, No. 1, February 2014.
 [2]I. S. Mackenzie and W. Buxton, "Extending fitts' law to two dimensional
tasks," in Proceedings of CHI'92, pp. 219-226, May 1992.
 [3]S. K. Kim, G. H. Park, S. H. Yim, S. M. Choi, and S. J. Choi, “Gesture
recognizing hand-held interface with vibrotactile feedback for 3D
interaction,” IEEE Trans. Consumer Electronics, vol. 55, no. 3, pp.1169-1177,
Aug. 2009.
 [4]H. Heo, E. C. Lee, K. R. Park, C. J. Kim, and M. C. Whang, “A realistic
game system using multi-modal user interfaces,” IEEE Trans. Consumer
Electronics, vol. 56, no. 3, pp. 1364-1372, Aug. 2010.
 [5]D. W. Lee, J. M. Lim, S. W. John, I. Y. Cho, and C. H. Lee, “Actual remote
control: a universal remote control using hand motions on a virtual menu,”
IEEE Trans. Consumer Electronics, vol. 55, no. 3, pp.1439-1446, Aug. 2009
AUSR FOR HAND GESTURE BASED LOW COST TV CONTROL SYSTEM
31

32. Questions please??
AUSR FOR HAND GESTURE BASED LOW COST TV CONTROL SYSTEM
32

33. THANK YOU