Successfully reported this slideshow.

Casestudy handgesture hasanlatif

1

Share

Loading in …3
×
1 of 38
1 of 38

More Related Content

Similar to Casestudy handgesture hasanlatif

Related Books

Free with a 14 day trial from Scribd

See all

Casestudy handgesture hasanlatif

  1. 1. Hand Gesture Recognition By Hasan Latif hasanlateef@outlook.com
  2. 2. To be Discussed in this Presentation • Introduction • Examples • Related Studies • Gesture Recognition System Pipeline • Feature Extraction Algorithm • Datasets • Results Hand Gesture Recognition 26/21/2019
  3. 3. What is Gesture? • A gesture is a non-verbal communication in which a part of body expresses some information especially by head or hand. Hand Gesture Recognition 36/21/2019 Introduction Hand Gesture Examples Related Studies Block Diagram Feature Extracting algorithm Datasets Results References
  4. 4. Few Hand Gesture Examples Hand Gesture Recognition 46/21/2019 Introduction Hand Gesture Examples Related Studies Block Diagram Feature Extracting algorithm Datasets Results References
  5. 5. Related Studies-i Many Researchers have proposed numerous methods: • Glove Based [1] • Requires the use of glove • Extra hardware • Variable Glove size for different user • Marker Based [2] • Use of colour markers on fingers. • Marker positions are fixed. • Complex to use multiple markers. Hand Gesture Recognition 56/21/2019 Introduction Hand Gesture Examples Related Studies Block Diagram Feature Extracting algorithm Datasets Results References
  6. 6. Related Studies-ii Many Researchers have proposed numerous methods: • Vision Based [3], [4], [5] • Assembly uses Camera and processing unit. • No Extra hardware required. • A complex problem due to high degree of freedom. Hand Gesture Recognition 66/21/2019 Introduction Hand Gesture Examples Related Studies Block Diagram Feature Extracting algorithm Datasets Results References
  7. 7. Gesture Recognition System Pipeline Hand Gesture Recognition 76/21/2019 Camera Input acquisition Image enhancement Feature Extraction Feature Matching Feature database If feature matches with the feature in database Gesture will be recognized yes No Gesture will not be recognized Introduction Hand Gesture Examples Related Studies Block Diagram Feature Extracting algorithm Datasets Results References
  8. 8. Features-Extraction Algorithms • Principle Component Analysis (PCA) [3], [4] • HOG features [5] • Scale invariant feature transform with SVM [7] Hand Gesture Recognition 86/21/2019 Introduction Hand Gesture Examples Related Studies Block Diagram Feature Extracting algorithm Datasets Results References
  9. 9. Features • Principle Component Analysis (PCA) • HOG features • SIFT features along with bag of words Hand Gesture Recognition 96/21/2019 Introduction Hand Gesture Examples Related Studies Block Diagram Feature Extracting algorithm Datasets Results References
  10. 10. Principal Component Analysis-Training Phase SUMMARY Hand Gesture Recognition 106/21/2019 Gesture-1 Training Images Gesture-2 Training Images Gesture-3 Training Images Gesture-4 Training Images Extracting PCA features and storing M best eigen vectors Projecting data into eigen space Introduction Hand Gesture Examples Related Studies Block Diagram Feature Extracting algorithm Datasets Results References
  11. 11. Principal Component Analysis-Training Phase Hand Gesture Recognition 116/21/2019 Step 1: • Convert each ‘N × 𝑁 ’ training image to column vector of 𝑁2× 1 . • Stack the column vectors horizontally. NxN image …… Dataset …… N2 × 𝑀 𝑇1 𝑇2 … . 𝐷𝑖𝑚 𝑇 = 𝑁2 𝑀 = 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑡𝑟𝑎𝑖𝑛𝑖𝑛𝑔 𝑖𝑚𝑎𝑔𝑒𝑠 Introduction Hand Gesture Examples Related Studies Block Diagram Feature Extracting algorithm Datasets Results References
  12. 12. Principal Component Analysis-Training Phase Hand Gesture Recognition 126/21/2019 Step 2:Normalization • Calculate average vector ‘u’ which is given by 𝒖 = 𝑖=1 𝑀 𝑻𝒊 𝑀 ,Where M = Number of training images • Subtract average vector 𝒖 from each of the vector in the training matrix which is given by ∅𝒊 = 𝑻𝒊 − 𝒖 𝑤ℎ𝑒𝑟𝑒, ∅𝑖 is the ith normalized training vector Introduction Hand Gesture Examples Related Studies Block Diagram Feature Extracting algorithm Datasets Results References
  13. 13. Principal Component Analysis-Training Phase Hand Gesture Recognition 136/21/2019 Step 3: • Calculation of Covariance Matrix ‘C’ which is given by: 𝐶 = 1 𝑁 𝑛=1 𝑁 ∅ 𝑛∅ 𝑛 𝑇 = 𝐴𝐴 𝑇 • Obtain 𝑀 best eigen vectors of 𝐴𝐴 𝑇 of corresponding eigen values. i-th eigen vector of covarince matrix be 𝑒𝑖𝑔𝑖 Introduction Hand Gesture Examples Related Studies Block Diagram Feature Extracting algorithm Datasets Results References
  14. 14. Principal Component Analysis-Training Phase Hand Gesture Recognition 146/21/2019 Step 4: • Stack ‘M’ eigen vectors such that eigen vector corresponding to biggest eigen value should be placed first and then transpose the feature matrix. 𝐹𝑒𝑎𝑡𝑢𝑟𝑒𝑀𝑎𝑡𝑟𝑖𝑥 = 𝑒𝑖𝑔1 𝑒𝑖𝑔2 𝑒𝑖𝑔3 . . . 𝑒𝑖𝑔 𝑀 𝑇 ; 𝐷𝑖𝑚 𝐹𝑒𝑎𝑡𝑢𝑟𝑒 𝑀𝑎𝑡𝑟𝑖𝑥 = 𝑀 × 𝑁2 • Multiply the Feature matrix with the mean adjusted data(normalized data) to project the data in to eigen space. Introduction Hand Gesture Examples Related Studies Block Diagram Feature Extracting algorithm Datasets Results References
  15. 15. Principal Component Analysis-Testing Phase Hand Segmentation Hand Gesture Recognition 156/21/2019 Mandeep’s Method Nasser H. Dardas’s Method Conversion from RGB to YCbCr color space. Face Detection Algorithm (Voila-Jones) Skin Color Segmentation Face Subtraction Ostu’s Thresholding Apply color based skin detection algorithm and contour comparison algorithm. Introduction Hand Gesture Examples Related Studies Block Diagram Feature Extracting algorithm Datasets Results References Uses only hand images
  16. 16. ` 6/21/2019 Hand Gesture Recognition 17 Convert the input image to a column vector Normalize the column vector Project Normalized column vectors onto the eigen space Calculate Distance between input weight vector and all the weight vector of training set €=|Ω–Ωi|2 i=1…M Is Distance is min ? UNKNOWN Gesture NO YES RECOGNIZED Gesture Face Detectiion and subtractionInput image Introduction Hand Gesture Examples Related Studies Block Diagram Feature Extracting algorithm Datasets Results References System Proposed By Naseer [3]
  17. 17. Principal Component Analysis-Testing Phase Hand Gesture Recognition 186/21/2019 Summary –testing phase: • Convert Test image to column vector. • Subtract the average vector (𝑢) from the input image (𝛾). • Projecting the test image into eigen space. • Use distance measure to compare features. if features matches 𝜸 is recognized as the 𝑗𝑡ℎ image in the training set. Introduction Hand Gesture Examples Related Studies Block Diagram Feature Extracting algorithm Datasets Results References
  18. 18. Features • Principle Component Analysis (PCA) • HOG features • Bag of Features and Support Vector Machines (SVM) Hand Gesture Recognition 196/21/2019
  19. 19. Histogram of Oriented Gradients (HOG) • Also known as HOG. • Proposed by N.Dalal and Triggs [4] for pedestrian detection. • Used with Support vector machine (SVM) for classification of the pedestrian detection. • K. Feng and F. Yuan [5] uses it for hand gesture recognition system. Hand Gesture Recognition 206/21/2019 Introduction Hand Gesture Examples Related Studies Block Diagram Feature Extracting algorithm Datasets Results References
  20. 20. HOG features (Training Phase) • Extraction of HOG features Hand Gesture Recognition 216/21/2019 Window on image(Preprocessing) Compute Gradient Project gradient to gradient direction of pixel Normalizaton HOG Features Introduction Hand Gesture Examples Related Studies Block Diagram Feature Extracting algorithm Datasets Results References
  21. 21. HOG features with SVM Steps of extracting HOG features are given: Step1: Pre-Processing • Select image patch to be analyzed. • Image Patch to be analyzed should have aspect ratio of 1:2.eg they can be 100x200,128x256 etc. Hand Gesture Recognition 226/21/2019 Introduction Hand Gesture Examples Related Studies Block Diagram Feature Extracting algorithm Datasets Results References
  22. 22. Histogram of Oriented Gradients (HOG) Step2: Image Gradient • Calculate the image gradient. • Easily done by filtering(convolving) the image with the following kernels. • Magnitude and direction is given by 𝐺 = 𝐺 𝑥 2 + 𝐺 𝑦 2 𝜃 = tan−1 𝐺 𝑦 𝐺 𝑥 Hand Gesture Recognition 236/21/2019 Introduction Hand Gesture Examples Related Studies Block Diagram Feature Extracting algorithm Datasets Results References
  23. 23. Histogram of Oriented Gradients (HOG) Step3 • Create Histogram of gradients. • Histogram contains 9 bins. i.e 0-180. • Each Bin is 20 degree. Hand Gesture Recognition 246/21/2019 Introduction Hand Gesture Examples Related Studies Block Diagram Feature Extracting algorithm Datasets Results References
  24. 24. Histogram of Oriented Gradients (HOG) Step3:Creating Histogram of gradients Hand Gesture Recognition 256/21/2019 Histogram of Gradients [7] Histogram Introduction Hand Gesture Examples Related Studies Block Diagram Feature Extracting algorithm Datasets Results References
  25. 25. Histogram of Oriented Gradients (HOG) Step4:Block Normalization • In order to get unit vector we, divide it with the norm of the that vector.(intensity invariant). 𝑢 = 𝑢 𝑢 By this way we have bunch of feature vectors. Hand Gesture Recognition 266/21/2019 Introduction Hand Gesture Examples Related Studies Block Diagram Feature Extracting algorithm Datasets Results References
  26. 26. 6/21/2019 Hand Gesture Recognition 27 HOG features with SVM (Testing Phase) Trained SVM UNKNOWN Gesture NOYES RECOGNIZED Gesture Extraction of HOG featuresTest image Introduction Hand Gesture Examples Related Studies Block Diagram Feature Extracting algorithm Datasets Results References
  27. 27. HOG features with SVM Support Vector Machine (SVM) • Supervised machine learning algorithm • Can be used for both classification and regression • Given 2 or more classes with labels, separates classes using hyperplane Hand Gesture Recognition 286/21/2019 Introduction Hand Gesture Examples Related Studies Block Diagram Feature Extracting algorithm Datasets Results References
  28. 28. HOG features with SVM Support Vector Machine (SVM) • Support vectors • Support vectors are the data points near to the hyperplane. • Act as critical element of dataset in building a classifier. Hand Gesture Recognition 296/21/2019 Introduction Hand Gesture Examples Related Studies Block Diagram Feature Extracting algorithm Datasets Results References
  29. 29. HOG features with SVM Support Vector Machine (SVM) • Hyperplane(Decision Line.): • A hyperplane in 𝑅 𝑛 is an 𝑛 − 1 dimensional subspace. Hand Gesture Recognition 306/21/2019 Hyperplane in 𝑅2 is a line. [8] Hyperplane in 𝑅3 is a plane. [8] Introduction Hand Gesture Examples Related Studies Block Diagram Feature Extracting algorithm Datasets Results References
  30. 30. HOG features with SVM Support Vector Machine (SVM) • Linear vs nonlinear classification? • Sometimes, Data is linearly separable(𝑦 = 𝑚𝑥 + 𝑐), no matter the dimension of feature vector (y = 𝑥 + 𝑧 + 𝑏 + 𝑞). • In case of nonlinear classification: Hand Gesture Recognition 316/21/2019 decision boundary is kernel. Introduction Hand Gesture Examples Related Studies Block Diagram Feature Extracting algorithm Datasets Results References
  31. 31. HOG features with SVM Support Vector Machine (SVM) • Commonly used Kernel: Hand Gesture Recognition 326/21/2019 Linear Kernel [8] Polynomial Kernel of degree 2 [8] RBF Kernel [8] Introduction Hand Gesture Examples Related Studies Block Diagram Feature Extracting algorithm Datasets Results References
  32. 32. HOG features with SVM • Experimental Results and Analysis • HOG features are given to SVM. • Kernel function: RBF. Hand Gesture Recognition 336/21/2019 Introduction Hand Gesture Examples Related Studies Block Diagram Feature Extracting algorithm Datasets Results References
  33. 33. Dataset. Nasser Et al. [4] dataset • Ten Video files with resolution 640x480 recored / hand gesture(fist, index, littel finger and palm) • Length of each file is 100 images. • Dataset not available publicly Hand Gesture Recognition 346/21/2019 Introduction Hand Gesture Examples Related Studies Block Diagram Feature Extracting algorithm Datasets Results References Mandeep Kaur Ahuja Et al. [3] Dataset • Number of Gestures: 4 • Per Gesture Training image:20 • Total number of training images:20 x4=80 images • Images are taken under constant background. • Dataset not available publicly.
  34. 34. Dataset. K. Feng Et all. [5] dataset • Training images : 1000 each gesture. • Total gestures :10 • Test images :100 • Dataset are not available publicly. Hand Gesture Recognition 356/21/2019 Introduction Hand Gesture Examples Related Studies Block Diagram Feature Extracting algorithm Datasets Results References
  35. 35. Results Hand Gesture Recognition 366/21/2019 Gesture Number of frames Correct Incorrect Fist 1000 933 67 Index 1000 928 72 Little 1000 912 88 Palm 1000 945 55 Results by Nasser H. Dardas and Nicolas D. Georganas [4] Gesture Number of frames Correct Incorrect Gesture1 20 18 2 Gesture2 20 19 1 Gesture3 20 17 3 Gesture4 20 19 1 Results by Mandeep Kaur Ahuja & Amardeep Singh [3] Introduction Hand Gesture Examples Related Studies Block Diagram Feature Extracting algorithm Datasets Results References
  36. 36. Results • Experimental Results and Analysis Hand Gesture Recognition 376/21/2019 Digital Gesture 0 1 2 3 4 5 Average Recognition rate(%) Training sample recognition rate(%) 100 100 100 100 100 100 100 Prediction sample recognition rate(%) 99.1 99.1 91.6 87.0 84.0 87.0 91.3 Introduction Hand Gesture Examples Related Studies Block Diagram Feature Extracting algorithm Datasets Results References Results by K. Feng and F. Yuan [5]
  37. 37. Results • Experimental Results and Analysis Hand Gesture Recognition 386/21/2019 Digital Gesture 0 1 2 3 4 5 Average Recognition rate(%) Recognition rate in Brighter illumination(%) 96.5 97.0 92.6 85.6 84.0 99.0 92.5 Recognition rate in darker illumination (%) 95.3 95.7 90.7 90.7 85.2 95.7 92.3 Introduction Hand Gesture Examples Related Studies Block Diagram Feature Extracting algorithm Datasets Results References Results by K. Feng and F. Yuan [5]
  38. 38. References [1]. Y. Huang, D. Monekosso, H. Wang and J. C. Augusto, "A Concept Grounding Approach for Glove-Based Gesture Recognition," 2011 Seventh International Conference on Intelligent Environments, Nottingham, 2011, pp. 358-361. [2]. C. Wang, X. H. Shi, L. W. Liu and S. C. Chan, "A marker-less two-hand gesture recognition system using kinect depth camera," 2015 IEEE International Conference on Signal Processing, Communications and Computing (ICSPCC), Ningbo, 2015, pp. 1-4. [3]. M. K. Ahuja and A. Singh, "Static vision based Hand Gesture recognition using principal component analysis," 2015 IEEE 3rd International Conference on MOOCs, Innovation and Technology in Education (MITE), Amritsar, 2015, pp. 402-406. [4]. N. H. Dardas and E. M. Petriu, "Hand gesture detection and recognition using principal component analysis," 2011 IEEE International Conference on Computational Intelligence for Measurement Systems and Applications (CIMSA) Proceedings, Ottawa, ON, Canada, 2011, pp. 1-6. [5]. K. Feng and F. Yuan, "Static hand gesture recognition based on HOG characters and support vector machines," 2013 2nd International Symposium on Instrumentation and Measurement, Sensor Network and Automation (IMSNA), Toronto, ON, 2013, pp. 936-938. [6]. N. Dalal and B. Triggs, "Histograms of oriented gradients for human detection," 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), San Diego, CA, USA, 2005, pp. 886-893 vol. 1. [7]. https://www.learnopencv.com/histogram-of-oriented-gradients/ [8]. https://github.com/llSourcell/Classifying_Data_Using_a_Support_Vector_Machine/blob/master/support_vector_machine_lesson.ipynb Hand Gesture Recognition 396/21/2019 Introduction Hand Gesture Examples Related Studies Block Diagram Feature Extracting algorithm Datasets Results References

Editor's Notes

  • A gesture is a type of nonverbal communication in which a part of body conveys some information especially by head or the hand.
  • These are the examples of few hand gestures.
  • Many Reserachers have proposed numerous methods .
    First one is Glove based.
    Requires the use of gloves which contain sensors which senses our hand gestures.
    Variable glove size for different users.
    Second one is Markker based.
    We mark the color markers on finger.
    Marker colours are fixed. But it is complex to use multiple markers.
  • Third one is vision based system.
    Assembly uses camera and processing unit.
    No extra hardware required.
    But this is a complex problem due to high degree of freedom involved by h
  • These are the feature extraction algorithms which I would discuss.
  • This is the brief summary of training phase.
    Suppose we have four different gestures of training images.
    We extracting the PCA features adn take only M best PCA featues.
    Then we project the training data in to PCA feature space.i.e we projected the data into lower dimensional feature space.
  • Convert each image in training set to column vector.

    Stack the column vectors horizontally.
  • Calculate the average vector .
    Subtract the average vector from each of the vector in training matrix.
  • Calculate the covaricne matrix. Of normalized data.
    Here pi-ith is the ith eigen vector.
  • Mandeep’s method comprises of conversion of test image

    Naseer H.dardas used images contains cluttered backgrounds. i.e face,other background.
    So,firstly he use face detector to detect faces.
    After removing face pixels he applied face subtraction and then applied contour comparison and color based skin detection techanique.




    HSV against lightening conditions,illummincagtion variations.

  • This is just the summary of hand gesture recognition system using PCA By Naseer
  • Second feature are the HOG features
  • N.Dalal uses it for pdestrain detection
    K.Feng and F.Yuan uses it for hand gesture recognition.
  • Read the slide as it is.
  • Select image patch to be analyzed .
    Image patch should have aspect ratio of 1:2.
  • Read the slide as it is.
  • Read as it is
  • Let we have matric of gradient direction and gradient magnitude.
    We have 9 bins. 2-in gradient magnitude goes to 80 th bin.
    4 has angle of 10degree. So it is divided in to 0th and 20th bin. By this way we have hog features.
  • For making the vector intensity invariant ,divide it with its magnitude.
  • Suppose these are the data points. Points near to hyper plane act as support vectors.
    Distance between support vectors and the hyper plane is called margin.
    Actually SVM find the optimal line for the data which maximizes the margin.
  • ×