Defender: Javier Barbadillo Amor Information and Communication Theory (ICT) Group Delft University of Technology Committee...
Outline : <ul><li>Introduction </li></ul><ul><ul><li>Single Person Pose Recognition and Tracking System </li></ul></ul><ul...
Introduction
Single Person Pose Recognition and Tracking System <ul><li>Real time  </li></ul><ul><li>One single camera </li></ul><ul><l...
Theory: Background Subtraction by Mixture of Gaussians  <ul><li>Compare the current frame with a model of the background. ...
Theory: Background Subtraction by Mixture of Gaussians  <ul><li>History of pixel intensity values: </li></ul><ul><li>An in...
Theory: Particle Filter for tracking the torso and head <ul><li>Torso and head region detection </li></ul><ul><li>Hand Det...
Theory: Particle filter for tracking torso and head <ul><li>N particles are generated </li></ul><ul><li>Weights  π  assign...
Theory: Particle filter for tracking torso and head <ul><li>Sample sets of particles are generated for 3 states: x, y and ...
Theory: General skin color detection <ul><li>Hand Detection with general skin color model </li></ul>
<ul><li>Relative distances between hands and torso center. </li></ul><ul><li>Angles of the hands with the torso center. </...
<ul><li>Incoming observations = 6-feature-set </li></ul><ul><li>Classifier decides one Pose class. </li></ul><ul><li>Each ...
The goal of this research <ul><li>Improve the system performance </li></ul><ul><ul><li>Hand detection: fails for short sle...
Experiments and Results
<ul><li>Skin color detection combined with human blob silhouette for hand detection </li></ul><ul><li>Preliminary hand pos...
Skin color detection combined with human blob silhouette for hand detection <ul><li>We check if the blob is: </li></ul><ul...
Skin color detection combined with human blob silhouette for hand detection <ul><li>Al the cases where people are wearing ...
<ul><li>Non-Pose classification </li></ul>DEFINITION: Everything different from a predefined Pose More features Needed!
<ul><li>Clear Non-Pose: poses where one or both hand positions are in between positions corresponding to Poses </li></ul>N...
Non-Pose classification <ul><li>17 videos from 17 different people were recorded. </li></ul><ul><li>Features extracted fro...
Non-Pose classification Initial Dataset Improved Dataset
Non-Pose classification <ul><li>Experiments with Initial Dataset </li></ul>First approach: Second approach:
<ul><li>“ Leave one person out” method for realistic results. </li></ul><ul><li>Tested on Parzen, K-Nearest-Neighbor and G...
<ul><li>Mean error from LOPO: errors from all people summed and divided by the number of people. </li></ul><ul><li>Best re...
K-Nearest-Neighbor
<ul><li>Parzen classifier shows more interesting results for a particular person (Hasan). </li></ul>Non-Pose classificatio...
<ul><li>Two correct ways of performing the same Pose result in quite different features. </li></ul><ul><li>Errors in Parze...
Non-Pose classification: Initial Dataset 10-NN 5-NN <ul><li>All the samples from Carmen are missclassified as Non-Poses. <...
<ul><li>Results are much better with this approach than with the cascade. Single Pose classes seem to be better modelled t...
<ul><li>Experiments with Improved Dataset </li></ul><ul><ul><li>All classes have more than 100 samples </li></ul></ul>Non-...
<ul><li>Mean errors for detector trained on Non-Poses. </li></ul>Non-Pose classification: Improved Dataset <ul><li>Trainin...
Non-Pose classification <ul><li>Now, Carmen´s samples of Pose 3 are correctly detected as Poses. </li></ul><ul><li>Pose 3 ...
Non-Pose classification Decreased from 2% to 0% Increased from 0% to 1%!!! <ul><li>More samples of poses 3 and 4 improved ...
Conclusions <ul><li>The Improved hand detection is a simple method but robust, and solves the problem of wrong detection f...
Future Work <ul><li>Make a new Dataset with the improved hand detection. </li></ul><ul><li>Add a new feature for detecting...
I appreciate your attention  Questions?
Initial Dataset
Improved Dataset
Spatial Game
Upcoming SlideShare
Loading in …5
×

Single person pose recognition and tracking

1,858 views

Published on

Slides for my Master Thesis defense at TUDelft.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,858
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
30
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Single person pose recognition and tracking

  1. 1. Defender: Javier Barbadillo Amor Information and Communication Theory (ICT) Group Delft University of Technology Committee: Dr. Alan Hanjalic Dr. Emile. A. Hendriks PhD. Feifei Huo Dr. Pavel Paclik Single Person Pose Recognition and Tracking 25-06-2010
  2. 2.
  3. 3. Outline : <ul><li>Introduction </li></ul><ul><ul><li>Single Person Pose Recognition and Tracking System </li></ul></ul><ul><ul><li>Theory </li></ul></ul><ul><li>The goal of this research </li></ul><ul><ul><li>Improve Body Parts Detection and Pose recognition </li></ul></ul><ul><li>Experiments and Results </li></ul><ul><ul><li>Improved Hand Detection </li></ul></ul><ul><ul><li>Detection of a new class: Non-Pose Classification </li></ul></ul><ul><li>Conclusions </li></ul><ul><li>Future Work </li></ul>
  4. 4. Introduction
  5. 5. Single Person Pose Recognition and Tracking System <ul><li>Real time </li></ul><ul><li>One single camera </li></ul><ul><li>Game control with detected poses </li></ul>
  6. 6. Theory: Background Subtraction by Mixture of Gaussians <ul><li>Compare the current frame with a model of the background. </li></ul><ul><li>Obtain a binary image with the foreground pixels </li></ul>
  7. 7. Theory: Background Subtraction by Mixture of Gaussians <ul><li>History of pixel intensity values: </li></ul><ul><li>An intensity value belongs to a Gaussian Distribution if it is within [-2.5σ, 2.5σ] </li></ul><ul><li>Each pixel is modeled by K Gaussians. </li></ul>
  8. 8. Theory: Particle Filter for tracking the torso and head <ul><li>Torso and head region detection </li></ul><ul><li>Hand Detection </li></ul>
  9. 9. Theory: Particle filter for tracking torso and head <ul><li>N particles are generated </li></ul><ul><li>Weights π assigned according to measured probability. </li></ul><ul><li>Father particles spread into G sons </li></ul>
  10. 10. Theory: Particle filter for tracking torso and head <ul><li>Sample sets of particles are generated for 3 states: x, y and Scale </li></ul><ul><li>The probability of the state of the torso is given by </li></ul>Primitive for torso and head
  11. 11. Theory: General skin color detection <ul><li>Hand Detection with general skin color model </li></ul>
  12. 12. <ul><li>Relative distances between hands and torso center. </li></ul><ul><li>Angles of the hands with the torso center. </li></ul><ul><li>r , l and t stand for right , left and torso. </li></ul>Theory: Feature extraction
  13. 13. <ul><li>Incoming observations = 6-feature-set </li></ul><ul><li>Classifier decides one Pose class. </li></ul><ul><li>Each Pose number is a different action in the game </li></ul>Theory: Pose Classification
  14. 14. The goal of this research <ul><li>Improve the system performance </li></ul><ul><ul><li>Hand detection: fails for short sleeves and “skin color clothes” </li></ul></ul><ul><ul><li>Pose recognition: detect Non-Poses </li></ul></ul>Hands detected in the forearm The 9 Predefined Poses
  15. 15. Experiments and Results
  16. 16. <ul><li>Skin color detection combined with human blob silhouette for hand detection </li></ul><ul><li>Preliminary hand position is obtained from the center of gravity of the biggest skin blobs. </li></ul><ul><li>First, general skin color detection is applied using this mask: </li></ul>
  17. 17. Skin color detection combined with human blob silhouette for hand detection <ul><li>We check if the blob is: </li></ul><ul><li>Below the heep </li></ul><ul><li>Between the heep and the shoulder </li></ul><ul><li>Over the shoulder </li></ul>
  18. 18. Skin color detection combined with human blob silhouette for hand detection <ul><li>Al the cases where people are wearing short sleeves or “skin color clothes” are correct now. </li></ul>
  19. 19. <ul><li>Non-Pose classification </li></ul>DEFINITION: Everything different from a predefined Pose More features Needed!
  20. 20. <ul><li>Clear Non-Pose: poses where one or both hand positions are in between positions corresponding to Poses </li></ul>Non-Pose classification
  21. 21. Non-Pose classification <ul><li>17 videos from 17 different people were recorded. </li></ul><ul><li>Features extracted from each frame by processing the videos. </li></ul><ul><li>Multiple labeling with PRSD Studio, Matlab. </li></ul>
  22. 22. Non-Pose classification Initial Dataset Improved Dataset
  23. 23. Non-Pose classification <ul><li>Experiments with Initial Dataset </li></ul>First approach: Second approach:
  24. 24. <ul><li>“ Leave one person out” method for realistic results. </li></ul><ul><li>Tested on Parzen, K-Nearest-Neighbor and Gaussian classifiers. </li></ul>Non-Pose classification: Initial Dataset ROC curve
  25. 25. <ul><li>Mean error from LOPO: errors from all people summed and divided by the number of people. </li></ul><ul><li>Best results detecting Poses are for K-Nearest-Neighbor, in general. </li></ul><ul><li>Parzen and Gaussian are considaribily worse. </li></ul>Non-Pose classification: Initial Dataset
  26. 26. K-Nearest-Neighbor
  27. 27. <ul><li>Parzen classifier shows more interesting results for a particular person (Hasan). </li></ul>Non-Pose classification: Initial Dataset Results for Hasan´s samples as Test, from a single experiment. <ul><li>Pose 9 has 404 samples in total. </li></ul><ul><ul><li>-120 from Hasan (Test) </li></ul></ul><ul><ul><li>-171 from Saleem (Training) </li></ul></ul>Is there any relation?
  28. 28. <ul><li>Two correct ways of performing the same Pose result in quite different features. </li></ul><ul><li>Errors in Parzen give us an idea on how to improve even more K-NN performance. </li></ul>Non-Pose classification: Initial Dataset
  29. 29. Non-Pose classification: Initial Dataset 10-NN 5-NN <ul><li>All the samples from Carmen are missclassified as Non-Poses. </li></ul>Cascade of detector and classifier
  30. 30. <ul><li>Results are much better with this approach than with the cascade. Single Pose classes seem to be better modelled than the whole Poses class with K-Nearest-Neighbor. </li></ul>Non-Pose classification: Initial Dataset Second approach:
  31. 31. <ul><li>Experiments with Improved Dataset </li></ul><ul><ul><li>All classes have more than 100 samples </li></ul></ul>Non-Pose classification: Improved Dataset <ul><li>For 10-NN the error on Poses decreased 1.5% and the error on Non-Poses decreased 3%. </li></ul><ul><li>Having more samples from singles Poses makes the whole Poses class more robust. </li></ul>Mean errors for the detector trained on Poses.
  32. 32. <ul><li>Mean errors for detector trained on Non-Poses. </li></ul>Non-Pose classification: Improved Dataset <ul><li>Training on Non-Poses doesn´t improve detection. </li></ul><ul><li>Non-Poses are more difficult to model than Poses. </li></ul>
  33. 33. Non-Pose classification <ul><li>Now, Carmen´s samples of Pose 3 are correctly detected as Poses. </li></ul><ul><li>Pose 3 class is more compact. </li></ul>Initial Dataset Improved Dataset
  34. 34. Non-Pose classification Decreased from 2% to 0% Increased from 0% to 1%!!! <ul><li>More samples of poses 3 and 4 improved Detection on Poses and Non-Poses detection, but didn´t improve classification of the Pose classes. </li></ul>
  35. 35. Conclusions <ul><li>The Improved hand detection is a simple method but robust, and solves the problem of wrong detection for short sleeves. </li></ul><ul><li>Non-Pose class is difficult to model because it overlaps with Poses and it is not a compact class. Anyway, almost 80% of Non-Poses can be detected. </li></ul><ul><li>Having a good dataset might improve results drastically. </li></ul><ul><ul><li>Samples must represent different people and ways of performing poses </li></ul></ul><ul><ul><li>Samples of wrong hand detections increase the error rate </li></ul></ul><ul><li>The K-Nearest-Neighbor is the best method for modelleing this Pose classes. </li></ul><ul><li>The more restrictive the system is, the better results will be: Comprimise Solution </li></ul>
  36. 36. Future Work <ul><li>Make a new Dataset with the improved hand detection. </li></ul><ul><li>Add a new feature for detecting more Non-Poses, e.g., face detection. </li></ul><ul><li>Elbow detection. </li></ul>
  37. 37. I appreciate your attention Questions?
  38. 38. Initial Dataset
  39. 39. Improved Dataset
  40. 40. Spatial Game

×