0
Ing. Matteo Valoriani               matteo.valoriani@studentpartner.comKINECT Programming
Gesture• What is a gesture?   • An action intended to communicate feelings or intentions• What is “Gesture Detection” or “...
Interaction metaphors   • Depend by the tasks   • Important aspect in design of UICursors (hands tracking):             Av...
The shadow/mirror effectShadow Effect:                              Mirror Effect:•   I see the back of my avatar         ...
KINECT Programming
IR Emitter          User Interaction Game mindset               ≠        UI mindsetChallenging = fun                Challe...
Gesture semantically fits user taskAbstract                 Meaningful                 KINECT Programming
User action fits UI reaction            1 2 3 4 5System’s UI feedback relates to the user’s physicalmovement              ...
User action fits UI reaction5 61 72 83 94 10               5System’s UI feedback relates to the user’s physicalmovement   ...
Gestures family-up            1 2 3 4 5Each gesture feels related and cohesivewith entire gesture set                     ...
Handed gestures            1 2 3 4 5Different gesture depending on hand: only left handcan do gesture A                   ...
Repeting Gesture?Will users want/need to perform the proposed gesturerepeatedly?                       KINECT Programming
Repeting Gesture?Will users want/need to perform the proposed gesturerepeatedly?                       KINECT Programming
Number of Hands         1 2 3 4 5One-handed gestures are preferred                    KINECT Programming
Symmetrical two-handed gestureTwo hand gesture should be symmetrical                    KINECT Programming
Gesture payoff          1 2 3 4 5Interactions requiring more work and effort shouldhave a higher payoff                   ...
Fatigue kills gesture Fatigue is the start of downward that kills gestureFatigue increase messiness  poor performance fr...
Gorilla Arm problemGorilla arm problem: try to put the hand up for 10minutes…                    KINECT Programming
Confortable positions      KINECT Programming
User PostureUser posture may affect design of a gesture                   KINECT Programming
The challenges•   Physical variable•   Environment•   Recognizing intent•   Input variability                     KINECT P...
KINECT Programming
Heuristics• Experience-based techniques for problem solving, learning, and  discovery                                Cost•...
Define What Constitutes a Gesture• Some players have more energy (or enthusiasm) than  others• Some players will “optimize...
Select the Right Triggers• Use skeleton view to analyze whole skeleton behavior• Use joint view to isolate and analyze spe...
Define Key Stages of a Gesture• Determine   • When the gesture begins   • When the gesture ends• Determine other key stage...
Determine the Type of Outcome• Definite gesture                   • Continuous gesture  • Contact or release              ...
Run a Detection Filter Only When Necessary •   Define clear context for when a gesture is expected •   Provide clear feedb...
Causes of Missing Information• Self Occlusion   • Side poses   • Player’s position in play space• Obstacles   • Other play...
KINECT Programming
class GestureRecognizer    {          public Dictionary<JointType, List<Joint>> skeletonSerie = new Dictionary<JointType, ...
const int bufferLenght=10;public void Recognize(JointCollection jointCollection, DateTime date)    {            timeList.A...
Boolean isHOHRecognitionStarted;DateTime StartTimeHOH = DateTime.Now;private Gesture HandOnHeadReconizerRT (JointType hand...
How to notify a gesture?• Synchronous Solution:    • Return gesturesList to GUI• Asynchronous Solution:    • Use Eventpubl...
KINECT Programming
const   float SwipeMinimalLength = 0.08f;constconst        float SwipeMaximalHeight = 0.02f;        int SwipeMinimalDurati...
public delegate void SwipeHadler(object sender, GestureEventArgs e);public event SwipeHadler Swipe;private Gesture Horizzo...
Performance• Skeleton processing is an expensive operation.• Use VS2010 Performance Tool                           KINECT ...
KINECT Programming
Pros & ConsPROs• Easy to understand• Easy to implement (for simple gestures)• Easy to debug                               ...
KINECT Programming
Gesture DefinitionDefine gesture as weighted network• Simple neural network• Simple algorithmic gestures as input nodes• U...
Abstract Neuron     x1           1x2        2                                     f (i 1 ixi )                       ...
Perceptron• Simple network using weighted threshold elements    P1               1  P2      2                          ...
Example   HandAboveElbow AND HandInFrontOfShoulder Hand.y             HandAboveElbow                                      ...
Example   HandAboveElbow OR HandInFrontOfShoulder Hand.y             HandAboveElbow                                       ...
Network Definition for Detector• Similar to perceptron• Normalize using weights• Use probabilities, not Booleans   P1     ...
Surely This Will Suffice?    HeadAboveBaseLine                                 0.3    LeftKneeAboveBaseLine        0.1    ...
And We’re Done!                  HeadAboveBaseLine           0.3              LeftKneeAboveBaseLine           0.1         ...
But Wait, If We Know For Sure…              HeadAboveBaseLine      0.3          HeadFarAboveBaseLine                      ...
Implementation Overview•   Update height baseline values•   Update input nodes, i.e. algorithmic gestures•   Evaluate each...
Pros• Neural networks well understood    • Introduced in 1940’s• Learning algorithm can be used to find optimum    • Param...
Cons• Lots of parameters, weights, and thresholds  • Small changes can have dramatic changes in results  • Very time consu...
Recommendation• Use for more complex gestures  • Jump, duck, punch• Break complex gestures into collection of simple  gest...
KINECT Programming
Gesture Definition• Define gesture as pre-recorded animations  • Motion capture animations  • Record different people doin...
Exemplar• Definition: ideal example to compare against• Pre-recorded animations are exemplars                   KINECT Pro...
Exemplar Matching• Need to compare skeleton frames  • Define error metric for skeleton  • Angular difference for each join...
Exemplar Matching• Search for best matching frames  • Best matching frame has strongest signal  • Different classifiers ca...
Exemplar Matching25201510                                                PSNR50     1   2   3   4       5            6   7...
Pros•   Works well for context-sensitive gesture detection•   Works well for animation blending•   Very complex gestures c...
Cons• Requires lots of resources to be robust  • Multiple recordings of multiple people for one    gesture     • i.e. requ...
Example• 10 Gestures, 10 People, 5 times = 500 Exemplars  • K-Nearest         180       • 46 ms        160  •   DTW       ...
Recommendation• Use for context-sensitive gesture detection• Use for complex gestures   • Dancing, fitness exercises• Use ...
KINECT Programming
Building Great Gesture Detection           Data Collection            Development               Testing           KINECT P...
Data Collection                                                                     Jump                                  ...
DevelopmentPhase 1 – Exemplar Data      Tagged GesturePhase 2 – Sequence Data        RecordingsPhase 3 – General Data    F...
Testing               Live Camera                  Tagged Gesture                  Stream                      Recordings ...
Takeaways• A system, not just a detector   • Detector is small component   • Invest equally in other components• Manage da...
References•   “A Brief History of Human Computer Interaction Technology” – Brad A. Myers•   “Neural Networks – A Systemati...
Upcoming SlideShare
Loading in...5
×

5 track kinect@Bicocca - gesture

782

Published on

Published in: Technology, Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
782
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
48
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "5 track kinect@Bicocca - gesture"

  1. 1. Ing. Matteo Valoriani matteo.valoriani@studentpartner.comKINECT Programming
  2. 2. Gesture• What is a gesture? • An action intended to communicate feelings or intentions• What is “Gesture Detection” or “Gesture Recognition”? • Computer’s ability to understand human gestures as input • First used in 1963 with pen-based input device• What is it used for? • Mouse movements, Handwriting recognition, Sign language, recognition, Touch screen input, Kinect KINECT Programming
  3. 3. Interaction metaphors • Depend by the tasks • Important aspect in design of UICursors (hands tracking): Avatars (body tracking):Target an object Interaction with virtual space KINECT Programming
  4. 4. The shadow/mirror effectShadow Effect: Mirror Effect:• I see the back of my avatar • I see the front of my avatar• Problems with Z movements • Problem with mapping left/right movements KINECT Programming
  5. 5. KINECT Programming
  6. 6. IR Emitter User Interaction Game mindset ≠ UI mindsetChallenging = fun Challenging = easy and effective KINECT Programming
  7. 7. Gesture semantically fits user taskAbstract Meaningful KINECT Programming
  8. 8. User action fits UI reaction 1 2 3 4 5System’s UI feedback relates to the user’s physicalmovement KINECT Programming
  9. 9. User action fits UI reaction5 61 72 83 94 10 5System’s UI feedback relates to the user’s physicalmovement KINECT Programming
  10. 10. Gestures family-up 1 2 3 4 5Each gesture feels related and cohesivewith entire gesture set KINECT Programming
  11. 11. Handed gestures 1 2 3 4 5Different gesture depending on hand: only left handcan do gesture A KINECT Programming
  12. 12. Repeting Gesture?Will users want/need to perform the proposed gesturerepeatedly? KINECT Programming
  13. 13. Repeting Gesture?Will users want/need to perform the proposed gesturerepeatedly? KINECT Programming
  14. 14. Number of Hands 1 2 3 4 5One-handed gestures are preferred KINECT Programming
  15. 15. Symmetrical two-handed gestureTwo hand gesture should be symmetrical KINECT Programming
  16. 16. Gesture payoff 1 2 3 4 5Interactions requiring more work and effort shouldhave a higher payoff KINECT Programming
  17. 17. Fatigue kills gesture Fatigue is the start of downward that kills gestureFatigue increase messiness  poor performance frustration  bad UX KINECT Programming
  18. 18. Gorilla Arm problemGorilla arm problem: try to put the hand up for 10minutes… KINECT Programming
  19. 19. Confortable positions KINECT Programming
  20. 20. User PostureUser posture may affect design of a gesture KINECT Programming
  21. 21. The challenges• Physical variable• Environment• Recognizing intent• Input variability KINECT Programming
  22. 22. KINECT Programming
  23. 23. Heuristics• Experience-based techniques for problem solving, learning, and discovery Cost• Cost effective• Helps reconstruct missing information• Helps compute outcome of a gesture Gesture Heuristics Machine Learning Complexity KINECT Programming
  24. 24. Define What Constitutes a Gesture• Some players have more energy (or enthusiasm) than others• Some players will “optimize” their gestures• Most players will not perform the gesture precisely as intended KINECT Programming
  25. 25. Select the Right Triggers• Use skeleton view to analyze whole skeleton behavior• Use joint view to isolate and analyze specific joints and axis behavior• Use data sheet view: to get the real numbers• Not all joints are needed• Player location in the play area can cause some joints to become occluded KINECT Programming
  26. 26. Define Key Stages of a Gesture• Determine • When the gesture begins • When the gesture ends• Determine other key stages • Changes in motion direction • Pauses • …• You could simply signal that the gesture has been completed, or• You could keep a progress, or• You could use distinct states KINECT Programming
  27. 27. Determine the Type of Outcome• Definite gesture • Continuous gesture • Contact or release • Frequency point • Amplitude • Direction • Initial velocity KINECT Programming
  28. 28. Run a Detection Filter Only When Necessary • Define clear context for when a gesture is expected • Provide clear feedback to the player • Run the gesture filter when the context warrants it • Cancel the gesture if context changes KINECT Programming
  29. 29. Causes of Missing Information• Self Occlusion • Side poses • Player’s position in play space• Obstacles • Other players • Furniture• Outside the camera’s field of view • Left or right (easy to fix) • Top or bottom (hard to avoid) KINECT Programming
  30. 30. KINECT Programming
  31. 31. class GestureRecognizer { public Dictionary<JointType, List<Joint>> skeletonSerie = new Dictionary<JointType, List<Joint>>() { { JointType.AnkleLeft, new List<Joint>()}, { JointType.AnkleRight, new List<Joint>()}, { JointType.ElbowLeft, new List<Joint>()}, { JointType.ElbowRight, new List<Joint>()}, { JointType.FootLeft, new List<Joint>()}, { JointType.FootRight, new List<Joint>()}, { JointType.HandLeft, new List<Joint>()}, { JointType.HandRight, new List<Joint>()}, { JointType.Head, new List<Joint>()}, { JointType.HipCenter, new List<Joint>()}, { JointType.HipLeft, new List<Joint>()}, { JointType.HipRight, new List<Joint>()}, { JointType.KneeLeft, new List<Joint>()}, { JointType.KneeRight, new List<Joint>()}, { JointType.ShoulderCenter, new List<Joint>()}, { JointType.ShoulderLeft, new List<Joint>()}, { JointType.ShoulderRight, new List<Joint>()}, { JointType.Spine, new List<Joint>()}, Key Value { JointType.WristLeft, new List<Joint>()}, { JointType.WristRight, new List<Joint>()} AnkleLeft <Vt1, Vt2, Vt3, Vt4,..> }; AnkleRight <V , V , V , V ,..> t1 t2 t3 t4 protected List<DateTime> timeList; ElbowLeft <Vt1, Vt2, Vt3, Vt4,..> private static List<JointType> typesList = new List<JointType>() {JointType.AnkleLeft, JointType.AnkleRight,JointType.ElbowLeft, JointType.ElbowRight, JointType.FootLeft, JointType.FootRight, JointType.HandLeft,JointType.HandRight, JointType.Head, JointType.HipCenter, JointType.HipLeft, JointType.HipRight, JointType.KneeLeft,JointType.KneeRight, JointType.ShoulderCenter, JointType.ShoulderLeft, JointType.ShoulderRight, JointType.Spine,JointType.WristLeft, JointType.WristRight }; //... continue} KINECT Programming
  32. 32. const int bufferLenght=10;public void Recognize(JointCollection jointCollection, DateTime date) { timeList.Add(date); foreach (JointType type in typesList) { skeletonSerie[type].Add(jointCollection[type]); if (skeletonSerie[type].Count > bufferLenght) { skeletonSerie[type].RemoveAt(0); } } startRecognition(); }List<Gesture> gesturesList = new List<Gesture>();private void startRecognition() { gesturesList.Clear(); gesturesList.Add(HandOnHeadReconizerRT(JointType.HandLeft, JointType.ShoulderLeft)); // Do ... } KINECT Programming
  33. 33. Boolean isHOHRecognitionStarted;DateTime StartTimeHOH = DateTime.Now;private Gesture HandOnHeadReconizerRT (JointType hand, JointType shoulder) { // Correct Position if (skeletonSerie[hand].Last().Position.Y > skeletonSerie[shoulder].Last().Position.Y + 0.2f) { if (!isHOHRecognitionStarted) { isHOHRecognitionStarted = true; StartTimeHOH = timeList.Last(); } else { double totalMilliseconds = (timeList.Last() - StartTimeHOH).TotalMilliseconds; // time ok? if ((totalMilliseconds >= HandOnHeadMinimalDuration)) { isHOHRecognitionStarted = false; return Gesture.HandOnHead; Alternative: count } number of } occurrences } else {//Incorrect Position if (isHOHRecognitionStarted) { isHOHRecognitionStarted = false; } } return Gesture.None; } KINECT Programming
  34. 34. How to notify a gesture?• Synchronous Solution: • Return gesturesList to GUI• Asynchronous Solution: • Use Eventpublic delegate void HandOnHeadHadler(object sender, EventArgs e);public event HandOnHeadHadler HandOnHead;private Gesture HandOnHeadReconizerRTWithEvent(JointType hand, JointType shoulder) { Gesture g = HandOnHeadReconizerRT(hand, shoulder); if (g == Gesture.HandOnHead) { if (HandOnHead != null) HandOnHead(this, EventArgs.Empty); } return g;} KINECT Programming
  35. 35. KINECT Programming
  36. 36. const float SwipeMinimalLength = 0.08f;constconst float SwipeMaximalHeight = 0.02f; int SwipeMinimalDuration = 200; ∆x too small or ∆y tooconstconst int SwipeMaximalDuration = 1000; int MinimalPeriodBetweenGestures = 0; big  shift startprivate Gesture HorizzontalSwipeRecognizer(List<Joint> positionList) { int start = 0; ∆x > minimal lenght for (int index = 0; index < positionList.Count - 1; index++) { if ((Math.Abs(positionList[0].Position.Y - positionList[index].Position.Y) > SwipeMaximalHeight) || Math.Abs((positionList[index].Position.X - positionList[index + 1].Position.X)) < 0.01f) { start = index; } ∆t in the accepted range if ((Math.Abs(positionList[index].Position.X - positionList[start].Position.X) > SwipeMinimalLength)) { double totalMilliseconds = (timeList[index] - timeList[start]).TotalMilliseconds; if (totalMilliseconds >= SwipeMinimalDuration && totalMilliseconds <= SwipeMaximalDurati { if (DateTime.Now.Subtract(lastGestureDate).TotalMilliseconds > MinimalPeriodBetweenGestures){ lastGestureDate = DateTime.Now; if (positionList[index].Position.X - positionList[start].Position.X < 0) return Gesture.SwipeRightToLeft; else return Gesture.SwipeLeftToRight; } } } } return Gesture.None; } KINECT Programming
  37. 37. public delegate void SwipeHadler(object sender, GestureEventArgs e);public event SwipeHadler Swipe;private Gesture HorizzontalSwipeRecognizer(JointType jointType) { Gesture g = HorizzontalSwipeRecognizer(skeletonSerie[jointType]); switch (g) { case Gesture.None: break; case Gesture.SwipeLeftToRight: if (Swipe != null) Swipe(this, new GestureEventArgs("SwipeLeftToRight")); break; case Gesture.SwipeRightToLeft: if (Swipe != null) Swipe(this, new GestureEventArgs("SwipeRightToLeft")); break; default: break; } return g; }... Personalized EventArgs public class GestureEventArgs : EventArgs { public string text; public GestureEventArgs(string text) { this.text = text; } } KINECT Programming
  38. 38. Performance• Skeleton processing is an expensive operation.• Use VS2010 Performance Tool KINECT Programming
  39. 39. KINECT Programming
  40. 40. Pros & ConsPROs• Easy to understand• Easy to implement (for simple gestures)• Easy to debug Recommendation Use for simple gesturesCONs • Hand wave• Challenging to choose best values for parameters • Head movement• Doesn’t scale well for variants of same gesture• Gets challenging for complex gestures• Challenging to compensate for latency KINECT Programming
  41. 41. KINECT Programming
  42. 42. Gesture DefinitionDefine gesture as weighted network• Simple neural network• Simple algorithmic gestures as input nodes• Use fuzzy logic, i.e. probabilities, not Booleans HeadAboveBaseLine 1 LeftKneeAboveBaseLine 2  Jump? 3 RightKneeAboveBaseLine KINECT Programming
  43. 43. Abstract Neuron x1 1x2 2 f (i 1 ixi ) n f n xn KINECT Programming
  44. 44. Perceptron• Simple network using weighted threshold elements P1 1 P2 2   iPi   n i 1 n Pn KINECT Programming
  45. 45. Example HandAboveElbow AND HandInFrontOfShoulder Hand.y HandAboveElbow 1 Elbow.y (HandAboveElbow * 1) + 2 (HandInFrontOfShoulder * 1) >= 2 1 Hand.z HandInFrontOfShoulderShoulder.z KINECT Programming
  46. 46. Example HandAboveElbow OR HandInFrontOfShoulder Hand.y HandAboveElbow 1 Elbow.y (HandAboveElbow * 1) + 1 (HandInFrontOfShoulder * 1) >= 1 1 Hand.z HandInFrontOfShoulderShoulder.z KINECT Programming
  47. 47. Network Definition for Detector• Similar to perceptron• Normalize using weights• Use probabilities, not Booleans P1 1 P2 2  iPi n  i 1    i n n i 1 Pn KINECT Programming
  48. 48. Surely This Will Suffice? HeadAboveBaseLine 0.3 LeftKneeAboveBaseLine 0.1 0.1 0.8 Jump? RightKneeAboveBaseLine 0.5 LegsStraightPreviouslyBent• But due to noise, still many false positives• How can we reduce false positives? KINECT Programming
  49. 49. And We’re Done! HeadAboveBaseLine 0.3 LeftKneeAboveBaseLine 0.1 RightKneeAboveBaseLine 0.1 0.8 LegsStraightPreviouslyBent 0.5 1 HeadBelowBaseLine 2 Jump? 1 AND LeftKneeBelowBaseLine 1 OR NOT 1RightKneeBelowBaseLine 1 -1 1 0 LeftAnkleBelowBaseLine 1 1RightAnkleBelowBaseLine 1 BodyFaceUpwards KINECT Programming
  50. 50. But Wait, If We Know For Sure… HeadAboveBaseLine 0.3 HeadFarAboveBaseLine 0.1 1 LeftKneeAboveBaseLine RightKneeAboveBaseLine 0.1 0.8 OR 1 LegsStraightPreviouslyBent 0.5 1 Jump? HeadBelowBaseLine 2 1 AND LeftKneeBelowBaseLine 1 1 OR NOTRightKneeBelowBaseLine 1 -1 1 0 LeftAnkleBelowBaseLine 1 1RightAnkleBelowBaseLine 1 BodyFaceUpwards KINECT Programming
  51. 51. Implementation Overview• Update height baseline values• Update input nodes, i.e. algorithmic gestures• Evaluate each node in network• Calculate probability of gesture KINECT Programming
  52. 52. Pros• Neural networks well understood • Introduced in 1940’s• Learning algorithm can be used to find optimum • Parameters, weights, and thresholds• Complex gestures can be detected• Scale well for variants of same gesture• Nodes can be reused in different gestures• Easy to visualize as node graph• Good CPU performance • 0.095 ms to execute Jump Detector KINECT Programming
  53. 53. Cons• Lots of parameters, weights, and thresholds • Small changes can have dramatic changes in results • Very time consuming to choose manually• Not easy to debug • Is the code wrong or are parameters not optimal• Challenging to compensate for latency KINECT Programming
  54. 54. Recommendation• Use for more complex gestures • Jump, duck, punch• Break complex gestures into collection of simple gestures• Use learning algorithm• Debug visualization is essential KINECT Programming
  55. 55. KINECT Programming
  56. 56. Gesture Definition• Define gesture as pre-recorded animations • Motion capture animations • Record different people doing same gesture • Each person doing same gesture multiple times KINECT Programming
  57. 57. Exemplar• Definition: ideal example to compare against• Pre-recorded animations are exemplars KINECT Programming
  58. 58. Exemplar Matching• Need to compare skeleton frames • Define error metric for skeleton • Angular difference for each joint in local space • Peak Signal to Noise Ratio for whole skeleton 1 MSE   Distancei2 0.3 N PSNR  10 * log10 ( MAX 2 / MSE ) KINECT Programming
  59. 59. Exemplar Matching• Search for best matching frames • Best matching frame has strongest signal • Different classifiers can be used • K-Nearest • Dynamic Time Warping (DTW) • Hidden Markov Models (HMM) KINECT Programming
  60. 60. Exemplar Matching25201510 PSNR50 1 2 3 4 5 6 7 8 KINECT Programming
  61. 61. Pros• Works well for context-sensitive gesture detection• Works well for animation blending• Very complex gestures can be detected• DTW allows for different speeds• Can compensate for latency• Can scale for variants of same gesture • Just need more resources• Easy to visualize exemplar matching KINECT Programming
  62. 62. Cons• Requires lots of resources to be robust • Multiple recordings of multiple people for one gesture • i.e. requires lots of CPU and memory • K-Nearest • 1.5 ms for 16 exemplar matches • DTW • 5 ms for 16 exemplar matches KINECT Programming
  63. 63. Example• 10 Gestures, 10 People, 5 times = 500 Exemplars • K-Nearest 180 • 46 ms 160 • DTW 140 K-Nearest 120 • 156 ms 100 DTW • Weighted network 80 • 1 ms 60 Weighted 40 Network 20 0 KINECT Programming
  64. 64. Recommendation• Use for context-sensitive gesture detection• Use for complex gestures • Dancing, fitness exercises• Use when reducing latency is critical• Optimize by reducing exemplar matches • Preprocess exemplar data with key frames • Use context of game • Use another fast method first• Implement debug visualization KINECT Programming
  65. 65. KINECT Programming
  66. 66. Building Great Gesture Detection Data Collection Development Testing KINECT Programming
  67. 67. Data Collection Jump Identify Gestures PunchAt least depth & skeleton 1. Exemplar 2. Sequence of same gesture Record Gestures 3. General (actual game play)Old, young, male, female,overweight, handedness Meta data per recording, tag Tag Gesture Recordings start/stop events for each gestureUse custom tool,or export to Excel Someone other than tagger Verify Gesture Tagging should verify correctness Backup & Share KINECT Programming
  68. 68. DevelopmentPhase 1 – Exemplar Data Tagged GesturePhase 2 – Sequence Data RecordingsPhase 3 – General Data Filter Joints Normalize Skeleton Parameters Gesture Debug Weights Detector Visualization Thresholds Result Verification Machine Learning Algorithm Error KINECT Programming
  69. 69. Testing Live Camera Tagged Gesture Stream Recordings Filter Joints Normalize SkeletonParameters Weights GestureThresholds Detector Human Verification ResultVerification Feels No Data Error Robust? Collection KINECT Programming
  70. 70. Takeaways• A system, not just a detector • Detector is small component • Invest equally in other components• Manage data • You’ll have lots of it! • Most valuable component • Tagging correctly is essential • Collect real user data KINECT Programming
  71. 71. References• “A Brief History of Human Computer Interaction Technology” – Brad A. Myers• “Neural Networks – A Systematic Introduction” – Raúl Rojas• “A Gesture Processing Framework for Multimodal Interaction in Virtual Reality” – Marc E. Latoschik• Gamefest 2010 – “Gesture Recognition” – Lewey Geselowitz & J. McBride• Kinect Developer Summit 2011 – “Inside Kinect Skeletal Tracking Deep Dive” – Zsolt Mathe KINECT Programming
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×