Gesture Based Computing

4,670 views

Published on

Talk for Infolab21 Brunch Bytes series on 'Gesture Based Computing' covering both touch and free form gestures.

0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
4,670
On SlideShare
0
From Embeds
0
Number of Embeds
583
Actions
Shares
0
Downloads
79
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Gesture Based Computing

  1. 1. GESTURE BASED COMPUTING Paul CoultonBanksy
  2. 2. GESTURE BASED COMPUTING Paul Coulton O’Sullivan and IgoeBanksy
  3. 3. GESTURAL INTERFACES Allows physical movements to be detected by a digital systemDan Saffer without the aid of a traditional pointing device.
  4. 4. A wave, a head nod, a touch, a toe tap, or even a raised eyebrow could be a gesture GESTURAL INTERFACES Allows physical movements to be detected by a digital systemDan Saffer without the aid of a traditional pointing device.
  5. 5. TANGIBLE INTERFACESProviding a physical form of digital information and facilitates the direct manipulation of the associated bits
  6. 6. TextJesper Juul MIMETIC INTERFACES Mimetic interfaces require players to perform actions that closely resemble the physical activity required in reality.
  7. 7. NATURAL USER INTERFACES “natural refers to the user’sbehaviour and feeling during the experience rather than being the product of some organic process” Wigdor and Wixon and indeed suggest a natural experience “is NOT best achieved through mimicry”
  8. 8. TYPES OF GESTURAL INTERFACE Most gestural interfaces can be categorised as either touchscreen or free form. Touchscreens require the user to be touching device directly whereas freeform systems do not.Dan Saffer
  9. 9. DIRECT VS INDIRECT MANIPULATION
  10. 10. Direct manipulation features a natural representation of task objects and actions promoting the notion of people performing a task themselves (directly) not through an intermediary.DIRECT VS INDIRECT MANIPULATION
  11. 11. MICE VS FINGERSDan Saffer
  12. 12. A cursor is often unnecessary since the user is not constantly pointing at something. Finger moves from point to point whereas mouse makes a trail. MICE VS FINGERSDan Saffer
  13. 13. A cursor is often unnecessary Hovers and mouse over events since the user is not constantly are not employed as this cannot pointing at something. Finger be detected through touch moves from point to point screens. whereas mouse makes a trail. MICE VS FINGERSDan Saffer
  14. 14. A cursor is often unnecessary Hovers and mouse over events since the user is not constantly are not employed as this cannot pointing at something. Finger be detected through touch moves from point to point screens. whereas mouse makes a trail. Double click can be done but should be used with caution. A threshold has to be set duringwhich two touch events at same location count a double click. MICE VS FINGERSDan Saffer
  15. 15. A cursor is often unnecessary Hovers and mouse over events since the user is not constantly are not employed as this cannot pointing at something. Finger be detected through touch moves from point to point screens. whereas mouse makes a trail. Double click can be done but In general gesture interfaces don’t should be used with caution. A employ the right click to bring up threshold has to be set during another option as this tends to gowhich two touch events at same away from direct manipulation location count a double click. philosophy MICE VS FINGERSDan Saffer
  16. 16. A cursor is often unnecessary Hovers and mouse over events since the user is not constantly are not employed as this cannot pointing at something. Finger be detected through touch moves from point to point screens. whereas mouse makes a trail. Double click can be done but In general gesture interfaces don’t should be used with caution. A employ the right click to bring up threshold has to be set during another option as this tends to gowhich two touch events at same away from direct manipulation location count a double click. philosophyDrop down menus generally don’t work very well for same reason as right click menus combined with limitations over hover. MICE VS FINGERSDan Saffer
  17. 17. A cursor is often unnecessary Hovers and mouse over events since the user is not constantly are not employed as this cannot pointing at something. Finger be detected through touch moves from point to point screens. whereas mouse makes a trail. Double click can be done but In general gesture interfaces don’t should be used with caution. A employ the right click to bring up threshold has to be set during another option as this tends to gowhich two touch events at same away from direct manipulation location count a double click. philosophy Cut and Paste is nowDrop down menus generally don’t implemented on touch screen work very well for same reason device although presents difficulty as right click menus combined for accurate placement due to with limitations over hover. size of fingers MICE VS FINGERSDan Saffer
  18. 18. A cursor is often unnecessary Hovers and mouse over events since the user is not constantly are not employed as this cannot pointing at something. Finger be detected through touch moves from point to point screens. whereas mouse makes a trail. Double click can be done but In general gesture interfaces don’t should be used with caution. A employ the right click to bring up threshold has to be set during another option as this tends to gowhich two touch events at same away from direct manipulation location count a double click. philosophy Cut and Paste is nowDrop down menus generally don’t implemented on touch screen work very well for same reason device although presents difficulty as right click menus combined for accurate placement due to with limitations over hover. size of fingersAs humans have a limited number of fingers we have a limit onselecting multiple items. Normally this means some form of select mode is used. MICE VS FINGERSDan Saffer
  19. 19. A cursor is often unnecessary Hovers and mouse over events since the user is not constantly are not employed as this cannot pointing at something. Finger be detected through touch moves from point to point screens. whereas mouse makes a trail. Double click can be done but In general gesture interfaces don’t should be used with caution. A employ the right click to bring up threshold has to be set during another option as this tends to gowhich two touch events at same away from direct manipulation location count a double click. philosophy Cut and Paste is nowDrop down menus generally don’t implemented on touch screen work very well for same reason device although presents difficulty as right click menus combined for accurate placement due to with limitations over hover. size of fingersAs humans have a limited number Its hard to do an Undo gesture of fingers we have a limit on once a gesture is done thereforeselecting multiple items. Normally its better to have an easy way to this means some form of select cancel or directly undo the mode is used. action. MICE VS FINGERSDan Saffer
  20. 20. FITTS LAW
  21. 21. Fittss law is a model of humanmovement relating to pointing that predicts that the time required to rapidly move to atarget area is a function of thedistance to the target and the size of the target. FITTS LAW
  22. 22. Put simply a large object closerto the user is easier to point to than a large one far away. Fittss law is a model of human movement relating to pointing that predicts that the time required to rapidly move to a target area is a function of the distance to the target and the size of the target. FITTS LAW
  23. 23. FITTS LAW
  24. 24. This law is equally applicable to gestures. Visual targets should be designed to be close to the user to avoid reaching across the interface. Objects to be manipulated should be large enough to accomodate human fingerFITTS LAW
  25. 25. FINGER TIPS A general guide for the size acceptable targets ideallyshould be no smaller than the smallest average finger pad, typically a 1cm (0.4 inch) diameter is used.
  26. 26. PPI What 1cm translates to in pixels depends on the pixeldensity or Pixels Per Inch (PPI). Once you have PPI simplytimes by 0.4 to get number of pixels of your touch point
  27. 27. Model Diagonal Pixels PPI iPhone 3GS 3.5” 320X480 163 iPhone 4 3.5” 640x960 326 iPad, iPad2 9.7” 1024x768 132Google Nexus 1 3.7” 480x800 252Motorola Droid X 4.3” 854x480 228 Nokia N8 3.5” 640x360 209 Nexus S 4.0” 480x800 235 EXAMPLE PPI
  28. 28. Dan Saffer ICEBERG TIPS These are controls that have larger targets than what is visible. The implication is you need more space between objects.
  29. 29. Dan Saffer OK ICEBERG TIPS These are controls that have larger targets than what is visible. The implication is you need more space between objects.
  30. 30. ADAPTION The keyboard on the iPhone actually uses some of the smallest targets at 5mm (0.2 inches). It uses adaptive targets to get over this limitation.Dan Saffer
  31. 31. Dan Saffer ADAPTIVE TARGETS These are created algorithmically by guessing the next item the user will touch and increasing the touch target appropriately.
  32. 32. Dan Saffer 1st 2nd ADAPTIVE TARGETS These are created algorithmically by guessing the next item the user will touch and increasing the touch target appropriately.
  33. 33. Select Tap Drag FlickPinch Spread Slide Left to Right Slide Up and DownTOUCHSCREEN PATTERNS
  34. 34. CURSE OF TWO FINGERED ZOOM!
  35. 35. The more complicated thegesture the fewer people will use it! CURSE OF TWO FINGERED ZOOM!
  36. 36. USER GENERATED
  37. 37. FREE FORM GESTURES
  38. 38. Spatial Gesture Models 3D Model Based Appearance Based Deformable 2DSkeletal Volumetric Image Sequences Templates NURBS Primitives Super Quadratics SPATIAL GESTURE MODELS
  39. 39. Pointing Pointing WiimoteEXISTING FREE FORM GESTURE PATTERNS
  40. 40. Fukuda’s Proximity Wave AirSwitchAutomatic Light Door EXISTING FREE FORM GESTURE PATTERNS
  41. 41. Hands Inside Rotate NokiaDyson Air N93 Blade EXISTING FREE FORM GESTURE PATTERNS
  42. 42. Step ShakeDance SE W910i Mat EXISTING FREE FORM GESTURE PATTERNS
  43. 43. Clap Tilt Wii Balance BoardEXISTING FREE FORM GESTURE PATTERNS
  44. 44. Moving a Cursor or Avatar Select Swith On/Off Slide Tap Tap Head Tilt Stare Flick Turn Head Left/Right Point Stomp Lean Torso Left/Right Hand Gun Wave Point Clap Snap Confirmation Cancel Nod Shake No Smile Frown Okay Thumbs Down Thumbs Up Stop Nose Tap MATCHING GESTURE TODan Saffer ACTIVITY
  45. 45. Timo ArnallCOMMUNICATING GESTURES
  46. 46. KINECTXbox game controller proving a natural user interface using gestures and spoken commands
  47. 47. Infrared laser projector combined with a Sensors microphone array enables monochrome CMOS sensor capturesacoustic source localisation and ambient video data in 3D under any ambient light noise suppression conditions KINECT
  48. 48. KINECT SDK
  49. 49. SDK includes:1. Raw sensor streams: Access to low-levelstreams from the depth sensor, colour camerasensor, and four-element microphone array.2. Skeletal tracking: The capability to track theskeleton image of one or two people movingwithin the field of view for gesture-drivenapplications. KINECT SDK
  50. 50. SDK includes:1. Raw sensor streams: Access to low-levelstreams from the depth sensor, colour camerasensor, and four-element microphone array.2. Skeletal tracking: The capability to track theskeleton image of one or two people movingwithin the field of view for gesture-drivenapplications. 3. Advanced audio capabilities: Audio processing capabilities include sophisticated acoustic noise suppression and echo cancellation, beam formation to identify the current sound source, and integration with the Windows speech recognition API. 4. Sample code and Documentation. KINECT SDK
  51. 51. CONCLUSIONS• Currently gestures can be classified predominantly as touch or free form• The finger is not a direct replacement for the mouse.• The more complicated the gesture the fewer the number of people who will use it successfully.• If gestures are not obvious to the user the need to be clearly communicated• As yet our vocabulary of free form gestures is limited and needs much more development.• Free form gestures should make the user embarrassed.
  52. 52. @mysticmobileQUESTIONS

×