Introducing gestures and its relevance to the Flash platform.

1. Simple Expressions @ Play
Shashank Tiwari
www.treasuryofideas.com | www.shanky.org
shashank.>wari@treasuryofideas.com|
tshanky@gmail.com | twi?er: tshanky

2. What’s coming?
• Gestures everywhere!
• Gestures in Flash today
• What’s brewing?
• Where do I go from here?

3. What exactly is a gesture?
• Touch, mo>on, voice, thought?
• Sta>c – perform and hold
– Example – thumbs up!, touch
• Dynamic – movement over >me
– Example – wave, flick/pinch
• Well defined device constrained | free form

4. Gestures Everywhere!
Jeff Han
1
Of
15

5. Gestures Everywhere!
Wii
2
Of
15

6. Gestures Everywhere!
The Clapper
3
Of
15

7. Gestures Everywhere!
Google Nexus One
4
Of
15

8. Gestures Everywhere!
ATM
5
Of
15

9. Gestures Everywhere!
Airport Check‐in
kiosk
6
Of
15

10. Gestures Everywhere!
Glowing Places
7
Of
15

11. Gestures Everywhere!
MicrosoY Surface
8
Of
15

12. Gestures Everywhere!
Fukuda’s automa>c door
9
Of
15

13. Gestures Everywhere!
enMo>on’s Paper towel dispenser
10
Of
15

14. Gestures Everywhere!
Dyson Air Blade
11
Of
15

15. Gestures Everywhere!
Dance Dance Revolu>on
12
Of
15

16. Gestures Everywhere!
Cubit
13
Of
15

17. Gestures Everywhere!
Intel’s touch screen wall
14
Of
15

18. Gestures Everywhere!
Philips Imageo candle light
15
Of
15

19. Gestures in Flash
• It’s not new on the Flash plaorm!
• So how do we get gestures working
1
– If you are not blind to inputs, you can possible act Of
on it. 45
• Old Style – FP ‐>Socket ‐>device inputs
• New Style – FP 10.1 ‐> driver baked into the
player

20. Gestures in Flash
• Old Style
– Touchlib ‐‐ h?p://code.google.com/p/touchlib/
– Gestureworks ‐‐ h?p://gestureworks.com/ 2
Of
– Ideo ‐‐ 45
h?p://code.google.com/p/ideo‐mul>touch/
– Touche ‐‐ h?p://code.google.com/p/touche/
– (op>cal mul>‐touch)

21. Gestures in Flash
• New Style
– Build‐in support in Flash Player 10.1 and AIR 2
• For mul>‐touch gestures 3
Of
• Promising to be cross plaorm in future 45
– (provided that is possible!)

22. Gestures in Flash
• Before we learn about the mul>‐touch
support, lets see what’s influencing it…
4
Of
45

23. Gestures in Flash
• Types of touch devices:
• Resis>ve ‐‐ (example: Kiosks)
5
– 2 thin electrically conduc>ve layers separated by a Of
gap 45
– On touch panels get connected
– Panels act as voltage divider pair with connected
outputs
– Change in electric current registered as a touch
event

24. Gestures in Flash
• Capaci>ve –
– Insulator like glass covered with a conductor like
Indium Tin Oxide (ITO) 6
Of
– Human body is a good condutor 45
– On touch electrosta>c field is distorted, which is
recorded as a change in capacitance (ability of a
body to hold charge)

25. Gestures in Flash
• Op>cal Imaging –
– 2 or > image sensors placed around edges
– On the other side, in the camera’s field of view 7
Of
infrared light is placed 45
– touch translates to shadow
– Pair of cameras triangulated to locate a touch

26. Gestures in Flash
• Surface wave acous>cs –
– Ultrasonic waves pass over the surface
– Touch absorbs ultrasonic waves 8
Of
• Dispersive signal 45
– Sensors touch mechanical energy in the glass due
to touch
• Strain gauge
– Spring mounted on 4 corners
– Gauges determine deflec>on on touch

27. Gestures in Flash
• Number of touch points
– 2 seems too many!
9
Of
45

28. Gestures in Flash
• Mul>‐touch devices – what have we got?
– 3M — M2256PW with ten‐finger support
– Acer — Acer Aspire 1820PT & 5738PG
– Apple — iPhone, iPad, iPod Touch, MacBook, MacBook Air, 10
MacBook Pro, Magic Mouse Of
45
– Asus — EEE PC T91MT & T101MT
– Circle Twelve — DiamondTouch
– Dell — La>tude XT & XT2, Mini 5, Studio 17
– Google — Nexus One
– Hewle?‐Packard — HP Touchsmart
– HTC — HTC Hero, HTC HD2
– Ideum — MT‐50 Mul>touch Table

29. Gestures in Flash
– Lenovo — X200 & T400
– LG Electronics — Arena, BL40 New Chocolate
– MicrosoY — Surface, Zune HD
– Motorola — Droid 11
– MULTIVISION — Mul>‐Touch LCD — up to 32 fingers Of
– Nortd — TouchKit 45
– Palm — Pre, Pixi
– Percep>ve Pixel — Mul>‐Touch Collabora>on Wall
– Shu?le Inc. — Mul>‐touch LCD X50v2
– Sony — VAIO L Series All‐in‐one desktops
– Tyco Electronics/Elo TouchSystems ‐ IntelliTouch Plus Surface
Acous>c Wave Mul>‐touch Touch Technology
– Wacom — Bamboo tablets

30. Gestures in Flash
• For starters how do I find if FP 10.1 | AIR 2
knows of the touch events?
12
Of
45

31. Gestures in Flash
Mul>touch.supportsGestureEvents
– Does it?
13
Of
Mul>touch.supportsTouchEvents 45
– Does it?

32. Gestures in Flash
• Is it single touch or gestures?
14
– Set the correct inputMode Of
– Which is set flash.ui.Mul>touch.inputMode 45

33. Gestures in Flash
• flash.ui.Mul>touchInputMode
– GESTURE – gestures recognized, simple touch
event = mouse event 15
Of
– NONE – everything is treated as a mouse event 45
– TOUCH_POINT – only the basic touch events, no
gestures

34. Gestures in Flash
• How many touch points are supported?
16
– Mul>touch.maxTouchPoints Of
45

35. Gestures in Flash
• Touch event or Gesture event?
• flash.events.TouchEvent
17
Of
45
• flash.events.GestureEvent
• flash.events.TransformGestureEvent
• flash.events.PressAndTapGestureEvent

36. Gestures in Flash
• TouchEvent
– TOUCH_BEGIN
– TOUCH_END 18
Of
– TOUCH_MOVE 45
– TOUCH_OUT

37. Gestures in Flash
• TouchEvent (…cont)
– TOUCH_OVER
– TOUCH_ROLL_OUT 19
Of
– TOUCH_ROLL_OVER 45
– TOUCH_TAP

38. Gestures in Flash
• Proper>es the TouchEvent object has –
– touchPointID – unique ID a?ached to the touch
point 20
Of
– localX – x coordinate rela>ve to the sprite 45
– localY – y cordinate rela>ve to the sprite
– sizeX – width of the contact area
– sizeY – height of the contact area

39. Gestures in Flash
• Proper>es the TouchEvent object has (…cont)
–
– controlKey 21
Of
– commandKey 45
– altKey
– shiYKey

40. Gestures in Flash
• Raw touch events can be handled to
understand gestures, but there are some
already there. 22
Of
45

41. Gestures in Flash
• TransformGestureEvent
– GESTURE_PAN
– GESTURE_ROTATE 23
Of
– GESTURE_SWIPE 45
– GESTURE_ZOOM

42. Gestures in Flash
• GESTURE_PAN
24
Of
45
• h?p://gestureworks.com/wp‐content/themes/
gestureWorks/images/gesturePngs/lock_two_pan.png
• (h?p://i.msdn.microsoY.com/Dd940543.gestures%28en‐
us,VS.85%29.png)

43. Gestures in Flash
• GESTURE_ROTATE
25
Of
45
• h?p://gestureworks.com/wp‐content/themes/
gestureWorks/images/gesturePngs/two_finger_rotate.png
• (h?p://i.msdn.microsoY.com/Dd940543.gestures%28en‐
us,VS.85%29.png)

44. Gestures in Flash
• What about this?
26
Of
45
• h?p://gestureworks.com/wp‐content/themes/
gestureWorks/images/gesturePngs/
two_hand_rotate.png

45. Gestures in Flash
• GESTURE_SWIPE
27
Of
45
• h?p://regmedia.co.uk/2008/12/29/swipe_patent1.jpg
• (h?p://i.msdn.microsoY.com/Dd940543.gestures%28en‐
us,VS.85%29.png)

46. Gestures in Flash
• What about this?
28
Of
45
• One finger flick ‐‐ h?p://gestureworks.com/wp‐
content/themes/gestureWorks/images/
gesturePngs/one_finger_swipe.png

47. Gestures in Flash
• And this?
29
Of
45
• Two finger flick ‐‐ h?p://gestureworks.com/wp‐content/
themes/gestureWorks/images/gesturePngs/
two_finger_swipe.png

48. Gestures in Flash
• GESTURE_ZOOM
30
Of
45
• h?p://gestureworks.com/wp‐content/themes/
gestureWorks/images/gesturePngs/two_finger_zoom_in.png
• (h?p://i.msdn.microsoY.com/Dd940543.gestures%28en‐
us,VS.85%29.png)

49. Gestures in Flash
• What about this?
31
Of
45
• Mul> point zoom ‐‐ h?p://gestureworks.com/wp‐
content/themes/gestureWorks/images/gesturePngs/
two_hand_zoom_in.png

50. Gestures in Flash
• And this?
32
Of
45
• h?p://gestureworks.com/wp‐content/themes/
gestureWorks/images/gesturePngs/
pinch_zoom.png

51. Gestures in Flash
• GestureEvent
– GESTURE_TWO_FINGER_TAP
33
Of
45

52. Gestures in Flash
• GESTURE_TWO_FINGER_TAP
34
Of
45
• h?p://gestureworks.com/wp‐content/themes/
gestureWorks/images/gesturePngs/two_finger_tap.png
• h?p://i.msdn.microsoY.com/Dd940543.gestures
%28en‐us,VS.85%29.png

53. Gestures in Flash
• What about this?
35
Of
45
• Two finger double tap ‐‐ h?p://gestureworks.com/
wp‐content/themes/gestureWorks/images/
gesturePngs/two_finger_double_tap.png

54. Gestures in Flash
• PressAndTapGestureEvent
– GESTURE_PRESS_AND_TAP
36
Of
45

55. Gestures in Flash
• GESTURE_PRESS_AND_TAP
37
Of
45
• h?p://i.msdn.microsoY.com/Dd940543.gestures
%28en‐us,VS.85%29.png

56. Gestures in Flash
• Lets see a simple example (a media sorter)
– FP 10.1
– AIR 2 38
Of
45

57. Gestures in Flash
• So what works?
• Win 7 – all good
39
• Mac – AIR 2  | FP 10.1  Of
45
• Linux – some>mes works!

58. Gestures in Flash
• Another example (a soY keyboard)
– Older FP (earlier than the one that officially
supports touch and gestures) 40
Of
– FP 10.1 45
– AIR 2

59. Gestures in Flash
• Yet another example (an elementary rota>ng
globe)
41
Of
45

60. Gestures in Flash
• What about custom gestures?
– Combina>on | Aggrega>on
– Extension 42
Of
45

61. Gestures in Flash
• Aggrega>on
• GESTURE_ZOOM + TAP
43
– <>.addEventListener(TransformGestureEvent.GEST Of
URE_ZOOM , onZoom); 45
– func>on onZoom
(evt:TransformGestureEvent):void {
if (evt.phase==GesturePhase.BEGIN) { add event
listener for tap – which is translated to a mouse click }

62. Gestures in Flash
• Aggrega>on
• GESTURE_ZOOM + GESTURE_ROTATE
44
• ? Of
45

63. Gestures in Flash
• Extension
– Listen to raw touch events and use your brilliance!
45
Of
45

64. What’s brewing?
• Extend gesture support beyond the basics
– Gesturekit ‐‐
h?p://code.google.com/p/gesturekit/ 1
Of
5

65. What’s brewing?
• Help those who don’t have access to a mul>‐
touch device
– Start with mul>touchup.com 2
Of
– Rely on your mouse 5
– Mac track pad isn’t bad (with AIR 2)

66. What’s brewing?
• Touch drag support within the FP api
• Support on all plaorms
3
– Mobile seems a priority Of
5

67. What’s brewing?
• Newer devices – smartphones, slates and
netbooks
• Newer touch access points – kiosks, surfaces 4
Of
and walls 5

68. What’s brewing?
• Integra>on with gestures beyond touch
– Mo>on detec>on
– Sound recogni>on 5
Of
– Free form expressions 5
– Simultaneous tracking of each of these

69. Where do I go from here?
• Get FP 10.1 & AIR 2
• Get hold of a touch screen | device
1
• Read the documenta>on Of
4
– h?p://help.adobe.com/en_US/as3/dev/
WSb2ba3b1aad8a27b0‐6€37601221e58cc29‐80
00.html
– h?p://help.adobe.com/en_US/FlashPlaorm/
beta/reference/ac>onscript/3/flash/events/
TouchEvent.html

70. Where do I go from here?
• Write an app or two
• Give up your sleep tracking the TouchEvent!
2
Of
4

71. Where do I go from here?
• Explore
– Unistroke recognizer ‐‐
h?p://depts.washington.edu/aimgroup/proj/ 3
Of
dollar/ 4
– h?p://www.betriebsraum.de/projects/gestures/
• Gestureworks – Open source gesture library
– h?p://gestureworks.com/about/open‐source‐
gesture‐library/

72. Where do I go from here?
• Gesture recogni>on ‐‐
h?p://github.com/sqrtof5/
GestureRecogni>on 4
Of
• Touchlib source – 4
– h?p://code.google.com/p/touchlib/source/
browse/trunk/AS3/int/app/core/ac>on/gestures/
GestureNormalizer.as

73. This presenta>on & demos?
• h?p://shanky.org/publish‐and‐present/flex
• h?p://www.treasuryofideas.com/gestures/
touch/MediaSorter/MediaSorter.html
– I will tweet the exact links and when the updated
versions are ready – twi?er:tshanky
– Images for slides were downloaded from the web. The
copyrights is with their respec>ve owners
– References to material used between slides 4 and 18
is available at h?p://www.shanky.org/references/
simple_expressions_at_play_references.pdf.

74. Thanks for coming!
Shashank Tiwari
www.treasuryofideas.com | www.shanky.org
shashank.9wari@treasuryofideas.com| tshanky@gmail.com |
twi?er: tshanky