Multi Touch And Gesture Event Interface And Types


Published on

To show gesture event model based on multi-touch device like iPhone and Android.

  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Event Diagram 확인하여 다시 그려 주세요 .
  • Multi Touch And Gesture Event Interface And Types

    1. 1. Multi Touch & Gesture - Event Interface and Types June, 2010 TC Meeting in Minneapolis, MN Eunil SEO / [email_address] Siman KIM / [email_address] Sangchual CHA / [email_address]
    2. 2. Agenda <ul><li>UI Event - Mouse Event in W3C </li></ul><ul><li>Multi Touch and Gesture </li></ul><ul><li>Event Issues </li></ul><ul><li>W3C DOM Event Model Extension </li></ul>
    3. 3. Purpose of this presentation <ul><li>To collect current technical status on multi-touch and gesture event. </li></ul><ul><li>To reveal issues about multi-touch and gesture event. </li></ul><ul><li>To suggest multi-touch and gesture event model with architecture, interface and types. </li></ul>
    4. 4. Event Model in W3C <ul><li>Document Object Model (DOM) Level 3 Events Specification (Sep. 2009) </li></ul><ul><li> </li></ul><ul><li>Event Architecture </li></ul><ul><li>Event Interface </li></ul><ul><li>Event Module </li></ul><ul><ul><li>Event Types </li></ul></ul><ul><ul><li>Event Module Definition </li></ul></ul>
    5. 5. DOM Lvl3 Event Architecture <ul><li>Event Dispatch Mechanism </li></ul><ul><li>Event Flow </li></ul><ul><ul><li>Capture Phase </li></ul></ul><ul><ul><li>Bubble Phase </li></ul></ul><ul><ul><li>Target Phase </li></ul></ul>
    6. 6. DOM Lvl3 Event Interface
    7. 7. DOM Lvl3 MouseEvent Interface <ul><li>interface MouseEvent : UIEvent { </li></ul><ul><li>readonly attribute long screenX ; </li></ul><ul><li>readonly attribute long screenY ; </li></ul><ul><li>readonly attribute long clientX ; </li></ul><ul><li>readonly attribute long clientY ; </li></ul><ul><li>readonly attribute boolean ctrlKey ; </li></ul><ul><li>readonly attribute boolean shiftKey ; </li></ul><ul><li>readonly attribute boolean altKey ; </li></ul><ul><li>readonly attribute boolean metaKey ; </li></ul><ul><li>readonly attribute unsigned short button ; </li></ul><ul><li>readonly attribute EventTarget relatedTarget ; </li></ul><ul><li>… </li></ul><ul><li>… </li></ul><ul><li>}; </li></ul>
    8. 8. DOM Lvl3 MouseEvent Interface Cont’d <ul><li>interface MouseEvent : UIEvent { </li></ul><ul><li>… </li></ul><ul><li>… </li></ul><ul><li>void initMouseEvent(in DOMString typeArg , </li></ul><ul><li>in boolean canBubbleArg , </li></ul><ul><li>in boolean cancelableArg , </li></ul><ul><li>in views::AbstractView viewArg , </li></ul><ul><li>in long detailArg , </li></ul><ul><li>in long screenXArg , </li></ul><ul><li>in long screenYArg , </li></ul><ul><li>in long clientXArg , </li></ul><ul><li>in long clientYArg , </li></ul><ul><li>in boolean ctrlKeyArg , </li></ul><ul><li>in boolean altKeyArg , </li></ul><ul><li>in boolean shiftKeyArg , </li></ul><ul><li>in boolean metaKeyArg , </li></ul><ul><li>in unsigned short buttonArg , </li></ul><ul><li>in EventTarget relatedTargetArg ); </li></ul><ul><li>}; </li></ul>
    9. 9. DOM Lvl3 Event Types <ul><li>Event Iterfaces </li></ul><ul><li>MouseEvent </li></ul><ul><li>TextEvent </li></ul><ul><li>KeyboardEvent </li></ul><ul><li>CompositionEvent </li></ul><ul><li>MutationName Event </li></ul><ul><li>MouseWheelEvent </li></ul><ul><li>WheelEvent </li></ul><ul><li>Mouse Event Types </li></ul><ul><li>click event </li></ul><ul><li>dblclick event </li></ul><ul><li>mousedown event </li></ul><ul><li>mouseenter event </li></ul><ul><li>mouseleave event </li></ul><ul><li>mousemove event </li></ul><ul><li>mouseover event </li></ul><ul><li>mouseout event </li></ul><ul><li>mouseup event </li></ul><ul><li>mousewheel Event </li></ul><ul><li>wheel Event </li></ul>
    10. 10. RIA Event Spectrum Event Interpreter Device Event Device Event Device Event Event Interpreter Event Interpreter Interpreted Event Interpreted Event * Physical device with device driver * With Application Framework * RIA Platform includes Application Framework Device O/S RIA Platform Application Device RIA Platform
    11. 11. RIA Event Spectrum <ul><li>Flow through Device, O/S, RIA Platform, RIA Application Framework and RIA Application. </li></ul><ul><li>Device Event </li></ul><ul><ul><li>Events origin from device. </li></ul></ul><ul><ul><li>Keep its context through spectrum. </li></ul></ul><ul><li>Interpreted Event </li></ul><ul><ul><li>Combination of device events and interpreted events. </li></ul></ul><ul><ul><li>Represent with State Diagram. </li></ul></ul>
    12. 12. New Event Issues <ul><li>Single Touch - Click vs. Tap </li></ul><ul><ul><li>Click with stylus vs. Tap with Fingers </li></ul></ul><ul><ul><li>Interval vs. Duration </li></ul></ul><ul><ul><li>Point(x, y) vs. Region(x, y, width, height) </li></ul></ul><ul><li>Array of Touch Evet </li></ul><ul><ul><li>Sequence of touch vs. Set of touch </li></ul></ul><ul><li>Gesture </li></ul><ul><ul><li>How to understand the intension of action. </li></ul></ul><ul><ul><li>Gesture in Space (Gesture with Glove, Camera, etc) </li></ul></ul><ul><li>Shake and rotate (Gyro Sensor) </li></ul>
    13. 13. Multi-Touch Event <ul><li>Multi-touch is an enhancement to touch-screen technology, which provides the user with the ability to apply multiple finger gestures simultaneously onto the electronic visual display to send complex commands to the device. (Wikipedia) </li></ul><ul><li>Multi Touch = Sequence of Touch (Down, Up, Move) </li></ul>
    14. 14. Gesture Event <ul><li>Any physical movement that can be sensed and responded to by a digital system without the aid of a traditional input device such as a mouse or stylus. </li></ul><ul><li>To understand user intensions by recognizing multi-touch events, from the first finger down event to the last finger up event. </li></ul><ul><li>Multi-touch gesture, Mouse gesture, Motion gesture, etc </li></ul>
    15. 15. Sample Gestures in Touch Device Gestureworks Open Souce Gesture Library ( ) Tap Two finger zoom out Two finger zoom in Double Tap Touch Down Touch Up One finger drag Pinch zoom Two finger tap One finger swipe Two finger swipe Two finger double tap Two finger horizontal scroll Two finger vertical scroll
    16. 16. Gestures in iPhone O/S <ul><li>Gesture Types </li></ul><ul><ul><li>Tapping (any number of tabs) </li></ul></ul><ul><ul><li>Pinching in and out (for zooming a view) </li></ul></ul><ul><ul><li>Panning or dragging </li></ul></ul><ul><ul><li>Swing (in any direction) </li></ul></ul><ul><ul><li>Rotation (fingers moving in opposite directions) </li></ul></ul><ul><ul><li>Long Press (aks touch and hold) </li></ul></ul> Gesture Recognizers Events (iPhone OS)
    17. 17. Gestures in iPhone O/S cont’d <ul><li>Device (O/S) detect touch event </li></ul><ul><li>Interpret Touch Event in Cocoa Framework </li></ul><ul><li>UITouch Event Class </li></ul>
    18. 18. Gestures in iPhone O/S cont’d
    19. 19. Gestures in iPhone O/S cont’d <ul><li>touchesBegan:withEvent:  is called for touch objects in the Began phase. </li></ul><ul><li>touchesMoved:withEvent:  is called for touch objects in the Moved phase. </li></ul><ul><li>touchesEnded:withEvent:  is called for touch objects in the Ended phase. </li></ul><ul><li>touchesCancelled:withEvent:  is called when some external event (e.g. Incomming Call). </li></ul>
    20. 20. Gestures in Android <ul><li>Event Listeners </li></ul><ul><ul><li>OnClickListner </li></ul></ul><ul><ul><li>OnLongClickListner </li></ul></ul><ul><ul><li>OnTouchListener </li></ul></ul><ul><li>Touch Event </li></ul><ul><ul><li>public abstract boolean  onTouch  ( View  v,  MotionEvent  event) </li></ul></ul><ul><ul><ul><li>TouchDown </li></ul></ul></ul><ul><ul><ul><li>TouchMove </li></ul></ul></ul><ul><ul><ul><li>TouchUp </li></ul></ul></ul><ul><ul><ul><li>TouchTap </li></ul></ul></ul>
    21. 21. Gestures in Android cont’d <ul><li>OnGestureListeners </li></ul><ul><ul><li>OnGesture </li></ul></ul><ul><ul><li>OnOnGestureCancelled </li></ul></ul><ul><ul><li>OnGestureEnded </li></ul></ul><ul><ul><li>OnGestureStarted </li></ul></ul>OnGestureStarted OnGestureCancelled Ready OnGestureEnded OnGesture OnTouchDown OnTouchUp OnTouchTap Cancelling Event Ex) CallingEvent OnTouchDown OnTouchUp OnTouchMove OnTouchTap
    22. 22. Gestures in Android cont’d <ul><li>GestureDetector (SimpleGestureDetector) </li></ul><ul><ul><li>onDoubleTap </li></ul></ul><ul><ul><li>onDown </li></ul></ul><ul><ul><li>onFling </li></ul></ul><ul><ul><li>onLongPress </li></ul></ul><ul><ul><li>onScroll </li></ul></ul><ul><ul><li>onShowPress </li></ul></ul><ul><ul><li>onSingTapUp </li></ul></ul><ul><ul><li>onGestureRotate </li></ul></ul><ul><ul><li>onGestureScroll </li></ul></ul><ul><ul><li>onGestureScale </li></ul></ul>Reference : How to use Multi-touch in Android 2 in ZDNET
    23. 23. Gestures in MS Windows 7 <ul><li>System gesture Synthesized equivalent message </li></ul><ul><li>Hover (when supported) >> Mouse hover </li></ul><ul><li>Tap (down and up) >> Mouse left-click </li></ul><ul><li>Double tap (down and up twice)>> Mouse double left-click </li></ul><ul><li>Press and hold (down, pause, up)>>Mouse right-click </li></ul><ul><li>Drag (down, move, up) >> Mouse left-drag </li></ul><ul><li>Press, hold, and drag (down, pause, move, up) >> Mouse right-drag </li></ul><ul><li>Select (down, move over selectable objects, up) >> Mouse select </li></ul>
    24. 24. MS Windows - Panning <ul><li>Entry state: One or two fingers in contact with the screen. </li></ul><ul><li>Motion: Drag, with any additional fingers remaining in same position relative to each other. </li></ul><ul><li>Exit state: Last finger up ends the gesture. </li></ul><ul><li>Effect: Move the underlying object directly and immediately as the fingers move. Be sure to keep the contact point under the finger throughout the gesture. </li></ul>
    25. 25. MS Windows - Zoom <ul><li>Entry state: Two fingers in contact with the screen at the same time. </li></ul><ul><li>Motion: Fingers move apart or together (pinch) along an axis. </li></ul><ul><li>Exit state: Any finger up ends the gesture or the fingers break the axis. </li></ul><ul><li>Effect: Zoom the underlying object in or out directly and immediately as the fingers separate or approach on the axis. Be sure to keep the contact points under the finger throughout the gesture. </li></ul>
    26. 26. MS Windows - Rotate <ul><li>Entry state: Two fingers in contact with the screen at the same time. </li></ul><ul><li>Motion: One or both fingers rotate around the other, moving perpendicular to the line between them. </li></ul><ul><li>Exit state: Any finger up ends the gesture. </li></ul><ul><li>Effect: Rotate the underlying object the same amount as the fingers have rotated. Be sure to keep the contact points under the finger throughout the gesture. </li></ul>
    27. 27. MS Windows - Two-finger tap <ul><li>Entry state: Two fingers in contact with the screen at the same time. </li></ul><ul><li>Motion: No motion. </li></ul><ul><li>Exit state: Any finger up ends the gesture. </li></ul><ul><li>Effect: Alternatively zooms or restores the default view for the object between the fingers. </li></ul>
    28. 28. MS Windows - Press and tap <ul><li>Entry state: One finger in contact with the screen, followed by a second finger. </li></ul><ul><li>Motion: No motion. </li></ul><ul><li>Exit state: Second finger up ends the gesture. </li></ul><ul><li>Effect: Performs a right click for the object under the first finger </li></ul>
    29. 29. Related Issues <ul><li>Lack of standard : Different understanding in user actions. </li></ul><ul><li>Event Sharing between components : Inter-component event dispatch. (e.g. Drag & Drop) </li></ul><ul><li>We need RIA standard gesture event model. </li></ul><ul><li>W3C DOM Level 3 Event Model doesn’t support Gesture event. Browser War !! </li></ul>
    30. 30. Reference <ul><li>UXGuide - </li></ul>