Developing Multi Touch Applications


Published on

Presentation at Microsoft's Saint Louis Day of Dot Net.

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Developing Multi Touch Applications

    1. 1. Developing Multi-touch Applications The future is here Brian Blanchard Chief Architect Essence Technology
    2. 2. Agenda <ul><li>Introduction </li></ul><ul><ul><li>Past, Future, & Present </li></ul></ul><ul><li>Touch scenarios, “Good, Better, Best” model, and developer roadmap </li></ul><ul><li>Platform details: </li></ul><ul><ul><li>Native Win32 gesture and touch support </li></ul></ul><ul><ul><li>Manipulations and Inertia </li></ul></ul><ul><ul><li>WPF support </li></ul></ul><ul><li>UX guidelines for touch applications </li></ul><ul><li>Where to get started - SDK, drivers, etc </li></ul><ul><li>Wrap up </li></ul>
    3. 3. What is Multi-Touch? <ul><li>Multi-touch * denotes a set of interaction techniques which allow computer users to control graphical applications with several fingers. </li></ul><ul><li>The concept of Multi-touch technology was created to incorporate collaboration into the computing experience. </li></ul><ul><li>* Wikipedia definition </li></ul>
    4. 4. Multi-touch Past <ul><li>26 years in the making </li></ul><ul><li>Started in 82 by the university of Toronto </li></ul><ul><li>First commercial attempt in 84 by bell labs </li></ul><ul><li>Adopted by MS in 2001 </li></ul>
    5. 5. Multi-touch Future <ul><li>LucidTouch: Transparent handheld inputs </li></ul><ul><li>Obscura Digital: </li></ul><ul><ul><li>HP - D5: Multi-Touch wall </li></ul></ul><ul><ul><li>VisionAire: Holographic Multi-Touch </li></ul></ul>
    6. 6. Multi-touch Is Here! <ul><li>Hardware </li></ul><ul><ul><li>Multi-touch capable machines in market today, multi-touch appears in a broad set of form factors </li></ul></ul><ul><ul><li>Desktops: $1200 - $2000 </li></ul></ul><ul><ul><li>Laptops: $1800 - $3000 </li></ul></ul><ul><li>Software </li></ul><ul><ul><li>Windows 7, PDC partner demonstrations & Touch platform announcement, .Net 4.0 </li></ul></ul><ul><li>Consumers </li></ul><ul><ul><li>New scenarios, next wave of user experience, high “WOW” factor </li></ul></ul>
    7. 7. Touch Scenarios And Windows 7 <ul><li>We’ve identified several key scenarios for multi-touch: </li></ul><ul><li>Navigating and consuming the web </li></ul><ul><li>Reading & sorting email </li></ul><ul><li>Viewing photos </li></ul><ul><li>Playing casual games </li></ul><ul><li>Consuming music and video </li></ul><ul><li>Navigating files & arranging windows </li></ul><ul><li>Using Office applications </li></ul><ul><li>All focused on consumption </li></ul><ul><li>That led to four areas of investment: </li></ul><ul><li>Developer Platform : At the root is the touch developer platform that exposes touch APIs for any application </li></ul><ul><li>UI Enhancements : Focusing on the core scenarios, many parts of the core UI have been optimized for touch experiences. </li></ul><ul><li>Gestures : Multi-touch gestures have been added to enable consistent panning and zooming in most applications. </li></ul><ul><li>Applications : A set of multi-touch focused applications that demonstrate the power of touch. These will ship separate from Win7. </li></ul>
    8. 8. A Structured Approach To Adding Touch Support To Your Application <ul><li>Which application scenarios get the biggest benefit from touch? </li></ul><ul><ul><li>Content consumption, mobile, media and entertainment </li></ul></ul><ul><li>Decide which tier of investment is appropriate: </li></ul><ul><ul><li>Good – Free support, UX tweaks </li></ul></ul><ul><ul><li>Better – Gesture support, touch-friendly UX </li></ul></ul><ul><ul><li>Best – Touch-optimized experience </li></ul></ul>
    9. 9. Touch Development Roadmap Windows 7 Release NET 4.0 / Surface 2.0 Release Multi-Touch Controls Multi-Touch API Surface Multi-Touch Controls & API Multi-Touch API Multi-Touch API and Controls
    10. 10. Touch Platform Overview Good Better Best APIs <ul><li>For Free! </li></ul><ul><li>Panning/zoom gestures </li></ul><ul><li>Right click gesture </li></ul><ul><li>Gesture notifications </li></ul><ul><li>Pan/zoom/rotate/etc </li></ul><ul><li>Raw touch data </li></ul><ul><li>Manipulation and Inertia processors </li></ul>Native Win32 <ul><li>Controls with standard scrollbars </li></ul><ul><li>WM_GESTURE message </li></ul><ul><li>WM_TOUCH </li></ul><ul><li>COM based Manipulation and Inertia Processors </li></ul>WPF <ul><li>WPF 4.0 pan support in ScrollViewer </li></ul><ul><li>Gesture events </li></ul><ul><li>Inertia configuration </li></ul><ul><li>Touch events </li></ul><ul><li>Manipulation and Inertia Processors </li></ul>WinForms <ul><li>Controls with standard scrollbars </li></ul><ul><li>WM_GESTURE message </li></ul><ul><li>P/Invoke </li></ul><ul><li>Manipulation and Inertia Processors in Microsoft.Ink.DLL </li></ul><ul><li>Real-time Stylus or Ink Collector </li></ul>
    11. 11. Windows 7 Gestures & WM_GESTURECOMMAND <ul><li>Brian Blanchard </li></ul><ul><li>Chief Architect </li></ul><ul><li>Essence Technology </li></ul>
    12. 12. WM_GESTURECOMMAND <ul><li>WM_GESTURECOMMAND </li></ul><ul><ul><li>Better : Good gesture support is key </li></ul></ul><ul><ul><li>Window message conveying gesture notifications and information to applications </li></ul></ul><ul><ul><li>Why use it? Richer than legacy notifications, smoother more responsive scrolling and zooming </li></ul></ul><ul><li>Compound gestures </li></ul><ul><li>Coalescing </li></ul><ul><li>Message bubbling and capture </li></ul><ul><li>Legacy handling & SFP </li></ul>
    13. 13. WM_GESTURECOMMAND <ul><li>BOOL WINAPI GetGestureCommandInfo( </li></ul><ul><li>__in UINT uMsg , </li></ul><ul><li>__in WPARAM wParam , </li></ul><ul><li>__in LPARAM lParam , </li></ul><ul><li>__in LPARAM lExtraInfo , </li></ul><ul><li>__inout PGESTUREINFO pGestureInfo </li></ul><ul><li>); </li></ul><ul><li>typedef struct _GESTURECOMMANDINFO { UINT cbSize; DWORD dwFlags; DWORD dwCommand; DWORD dwArguments; POINTS ptsLocation; </li></ul><ul><li>} GESTURECOMMANDINFO, *PGESTURECOMMANDINFO; </li></ul>
    14. 14. WM_GESTURECOMMAND <ul><li>Interpreting gestures: </li></ul>dwCommand dwArguments ptsLocation Pan Distance between contacts Current center of gesture Zoom Distance between contacts Current center of gesture Rotate Absolute angle on rotate start, delta on updates Current center of gesture Two-finger tap NA Current center of gesture Right click gesture NA First finger down
    15. 15. WM_GESTURE Future Windows 7 Build <ul><li>Minor API surface changes: </li></ul><ul><ul><li>Naming: WM_GESTURE, GESTUREINFO, GetGestureInfo(), etc </li></ul></ul><ul><ul><li>Additional fields in GESTUREINFO </li></ul></ul><ul><li>Added gesture configuration support: </li></ul><ul><ul><li>WM_GESTURENOTIFY </li></ul></ul><ul><ul><li>GESTURECONFIG struct </li></ul></ul><ul><ul><li>Set/GetGestureConfig() </li></ul></ul>
    16. 16. Single-finger Panning Future Windows 7 Build <ul><li>Single-finger panning (SFP) provides an even easier way to pan where it is natural </li></ul><ul><li>How to control SFP in your application: </li></ul><ul><ul><li>SetGestureConfig() for GC_PAN </li></ul></ul><ul><ul><li>Specify GC_PAN_WITH_SINGLE_FINGER_VERTICALLY and/or GC_PAN_WITH_SINGLE_FINGER_HORIZONTALLY </li></ul></ul><ul><li>If you aren’t getting the SFP default: </li></ul><ul><ul><li>Consider standard scrollbars </li></ul></ul><ul><ul><li>SBM_GETSCROLLINFO (remarks here are helpful) </li></ul></ul><ul><ul><li>Implement WM_GESTURE support </li></ul></ul>
    17. 17. WM_TOUCH <ul><li>Brian Blanchard </li></ul><ul><li>Chief Architect </li></ul><ul><li>Essence Technology </li></ul>
    18. 18. WM_TOUCH <ul><li>WM_TOUCH* family of messages </li></ul><ul><ul><li>Best: WM_TOUCH enables touch-optimized experiences </li></ul></ul><ul><ul><li>WM_TOUCHDOWN, WM_TOUCHMOVE, WM_TOUCHUP </li></ul></ul><ul><ul><li>Semantically similar to mouse messages </li></ul></ul><ul><ul><li>Conveys raw touch data to Win32 apps </li></ul></ul><ul><ul><li>Scenario examples: </li></ul></ul><ul><ul><ul><li>Finger painting, custom gestures, feeding higher level controls, etc </li></ul></ul></ul><ul><li>Capture </li></ul><ul><li>Coalescing </li></ul>
    19. 19. WM_TOUCH <ul><li>BOOL WINAPI RegisterTouchWindow( </li></ul><ul><li>__in HWND hwnd, </li></ul><ul><li>__in ULONG ulFlags); </li></ul><ul><li>BOOL WINAPI GetTouchInputInfo( </li></ul><ul><li>__in HANDLE hTouchInput, // input event handle </li></ul><ul><li>__in UINT cInputs, // number of elts in the array </li></ul><ul><li>__out_ecount(cInputs) PTOUCHINPUT pInputs, // array of touch inputs </li></ul><ul><li>__in int cbSize); // sizeof(TOUCHINPUT) </li></ul><ul><li>typedef struct tagTOUCHINPUT { </li></ul><ul><li>LONG x; </li></ul><ul><li>LONG y; </li></ul><ul><li>HANDLE hSource; </li></ul><ul><li>DWORD dwID; </li></ul><ul><li>DWORD dwFlags; </li></ul><ul><li>DWORD dwMask; </li></ul><ul><li>DWORD dwTime; </li></ul><ul><li>ULONG_PTR dwExtraInfo; </li></ul><ul><li>DWORD cxContact; </li></ul><ul><li>DWORD cyContact; </li></ul><ul><li>} TOUCHINPUT, *PTOUCHINPUT; </li></ul><ul><li>typedef TOUCHINPUT const * PCTOUCHINPUT; </li></ul><ul><li>BOOL WINAPI CloseTouchInputHandle( </li></ul><ul><li>__in HANDLE hTouchInput); // input event handle </li></ul>
    20. 20. WM_TOUCH Future Windows 7 Build <ul><li>WM_TOUCH consolidated to a single message </li></ul>
    21. 21. Manipulations <ul><li>Best: Manipulations are a great foundation for touch-optimized experiences </li></ul><ul><li>Examples of higher level APIs/controls </li></ul><ul><li>Manipulations are: </li></ul><ul><ul><li>2D affine transformations (translate, scale, rotate) </li></ul></ul><ul><ul><li>Superset of supported gestures </li></ul></ul><ul><ul><li>Supports multiple concurrent manipulations </li></ul></ul><ul><li>Need a source of raw data: WM_TOUCH or RTS </li></ul><ul><li>Similar to Surface APIs </li></ul><ul><li>Interfaces: </li></ul><ul><ul><li>IManipulationProcessor </li></ul></ul><ul><ul><li>IManipulationEvents </li></ul></ul>
    22. 22. Inertia <ul><li>Provides basic physics </li></ul><ul><li>Works hand in hand with Manipulations </li></ul><ul><li>Interfaces: </li></ul><ul><ul><li>IInertiaProcessor </li></ul></ul><ul><ul><li>IManipulationEvents – Same event interface as Manipulations </li></ul></ul>
    23. 23. Multi-touch In .Net Framework <ul><li>For WPF: </li></ul><ul><ul><li>3.5 update near Windows 7 launch </li></ul></ul><ul><ul><ul><li>Multi-touch, gesture APIs </li></ul></ul></ul><ul><ul><li>4.0 release </li></ul></ul><ul><ul><ul><li>Multi-touch specific new controls </li></ul></ul></ul><ul><li>For WinForms (Windows 7 launch): </li></ul><ul><ul><li>Interop to native Win32 APIs – see sample </li></ul></ul><ul><ul><li>Use managed Manipulations/Inertia wrappers </li></ul></ul>
    24. 24. Multi-Touch In WPF <ul><li>UIElement and UIElement3D changes </li></ul><ul><ul><li>Gesture events (tracking) </li></ul></ul><ul><ul><li>Touch system gesture events (single) </li></ul></ul><ul><ul><li>Raw touch events </li></ul></ul><ul><ul><li>Touch and touch digitizer devices </li></ul></ul><ul><li>Multi-touch support in controls: </li></ul><ul><ul><li>ScrollViewer update to accept pan gestures </li></ul></ul><ul><ul><li>Base controls updated to be multi-touch aware </li></ul></ul><ul><ul><li>Multi-capture support </li></ul></ul><ul><ul><li>New multi-touch specific controls </li></ul></ul><ul><li>Compatible with Surface SDK 2.0 </li></ul>
    25. 25. Gestures In WPF <ul><li>Gesture events on UIElement & UIElement3D </li></ul><ul><ul><li>TouchGestureStarted, TouchGestureChanged, TouchGestureEnded </li></ul></ul><ul><ul><li>NOT routed, to enable multi-capture scenarios </li></ul></ul><ul><li>Gesture in WPF based on Manipulations </li></ul><ul><li>Enable Inertia at TouchGestureStarted </li></ul><ul><ul><li>Inertial movements as TouchGestureChanged Events </li></ul></ul><ul><li>Touch system gestures </li></ul>
    26. 26. Touch Events In WPF <ul><li>Touch Events on UIElement and UIElement3D </li></ul><ul><ul><li>TouchDown, TouchMove, TouchUp </li></ul></ul><ul><ul><li>TouchEnter, TouchLeave </li></ul></ul><ul><ul><li>Preview variants </li></ul></ul><ul><li>Promotion: Touch -> Stylus -> Mouse </li></ul><ul><li>Multi-Capture </li></ul><ul><ul><li>CaptureTouchDevice(TouchDevice) </li></ul></ul><ul><ul><li>ReleaseTouchDeviceCapture(TouchDevice) </li></ul></ul><ul><ul><li>ReleaseAllTouchDeviceCaptures() </li></ul></ul><ul><ul><li>TouchDeviceCollection TouchDevicesCaptured { get; } </li></ul></ul><ul><ul><li>TouchDeviceCollection TouchDevicesCapturedWithin { get; } </li></ul></ul><ul><ul><li>IsAnyTouchDeviceCaptured(); </li></ul></ul><ul><ul><li>IsAnyTouchDeviceCapturedWithin(); </li></ul></ul>
    27. 27. Tablet Platform Extensions <ul><li>Real-time Stylus (RTS) </li></ul><ul><ul><li>RTS is an excellent raw touch data provider </li></ul></ul><ul><ul><li>IRealTimeStylus3::MultiTouchEnabled() </li></ul></ul><ul><li>Getting multi-touch data via InkCollector is a little trickier. E.g.: </li></ul>ATOM m_atom = ::GlobalAddAtom(MICROSOFT_TABLETPENSERVICE_PROPERTY); m_dwProperty = TABLET_ENABLE_MULTITOUCHDATA ; ::SetProp(m_hwnd, (LPTSTR)m_atomPenService, (HANDLE)m_dwProperty);
    28. 28. UX Considerations
    29. 29. <ul><li>ISVs are enthusiastic about multi-touch! </li></ul><ul><li>We are working with several ISVs today on multi-touch applications, including: </li></ul><ul><ul><li>AutoDesk </li></ul></ul><ul><ul><li>Avoco Secure </li></ul></ul><ul><ul><li>Corel </li></ul></ul><ul><ul><li>Cyberlink </li></ul></ul><ul><ul><li>Fuel Industries </li></ul></ul><ul><ul><li>Identity Mine </li></ul></ul><ul><ul><li>Sonic </li></ul></ul>
    30. 30. Call To Action <ul><li>Decide which application scenarios make sense for touch </li></ul><ul><li>Decide which tier of investment is appropriate </li></ul><ul><li>Build something amazing! </li></ul>
    31. 31. Getting Started <ul><li>What do you need to start coding? </li></ul><ul><ul><li>Hardware: </li></ul></ul><ul><ul><ul><li>Multi-touch beta drivers for in-market devices: </li></ul></ul></ul><ul><ul><ul><ul><li>HP TouchSmart 2 Dell Latitude XT </li></ul></ul></ul></ul><ul><ul><li>Software: </li></ul></ul><ul><ul><ul><li>Window 7 M3 build – On the drive </li></ul></ul></ul><ul><ul><ul><li>Touch Platform SDK and samples – On the PDC drive </li></ul></ul></ul><ul><ul><ul><li>The Multi-point touch drivers are not yet a part of any OS. Your choice in hardware will impact the software needed. </li></ul></ul></ul><ul><li>WPF Multi-touch </li></ul><ul><ul><li>.NET 4.0 Preview Releases in 2009 </li></ul></ul><ul><ul><li>.NET 3.5 update to be announced </li></ul></ul>
    32. 33. <ul><li>LucidTouch: </li></ul><ul><ul><ul><li> </li></ul></ul></ul><ul><li>Obscura Digital: </li></ul><ul><ul><li>HP - D5: Multi-Touch wall </li></ul></ul><ul><ul><ul><li> </li></ul></ul></ul><ul><ul><li>VisionAire: Holographic Multi-Touch </li></ul></ul><ul><ul><ul><li> </li></ul></ul></ul>