Windows 7 Multi-touch Support & Application DevelopmentLynn LangitDeveloper Evangelist – Microsofthttp://blogs.msdn.com/SoCalDevGal

AgendaMulti-touch overview & Demos!Control Panel settings & UX guidelinesTouch scenarios, “Good, Better, Best”Platform details:Native Win32 gesture and touch supportManipulations and InertiaWPF supportFuture Windows 7 release and later .NET 4.0 release

Richer Application ExperiencesRibbon User InterfaceSensor and LocationEnhanced GraphicsMulti-touch

Windows Paint and IE8Windows 7 built-in Multi-touchDemo or Video

Multi - Touch ScenariosThat led to four areas of investment:Developer Platform: At the root is the touch developer platform that exposes touch APIs for any application UI Enhancements: Focusing on the core scenarios, many parts of the core UI have been optimized for touch experiences. Gestures: Multi-touch gestures have been added to enable consistent panning and zooming in most applications.Applications: A set of multi-touch focused applications that demonstrate the power of touch. These will ship separate from Win7.We’ve identified several key scenarios for multi-touch:Navigating and consuming the webReading & sorting emailViewing photosPlaying casual gamesConsuming music and videoNavigating files & arranging windowsUsing Office applicationsAll focused on consumption

Multi-touch Hardware AvailabilityHP TouchSmart All-in-One PCNextWindow digitizerDrivers: http://www.nextwindow.com/windriver/index.htmlNextWindow Touch OverlayNextWindow digitizerDrivers: http://www.nextwindow.com/windriver/index.htmlDell Latitude XT or Tx2N-trig digitizerDrivers: in progress and are targeted for Beta availability.HP TouchSmart tx2z seriesN-trig digitizerDrivers: in progress and are targeted for Beta availability.

Multi- Touch Dev RoadmapWindows 7 ReleaseNOW>Windows 7>NET 4.0 / Surface 2.0 ReleaseNativeWin32ApplicationWinFormsApplicationWPF ApplicationSurface ApplicationSurface SDK1.0Surface SDK 2.0Multi-Touch ControlsSurface Multi-Touch Controls & APIMulti-Touch APIWPF 3.5 SP1WPF 4.0WPF 3.5Managed Wrapper and InteropManaged Wrapper and InteropMulti-Touch API and ControlsSurface HardwareWindows 7Surface HardwareWindows VistaWindows 7Multi-Touch API

Control Panel – Pen and Touch

Touch Dev Terms:Fingers when indicating user's hand as input deviceIndicate number of hands and number of fingersTap (and double-tap) instead of click when documenting procedures specific to using a finger or pen. Tap means to press the screen and then lift before a hold time. It may or may not be used to generate a mouse click. For interactions that don't involve the finger or pen, continue to use click. Touchscreen and touchpad are single, compound words.

Gestures & ManipulationsA Gesture is…a quick movement of one or more fingers on a screen that the computer interprets as a command, rather than as a mouse movement (or writing or drawing)Pre-defined subset of all possible touch typesExample is a ‘flick’ gestureA Manipulation is…a real-time, physical handling of an objectdiffers from a gesture in that the input corresponds directly to how the object would react naturally to the action in the real world. Example is a photo viewing application might allow users to manipulate a photo by moving, zooming, resizing, and rotating the image. Multi touch manipulations use multiple contact points simultaneously.

Development Tiers

Multi-Touch SoftwareGood-Better-BestGood: No specific touch APIs are used but the application UI is appropriately sized and works well with the built-in gesturesBetter: The gesture APIs are supported to give smooth natural interactions – SINGLE gestureBest: Deep touch focused experiences designed to take advantage of Multi-touch capturing manipulations – MULTIPLE touches

UX ConsiderationsGeneral guidelines:Big targetsWhitespaceAvoid hoverSee UX guidelinesBe aware of hardware:Form factorAvoid on-hover UIEdges, jitter, etcGesture guidelines:Use common gesturesGestures need to be intuitive and naturalMoving away from shortcut style gestures to manipulations and direct manipulation

Predefined Gestures – Drag or PanPlace one or two fingers in the application window and drag in the direction you want

Zoom and RotateRotateTouch the image with two fingers and turn fingers in a circleZoomTouch the image with two fingers and move them closer or further apart

Two Finger Tapand Finger RollTwo Finger TapTap once with both fingersFinger RollPlace one finger on the screen, place second finger on the screen, lift the second finger, and then lift the first finger

Multi-Touch in .NET FrameworkFor WPF:Interop sample library for .NET Framework 3.5Single gesture sequences can add inertia –OR-Multi-touch, series of one or more manipulations can add/define inertia.NET Framework 4.0 releaseMulti-touch specific new controlsFor WinForms (Windows 7 launch):Interoperability to native Win32 APIs – TouchWrapper

Windows 7 Integration LibraryWindows7.Multitouch.Handler is base class for Touch & Gesture handlersUse Factory to create one of the handlersFor WinFormsmanaged Win32 hWnd, & WPF gesture support handler subclass Window (hWnd)For WPF touch support, use stylus event with help of Factory.EnableStylusEvents() method

Setting Gesture ConfigurationBy default application receives all gesture messages except rotateYou can configure which gestures will be sent:BOOL WINAPI SetGestureConfig(HWND hWnd, DWORD dwReserved, UINT cIDs, PGESTURECONFIG pGestureConfig, UINT cbSize)This API gets an array of GESTURECONFIG structurescbSizeis the size of the array, cIDs is the lengthdwReservedshould be 0

The GESTURECONFIGStructuretypedefstruct _GESTURECONFIG { DWORD dwID; //One of GID_* (0, ZOOM, PAN, // ROTATE, TWOFINGERTAP, ROLLOVER DWORD dwWant; //One of GC_* (ALLGESTURES, // ZOOM, PAN, PAN_*, ROTATE, // TWOFINGERTAP, ROLLOVER DWORD dwBlock; //Same arguments as dwWant // but block the message } GESTURECONFIG, *PGESTURECONFIG;

Dynamically Changing Gesture ConfigsThe WM_GESTURENOTIFYmessage is sent to indicate that a gesture message is about to be receivedcase WM_GESTURENOTIFY:{ GESTURECONFIG gc = {0,GC_ALLGESTURES,0};SetGestureConfig(hWnd, 0, 1, &gc,sizeof(GESTURECONFIG));} break;

Registering for Touch XOR Gesture MessagesBOOL InitInstance(HINSTANCE hInstance, intnCmdShow){ HWND hWnd;hWnd = CreateWindow(…) if (!hWnd) return FALSE; //We will receive WM_GESTURE messages by default //Calling RegisterTouchWindow stops gesture message if (bTouchMessages && bMultiTouch) RegisterTouchWindow(hWnd, 0); ShowWindow(hWnd, nCmdShow);UpdateWindow(hWnd); return TRUE; }

Configure hWnd To Get All GesturesGESTURECONFIG gestureConfig;gestureConfig.dwID = 0;gestureConfig.dwBlock = 0;gestureConfig.dwWant = GC_ALLGESTURES;SetGestureConfig(hWnd, 0, 1, &gestureConfig, sizeof(gestureConfig));

WM_GESTURE MessageGESTUREINFO gi;gi.cbSize = sizeof(GESTUREINFO);gi.dwFlags = 0; gi.ptsLocation.x = 0; …GetGestureInfo((HGESTUREINFO)lParam, &gi);// now interpret the gestureswitch (gi.dwID){ case GID_ZOOM: // Code for zooming goes here break; case GID_PAN: // Code for panning goes here break; case GID_ROTATE: …}CloseGestureInfoHandle((HGESTUREINFO)lParam);

GESTUREINFO Structuretypedefstruct _GESTUREINFO { UINT cbSize; DWORD dwFlags; //GF_* (BEGIN, INERTIA, END) DWORD dwID; //GID_* (BEGIN, END, ZOOM, PAN, //ROTATE, TWOFINGERTAP, ROLLOVER) HWND hwndTarget; POINTS ptsLocation; DWORD dwInstanceID; DWORD dwSequenceID; ULONGLONG ullArguments; //8 Bytes Gesture Arg UINT cbExtraArgs;} GESTUREINFO, *PGESTUREINFO;

Decoding Gesture(dwFlags & GF_BEGIN) != 0Indicates that this is a new gestureIf dwFlags is 0 (or GF_INERTIA)For Pan:We have to calculate the translation delta since last messageFor Rotate:We have to calculate the angle delta since last messageFor Zoom:We have to calculate the zoom factor delta since last message

Using the GestureHandler public MainForm() {InitializeComponent(); … _gestureHandler = TouchBridge.Handler.CreateHandler<TouchBridge.GestureHandler>(this); _gestureHandler.Pan += ProcessPan; _gestureHandler.PanBegin += ProcessPan; _gestureHandler.PanEnd += ProcessPan; _gestureHandler.Rotate += ProcessRotate; _gestureHandler.RollOver += ProcessRollOver; _gestureHandler.TwoFingerTap += ProcessTwoFingerTap; _gestureHandler.Zoom += ProcessZoom;}

MTGestureWindows7.Multitouch.GestureHandlerDemo - WinForms

ManipulationsManipulations are the foundation for touch-optimized experiences. 2D affine transformations (translate, scale, rotate)Superset of supported gesturesSupports multiple concurrent manipulationsRaw Data source: WM_TOUCHSimilar to Surface APIsInterfaces IManipulationProcessor, IManipulationEvents

Windows7 Integration Library ManipulationA .NET wrapperYou must ‘opt in’ to get WM_TOUCH data (now either touch OR gesture info, may change in final bits)You need to forward touch eventsYou get back manipulation eventsBe ready to process translation, rotation and expansion all togetherIf there are many objects on the screenFind a good heuristic to decide which object to moveWindows7.Multitouch.dll (& .WinForms, .WPF)

WM_TOUCHSemantically similar to mouse messagesConveys raw touch data to Win32 appsScenario examples:Finger paintingcustom gesturesfeeding higher level controls

Handling WM_TOUCHUINT cInputs = LOWORD(wParam);PTOUCHINPUT pInputs = new TOUCHINPUT[cInputs];if (pInputs != NULL){ if (GetTouchInputInfo((HTOUCHINPUT)lParam, cInputs, pInputs,sizeof(TOUCHINPUT))) { // process pInputs } else { … } // error handling }else { … } // error handling, presumably out of memoryif (!CloseTouchInputHandle((HTOUCHINPUT)lParam)){ // error handling}

Using the TouchHandlerpublic MainForm(){InitializeComponent(); _touchHandler = Factory.CreateHandler<TouchHandler>(this); _touchHandler.TouchDown += OnTouchDownHandler; _touchHandler.TouchMove += OnTouchMoveHandler; _touchHandler.TouchUp += OnTouchUpHandler; Paint += new PaintEventHandler(this.OnPaintHandler);}

Using Manipulation public MainForm() {InitializeComponent(); _touchHandler = Factory.CreateHandler<TouchHandler>(this); _processor = newManipulationProcessor(ProcessorManipulations.ALL); _objectList = new List<DrawingObject> { … }; _touchHandler.TouchDown += (s,e ) => { _processor.ProcessDown((uint)e.Id, e.Location); }; _touchHandler.TouchUp += (s, e) => { _processor.ProcessUp((uint)e.Id, e.Location); }; _touchHandler.TouchMove += (s, e) => { _processor.ProcessMove((uint)e.Id, e.Location); }; _processor.ManipulationDelta += ProcessManipulationDelta; _processor.PivotRadius = 2;}

Using Manipulation private void ProcessManipulationDelta(object sender,ManipulationDeltaEventArgs e) {DrawingObjectobj = FindObject(Point.Round(e.Location)); if (obj == null) return;obj.Move(e.TranslationDelta.ToSize());obj.Rotate(e.RotationDelta, Point.Round(e.Location));obj.Zoom(e.ScaleDelta, Point.Round(e.Location)); Invalidate(); }

ManipulationmtManipulationDemo

Windows7 Integration Library InertiaIntegration library includes Manipulation Processor with Inertia capabilities (wraps COM object)Derived from ManipulationProcessorHas InertiaProcessor property that returns the inner inertia processorYou should find good values as an input to the inertia processorInitial Translation, Rotation and Expansion VelocitiesDeceleration velocity for each of themInterfaces:IInertiaProcessorIManipulationEvents – Same event interface as manipulations

InertiamtInertiaDemo

Other demosMicrosoft Touch Pack for Windows 7Demo

Multi-Touch in WPF 4.0UIElement and UIElement3D changesGesture events (tracking)Touch system gesture events (single)Raw touch eventsMulti-touch support in controls:ScrollViewer update to accept pan gesturesBase controls updated to be multi-touch awareMulti-capture supportNew multi-touch specific controlsCompatible with Surface SDK 2.0

Getting StartedWhat do you need to start coding?Hardware:Multi-touch beta drivers for in-market devices:HP TouchSmart 2Dell Latitude XTSoftware:Window 7 RC, RTM Oct 22, 2009Touch Platform SDK and samples WPF Multi-touch.NET 4.0 Preview Releases in 2009.NET 3.5 update to be announced

PDC08 ResourcesRelated PDC08 Sessions:PC46 – WPF RoadmapPC17 – Developing for Microsoft SurfaceHands on Lab:PCHOL18: Developing Multi-touch Applications in Windows 7PDC Whitepaper – Windows 7: Reach Out and Touch It!Contact the product group directly:wintouch@microsoft.com

Other ResourcesWindows 7 Training Kit for Developers MSDN Channel 9 videos – for Windows 7My blog series – http://blogs.msdn.com/SoCalDevGal#Win7DevSeriesMy MSDN show – MSDN geekSpeakMy Facebook group ‘Windows 7 Developers’