Harnessing the Power of AI with Windows Ink
•Download as PPTX, PDF•
2 likes•496 views
Windows Ink is an extremely powerful feature in Windows 10 and learn how your application gets the fastest ink on Windows across even more devices. Learn to easily incorporating inline handwriting in to your application and use Ink Analysis to differentiate your application on Windows.
Recommended
More Related Content
Similar to Harnessing the Power of AI with Windows Ink
Similar to Harnessing the Power of AI with Windows Ink (20)
More from Windows Developer
More from Windows Developer (20)
Recently uploaded
Recently uploaded (20)
Harnessing the Power of AI with Windows Ink
- 4. type this
- 5. 3D in Windows 10 Windows Ink Microsoft Edge Faster to done Microsoft Office Cortana More creative, more productive 3rd in purchase intent 212m U.S. adults use analog or digital pen Processor/Ram/Storage Battery Life Pen Switch Storage from HDD to SSD Increase screen resolution Windows Hello 2-in-1 purchase intent when shown new scenarios
- 6. 3D in Windows 10 Windows Ink Microsoft Edge Faster to done Microsoft Office Cortana More creative, more productive 3rd in purchase intent 212m U.S. adults use analog or digital pen 10x more pen enabled devices than iOS or Android 2x More pen devices year over year 2x More user engagement year over year 8pts higher average NPS scores for pen attached systems 60% Top 10 revenue producing apps use Windows Ink 50% More apps using Windows Ink in Windows Store
- 9. UWP and Win32 The Windows Ink Platform Low Level APIs / Engine Controls HID Hardware Pointer (PointerPoint, WM_Pointer) Radial Controller DirectInk Ink Presenter Rendering InkStrokeContainer Low Latency Beautiful Ink Core WetStroke Update Source Custom Dry Built-in Brushes and Drawing Attributes Acceleration Off-thread Input Stroke Creation Prediction Smoothing XAML Controls InkCanvas InkToolbar TextBox SmartInk InkAnalysis Local Ink Classification Layout Analysis Handwriting Recognition Shape Recognition InkAnalysis Cloud Ink Classification Layout Analysis Handwriting Recognition Shape Recognition Cloud Service APIs
- 10. UWP and Win32 The Windows Ink Platform Low Level APIs / Engine HID Hardware Pointer (PointerPoint, WM_Pointer) Radial Controller DirectInk Ink Presenter Rendering InkStrokeContainer Low Latency Beautiful Ink Core WetStroke Update Source Custom Dry Built-in Brushes and Drawing Attributes Acceleration Off-thread Input Stroke Creation Prediction Smoothing SmartInk InkAnalysis Local Ink Classification Layout Analysis Handwriting Recognition Shape Recognition InkAnalysis Cloud Ink Classification Layout Analysis Handwriting Recognition Shape Recognition Cloud Service APIs Controls XAML Controls InkCanvas InkToolbar TextBox
- 11. <InkCanvas x:Name=“m_inkCanvas”/> <InkToolbar TargetInkCanvas=“{x:Bind m_inkCanvas}”/> XAML Ink Canvas XAML Ink Toolbar
- 13. Delete Words Delete Lines Insert words into a sentence Join two words Add a new line Overwrite Words
- 14. UWP and Win32 The Windows Ink Platform Low Level APIs / Engine HID Hardware Pointer (PointerPoint, WM_Pointer) Radial Controller SmartInk InkAnalysis Local Ink Classification Layout Analysis Handwriting Recognition Shape Recognition InkAnalysis Cloud Ink Classification Layout Analysis Handwriting Recognition Shape Recognition Cloud Service APIs Controls XAML Controls InkCanvas InkToolbar TextBox DirectInk Ink Presenter Rendering InkStrokeContainer Low Latency Beautiful Ink Core WetStroke Update Source Custom Dry Built-in Brushes and Drawing Attributes Acceleration Off-thread Input Stroke Creation Prediction Smoothing
- 15. • Ballpoint Pen • Highlighter • Tilt Pencil Model how analog pencil works Natural pencil inking – dynamic range, layering • Erasing Stroke erase – erases full stroke Erase all • Stencils Built-in Brushes and Erasing
- 16. Pen Hardware & Ecosystem Update Ink Acceleration Using GPU preemption, Windows no longer has to wait for the next scanout to get ink on the screen. Microsoft Pen Program Microsoft Pen
- 17. UWP and Win32 The Windows Ink Platform Low Level APIs / Engine HID Hardware Pointer (PointerPoint, WM_Pointer) Radial Controller Controls XAML Controls InkCanvas InkToolbar TextBox DirectInk Ink Presenter Rendering InkStrokeContainer Low Latency Beautiful Ink Core WetStroke Update Source Custom Dry Built-in Brushes and Drawing Attributes Acceleration Off-thread Input Stroke Creation Prediction Smoothing SmartInk InkAnalysis Local Ink Classification Layout Analysis Handwriting Recognition Shape Recognition InkAnalysis Cloud Ink Classification Layout Analysis Handwriting Recognition Shape Recognition Cloud Service APIs
- 19. Ink Analysis Root WritingRegion Drawing Drawing DrawingWritingRegion … … ListItem x2Line Line InkWord InkWord InkWord InkWord … … InkBullet Paragraph
- 20. For Each Text Node • RecognizedText – The string representing the recognized text in the node Note: Line, ListItem, and Paragraph nodes also have a RecognizedText property • TextAlternates – The list of alternates for the recognized text • Understand slanted text via RotatedBoundingRect Text Recognition Avani hello, halo, hollo
- 21. Shape Recognition Triangles Equilateral Triangles Isosceles Triangles Right Triangles Squares Rectangles Diamonds Quadrilaterals Parallelograms TrapezoidsPentagons Hexagons Circles Ellipses Ink Drawing
- 22. Microsoft Confidential Demo - InkEditor
- 23. Important InkAnalysis Concepts and Tips
- 24. Microsoft Confidential Stateless Ink Analysis Find the nodes you’re interested in Extract strokes Send to InkAnalyzer Analyze the strokes Do cool stuff var inkAnalyzer = new InkAnalyzer(); private async void AnalyzeButton_Clicked(object sender, RoutedEventArgs e) { if (!inkAnalyzer.IsAnalyzing) { inkAnalyzer.ClearDataForStrokes(strokes); var strokes = inkCanvas.InkPresenter.StrokeContainer.GetStrokes(); inkAnalyzer.AddDataForStrokes(strokes); var results = await inkAnalyzer.AnalyzeAsync(); } }
- 25. Microsoft Confidential Stateful (Incremental) Ink Analysis Find the nodes you’re interested in Strokes changed Update InkAnalyzer Analyze the strokes Do cool stuff private void InkPresenter_StrokesCollected(InkPresenter sender, InkStrokesCollectedEventArgs args) { inkAnalyzer.AddDataForStrokes(args.Strokes); } private void InkPresenter_StrokesEraseded(InkPresenter sender, InkStrokesErasedEventArgs args) { foreach (var stroke in args.Strokes) { inkAnalyzer.RemoveDataForStroke(stroke.Id); } }
- 26. Microsoft Confidential Stateful (Incremental) Ink Analysis Find the nodes you’re interested in Strokes changed Update InkAnalyzer Analyze the strokes Do cool stuff private void StrokesUpdated(IReadOnlyList<uint> updatedStrokeIds) { var strokeContainer = inkCanvas.InkPresenter.StrokeContainer; foreach(var id in updatedStrokeIds) { inkAnalyzer.ReplaceDataForStroke(strokeContainer.GetStrokeById(id)); } } private async void AnalyzeButton_Clicked(object sender, RoutedEventArgs e) { if (!inkAnalyzer.IsAnalyzing) { var results = await inkAnalyzer.AnalyzeAsync(); } }
- 27. Microsoft Confidential var results = await _inkAnalyzer.AnalyzeAsync(); var root = inkAnalyzer.AnalysisRoot; // Query for particular node kind var drawings = root.FindNodes(InkAnalysisNodeKind.InkDrawing); var paragraphs = root.FindNodes(InkAnalysisNodeKind.Paragraph); var words = root.FindNodes(InkAnalysisNodeKind.InkWord); // Document tree traversal foreach (var paragraph in writingRegion.Children) {…} foreach (var word in line.Children) {…} var line = word.Parent; Query Document Structure Find the nodes you’re interested in Strokes changed Update InkAnalyzer Analyze the strokes Do cool stuff
- 28. Microsoft Confidential var results = await _inkAnalyzer.AnalyzeAsync(); var root = inkAnalyzer.AnalysisRoot; var drawings = root.FindNodes(InkAnalysisNodeKind.InkDrawing); foreach (var drawing in drawings) { var shape = drawing as InkAnalysisInkDrawing; if(shape.DrawingKind == InkAnalysisDrawingKind.Circle) { convertInkToXAMLShape(shape); var strokeIds = shape.GetStrokeIds(); inkAnalyzer.removeDataForStrokes(strokeIds); foreach(var id in strokeIds) { strokeContainer.GetStrokeById(id).Selected = true; } strokeContainer.DeleteSelected(); } } Mapping between strokes and InkAnalysisNode
- 29. Microsoft Confidential Use Prior Knowledge Find the nodes you’re interested in Strokes changed Update InkAnalyzer Analyze the strokes Do cool stuff private void InkPresenter_StrokesCollected(InkPresenter sender, InkStrokesCollectedEventArgs args) { inkAnalyzer.AddDataForStrokes(args.Strokes); foreach (var stroke in args.Strokes) { inkAnalyzer.SetStrokeDataKind(stroke.Id, InkAnalysisStrokeKind.Writing); //inkAnalyzer.SetStrokeDataKind(stroke.Id, InkAnalysisStrokeKind.Drawing); } }
- 32. Microsoft Confidential Cloud Service For Smart Ink Consistent Ink Recognition Experience Across Devices and Platforms Faster and Frequent Improvements to the core recognition capabilities Personalized Recognition with the Microsoft Cloud and Microsoft Graph Compute Power for larger ML models for better accuracy
- 33. Microsoft Confidential Reachouttous! • Learn more: • aka.ms/WindowsInkSample • aka.ms/DigitalInkBlog • aka.ms/WinHEC2017 • aka.ms/InputScope • aka.ms/InkPresenterClass • aka.ms/HandwritingView • Code Samples • aka.ms/SimpleInkSample • aka.ms/ComplexInkSample • aka.ms/InkAnalysisSample Interested in being a co-engineering partner for our cloud solution, or have an ink related question for the team? Contact us here: WindowsInk@Microsoft.com Follow us on Twitter for the latest ink and pen news from our team! Twitter: WindowsInk
Editor's Notes
- 2
- 4
- 5
- 6
- 7
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 23
- 24
- 25
- 26
- 32