Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Exploring Ink Analysis


Published on

Published in: Technology, News & Politics
  • Login to see the comments

Exploring Ink Analysis

  1. 2. Frank La Vigne, Tablet PC MVP Lead Designer Applied Information Systems Reston, VA APP329 Practical Ink Analysis
  2. 3. Who Is Frank La Vigne? <ul><li>14 years of professional experience </li></ul><ul><li>Started career writing Microsoft Visual Basic 3 applications </li></ul><ul><li>Microsoft MVP: Tablet PC </li></ul><ul><li>User Group & Code Camp speaker </li></ul><ul><li>President Emeritus of Richmond.NET User Group </li></ul><ul><li>Assistant Vice President of Technology for INETA North America </li></ul><ul><li>Lead Designer @ Applied Information Sciences </li></ul><ul><ul><li>Blog: </li></ul></ul><ul><ul><li>Corporate: </li></ul></ul>
  3. 4. Agenda <ul><li>Digital Ink Basics </li></ul><ul><li>Limitations of Word Recognition </li></ul><ul><li>Moving from Strings to Documents </li></ul><ul><li>Meet the InkAnalyzer </li></ul><ul><li>Improving Recognition Results </li></ul><ul><li>Detecting Relationships in Ink </li></ul><ul><li>Shape Recognition </li></ul><ul><li>Additional Resources </li></ul><ul><li>Questions and Answers </li></ul>
  4. 5. Digital Ink Basics <ul><li>You are Tablet PC developer, or want to be one </li></ul><ul><li>You have at least a basic understanding of Ink </li></ul><ul><li>You yearn for more out of the recognizer APIs </li></ul><ul><li>You want to know the theory, but need practical solutions, not just abstract concepts </li></ul><ul><li>This presentation makes the following assumptions: </li></ul>
  5. 6. Digital Ink <ul><li>Ink object </li></ul><ul><li>Central to the Tablet platform </li></ul><ul><li>Contains stroke data </li></ul><ul><li>Contains all persistence mechanisms </li></ul><ul><li>Extensible via the ExtendedProperties collection </li></ul>
  6. 7. Strokes <ul><li>Stroke object </li></ul><ul><li>Represents a “stroke of the pen” </li></ul><ul><ul><li>One pen down, pen move, pen up sequence </li></ul></ul><ul><li>Has a rich API </li></ul><ul><ul><li>PacketData, Points, Bezier Points, BezierCusps </li></ul></ul><ul><li>Extensible via the ExtendedProperties collection </li></ul><ul><li>Controls rendering properties via the DrawingAttributes object </li></ul><ul><li>Resides in the Ink object in a collection </li></ul>
  7. 8. Capturing Ink <ul><li>Ink can be captured via: </li></ul><ul><ul><li>InkOverlay </li></ul></ul><ul><ul><li>InkCollector </li></ul></ul><ul><ul><li>InkCanvas (WPF) </li></ul></ul><ul><li>InkCollector provides core functionality </li></ul><ul><li>InkOverlay adds easy to use editing modes (Ink, Select, Erase) </li></ul><ul><li>Each contains one Ink object </li></ul><ul><li>Full set of event notifications </li></ul>
  8. 9. Recognizing Ink <ul><li>First-generation APIs placed emphasis on handwriting recognition </li></ul><ul><li>Ink can be converted to text rather easily </li></ul><ul><ul><li>Call the ToString method in the Strokes collection </li></ul></ul><ul><li>Recognition is centered on words </li></ul>string recognitionResults = inkCollector.Ink.Strokes.ToString();
  9. 10. Digital Ink Basics
  10. 11. Word Recognition <ul><li>Handwriting recognition engine works great </li></ul><ul><li>However, it was not designed to be a document parser. </li></ul><ul><li>Accordingly, it doesn’t understand document structure </li></ul><ul><ul><li>Bullet items </li></ul></ul><ul><ul><li>Paragraphs </li></ul></ul><ul><li>Callouts, underlines, diagrams, shapes </li></ul>
  11. 12. Moving from Strings to Documents <ul><li>After a more than a few words, ink content becomes a document rather than a string. </li></ul>Developers need an API to do the “heavy lifting” and convert the ink to the right to text and preserve the structure.
  12. 13. Meet the InkAnalyzer <ul><li>InkAnalyzer class contains everything developers need to perform ink analysis </li></ul><ul><ul><li>Methods </li></ul></ul><ul><ul><li>Properties </li></ul></ul><ul><ul><li>Events </li></ul></ul><ul><li>Does everything RecognizerContext does and more </li></ul><ul><li>Places the InkDivider and RecognizerContext under one API </li></ul><ul><li>InkAnalyzer automatically determines writing vs. drawing strokes </li></ul><ul><ul><li>Can be overridden via code </li></ul></ul>
  13. 14. InkAnalyzer in Action <ul><li>Microsoft Office OneNote 2007 supports ink analysis </li></ul>
  14. 15. Coding the InkAnalyzer <ul><li>InkAnalyzer constructor takes Ink object it collects ink from and the control associated with the InkAnalyzer </li></ul><ul><li>Call the AddStrokes method to tell the InkAnalyzer what ink to analyze </li></ul><ul><li>This is not redundant (really) </li></ul>InkAnalyzer analyzer = new InkAnalyzer(this.collector.Ink, this); analyzer.AddStrokes(this.collector.Ink.Strokes);
  15. 16. Coding the InkAnalyzer (cont'd) <ul><li>To remove Strokes </li></ul><ul><li>To invoke the Analyzer </li></ul><ul><li>To invoke the Analyzer on a background thread </li></ul>analyzer.RemoveStroke (this.collector.Ink.Strokes); analyzer.Analyze(); analyzer.BackgroundAnalyze();
  16. 17. Ink Analysis Results <ul><li>The InkAnalyzer produces a tree of ContextNode objects </li></ul><ul><li>Starts with RootNode that includes all analyzed Stroke collections </li></ul><ul><li>Stroke collections get smaller the further down the tree </li></ul><ul><li>Stroke collections have a ContextNodeType classification assigned to them by the InkAnalyzer </li></ul><ul><li>The FindNodesOfType method searches the entire tree for ContextNodes of a given type </li></ul>
  17. 18. ContextNode Types <ul><li>ContextNodeType s returned from the InkAnalyzer </li></ul><ul><ul><li>Root </li></ul></ul><ul><ul><li>WritingRegion </li></ul></ul><ul><ul><li>Paragraph </li></ul></ul><ul><ul><li>Line </li></ul></ul><ul><ul><li>InkBullet </li></ul></ul><ul><ul><li>InkDrawing </li></ul></ul><ul><ul><li>InkWord </li></ul></ul>
  18. 19. InkAnalyzer Results
  19. 20. Recursing the Tree <ul><li>// Kick off the Recursion with the RootNode </li></ul><ul><li>TraverseNodes ( inkAnalyzer.RootNode , sbNodes, 0); </li></ul><ul><li>// Parse ContextNode Tree </li></ul><ul><li>private void TraverseNodes ( ContextNode contextNode , StringBuilder nodes, int Depth) </li></ul><ul><li>{ </li></ul><ul><li>nodes.Append(string.Empty.PadLeft(Depth * 4, ' ')); </li></ul><ul><li>nodes.Append(contextNode.GetType().ToString() + &quot; &quot;); </li></ul><ul><li>nodes.Append(&quot;[&quot; + </li></ul><ul><li>contextNode.Strokes.ToString() </li></ul><ul><li>+ &quot;]&quot;); </li></ul><ul><li>nodes.Append(&quot; &quot;); </li></ul><ul><li>foreach (ContextNode subNode in contextNode.SubNodes ) </li></ul><ul><ul><li>{ </li></ul></ul><ul><li>TraverseNodes (subNode, nodes, Depth + 1); </li></ul><ul><li>} </li></ul><ul><li>} </li></ul>
  20. 21. InkAnalysis Sample App
  21. 22. Wouldn't It Be Great If… <ul><li>XML is: </li></ul><ul><li>Great for preserving structure </li></ul><ul><li>Widely understood </li></ul><ul><li>Easily transformed into other formats </li></ul><ul><li>With ink analysis, we can convert the ContextType nodes to XML nodes, preserving the structure and the content </li></ul><ul><li>… handwritten documents could be converted to XML? </li></ul>
  22. 23. Ink to XML Converter
  23. 24. Improving Recognition Results <ul><li>Hints and Factoids </li></ul><ul><ul><li>Apply to an entire Ink collection object </li></ul></ul><ul><ul><li>Apply to a specific region on the Ink collection object </li></ul></ul><ul><ul><li>Biases results to certain types of data </li></ul></ul><ul><ul><ul><li>Numeric </li></ul></ul></ul><ul><ul><ul><li>URL </li></ul></ul></ul><ul><ul><ul><li>E-mail Address </li></ul></ul></ul><ul><li>Custom word lists </li></ul><ul><ul><li>Limits the possibilities the recognizer needs to consider </li></ul></ul><ul><ul><li>Results can be limited to the word list </li></ul></ul>
  24. 25. Factoids in Action <ul><li>InkBrowser Application </li></ul>
  25. 26. Hints & Factoids <ul><li>There are 32 different types of Factoids, including: </li></ul><ul><ul><li>Currency </li></ul></ul><ul><ul><li>Date </li></ul></ul><ul><ul><li>Default </li></ul></ul><ul><ul><li>Digit </li></ul></ul><ul><ul><li>Email </li></ul></ul><ul><ul><li>Filename </li></ul></ul><ul><ul><li>LowerChar </li></ul></ul><ul><ul><li>Number </li></ul></ul><ul><ul><li>NumberSimple </li></ul></ul><ul><ul><li>OneChar </li></ul></ul><ul><ul><li>Percent </li></ul></ul><ul><ul><li>PostalCode </li></ul></ul><ul><ul><li>PuncChar </li></ul></ul><ul><ul><li>SystemDictionary </li></ul></ul><ul><ul><li>Telephone </li></ul></ul><ul><ul><li>Time </li></ul></ul><ul><ul><li>Web </li></ul></ul><ul><ul><li>WordList </li></ul></ul>
  26. 27. Coding Hints & Factoids <ul><li>Created an AnalysisHintNode </li></ul><ul><li>Set the Factoid type to Web </li></ul><ul><li>Use some Coercion </li></ul><ul><li>Call the InkAnalyzer.Analyze method </li></ul>AnalysisHintNode hintNode = inkAnalyzer.CreateAnalysisHint(); hintNode.Factoid = Factoid.Web; hintNode.CoerceToFactoid = true;
  27. 28. InkWeb Browser
  28. 29. Using a Custom Word List <ul><li>Create a String array & an AnalysisHintNode </li></ul><ul><li>Set the Factoid type to Web </li></ul><ul><li>Insert string array using SetWordList method </li></ul>string[] wordListArray = { “Blogging&quot;, “AJAX&quot;, “Podcast“, “PVR” }; AnalysisHintNode hintNode = inkAnalyzer.CreateAnalysisHint(); hintNode.Factoid = Factoid.WordList; hintNode.SetWordlist(wordListArray);
  29. 30. Ink to Blog Application
  30. 31. Detecting Relationships in Ink <ul><li>InkAnalyzer can understand: </li></ul><ul><ul><li>Callouts </li></ul></ul><ul><ul><li>Underlines </li></ul></ul><ul><ul><li>Diagrams </li></ul></ul><ul><li>The ContextLink object represents the relationship between two non–Parent-Child ContextNode objects </li></ul>
  31. 32. ContextLink Object <ul><li>Represents relationships between two nodes </li></ul><ul><li>Important properties </li></ul><ul><ul><li>SourceNode </li></ul></ul><ul><ul><ul><li>The ContextNode initiating the relationship (blue) </li></ul></ul></ul><ul><ul><li>DestinationNode </li></ul></ul><ul><ul><ul><li>The ContextNode the source node points to (red) </li></ul></ul></ul><ul><ul><li>LinkDirection </li></ul></ul><ul><ul><ul><li>Type of link the ContextLink is set to </li></ul></ul></ul>
  32. 33. Coding the ContextLink Object <ul><li>Use the Links property of the ContextNode object </li></ul><ul><ul><li>It is a ContextLink collection </li></ul></ul>foreach (ContextLink cl in contextNode. Links ) { // What's doing the pointing cl.SourceNode.Strokes.ModifyDrawingAttributes(new DrawingAttributes( Color.Blue )); // What's being highlighted cl.DestinationNode.Strokes.ModifyDrawingAttributes( new DrawingAttributes( Color.Red )); }
  33. 34. Finding Context Links
  34. 35. Shape Recognition <ul><li>The InkAnalyzer can recognize the following shapes: </li></ul><ul><ul><li>Ellipse </li></ul></ul><ul><ul><li>Circle </li></ul></ul><ul><ul><li>Triangle </li></ul></ul><ul><ul><li>IsoscelesTriangle </li></ul></ul><ul><ul><li>EquilateralTriangle </li></ul></ul><ul><ul><li>RightTriangle </li></ul></ul><ul><ul><li>Quadrilateral </li></ul></ul><ul><ul><li>Rectangle </li></ul></ul><ul><ul><li>Square </li></ul></ul><ul><ul><li>Diamond </li></ul></ul><ul><ul><li>Trapezoid </li></ul></ul><ul><ul><li>Parallelogram </li></ul></ul><ul><ul><li>Pentagon </li></ul></ul><ul><ul><li>Hexagon </li></ul></ul>
  35. 36. Coding Shape Recognition <ul><li>The InkDrawingNode type represents either enumerated shapes or abstract drawings </li></ul><ul><li>InkDrawingNode s do not have children </li></ul><ul><li>Drawings are generally not recognized as writing </li></ul>ContextNodeCollection drawingNodes = inkAnalyzer.FindNodesOfType(ContextNodeType.InkDrawing); string results = &quot;Drawings detected: &quot;; foreach (InkDrawingNode drawingNode in drawingNodes) { results += drawingNode. GetShapeName() + &quot; &quot;; } MessageBox.Show(results);
  36. 37. Shape Recognition Game
  37. 38. Additional Resources <ul><li>Ink Analysis Resources </li></ul><ul><li>“ Exploring Ink Analysis:” article in CoDe Focus </li></ul><ul><li>Ink Analyzer Explorer </li></ul><ul><li>Tablet PC Development Resources </li></ul><ul><li>Mobile PC Developer Center </li></ul><ul><li>My Blog </li></ul>
  38. 39. Questions & Answers
  39. 40. While You're Here Fill out your session evaluation Enter to win a Windows Mobile ® phone or Zune™ Geek out with a huge rack of servers Enterprise Mobility in Action in the Expo Hall Go on a date ISV Speed Dating lets you pitch your app to Mobile Operators and OEMs (May 1 only) Meet the geeks The Expert Cabana is packed with MEDC speakers and MVPs
  40. 41. © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.