InkSeine: In Situ Search for Active Note Taking
•Download as PPT, PDF•
0 likes•76 views
InkSeine is a system that allows users to search for information directly from digital ink notes. It combines pen-based note taking with searching, linking, collecting, and sensemaking. The core interaction pattern involves inking notes, searching those notes, and gathering related information. This allows users to interleave inking, searching, and gathering without switching contexts. Queries are treated as first-class objects within the notes. InkSeine aims to support active note taking by providing easy access to supporting materials through an in situ search experience directly within the notes.
Recommended
More Related Content
Similar to InkSeine: In Situ Search for Active Note Taking
Similar to InkSeine: In Situ Search for Active Note Taking (20)
More from Kimberly Aguada
More from Kimberly Aguada (6)
Recently uploaded
Recently uploaded (20)
InkSeine: In Situ Search for Active Note Taking
- 1. Ken Hinckley, Shengdong Zhao, Raman Sarin, Patrick Baudisch, Edward Cutrell, Michael Shilman & Desney Tan oronto
- 2. In situ search for digital ink InkSeine : Ink + Seine -n: a fishing net -v: to fish with a seine = insanely cool ink search! InkSeine lets the user fish for useful information directly from ink notes. Fishermen catching salmon on the Columbia River using a seine. (courtesy Wikipedia)
- 3. Active note taking XLibris—pen-based system for active reading (reading + critical thinking + learning) InkSeine combines pen-based note taking with searching, linking, collecting & sensemaking:
- 4. Core interaction pattern: Ink-Search-Gather Ink – excels at capturing user’s thoughts & ideas with minimum distraction from formatting issues Ink
- 5. Ink-Search-Gather interaction pattern Search – digging through the file system on a tablet is really a pain Active note taking benefits from low overhead access to supporting materials & related docs Focus on personal search, but support Web as well Ink
- 6. Ink-Search-Gather interaction pattern Gather – Make it easy to use what you find by pulling content directly into unstructured notes Thumbnail hyperlinks to documents Clippings from documents Queries that can be revisited later Ink
- 7. Ink-Search-Gather interaction pattern InkSeine unifies these three elements in a fluid user experience that is integrated directly with inking Ink
- 8. In situ search Leverage preexisting ink to initiate search No context switch to “search app” – stay in the flow No tedious copying of information to a “search box” Queries as first class objects, commingled with ink notes Flag some ink for search for later Visible and salient search history as part of notes Queries can be copied, pasted, moved, etc. Interleave inking, searching & gathering No barrier between inking / searching – note ideas while searching, side-by-side searching, span app boundaries for content gathering Tightly couple queries with application content Queries naturally persist in originating context Easy re-visitation of prior queries Results of queries become new content
- 9. Rich tradeoffs in cost structure of sensemaking Continuum of time cost – can defer at any point: 1 second: Capture thought “I should search on this” 5 seconds: Trigger search & see initial results list 10s: Open a document from result list, or drag result into notes 10s-30s: Inspect details of a few results Apply filters to results Add a word to the query Grab a snapshot from a short document Correct occasional reco errors – but Tablet’s reco is amazingly good! Longer: sufficient depth for common info needs; not mired by last 10% of complex queries that require full featured search tools Scroll through longer results lists Inspect result documents to see if they meet information needs Sideways searches based on Title, Date, or Author of a result
- 12. Paper Prototype
- 13. Paper Prototype
- 14. Paper Prototype
- 15. Findings Existing desktop search tool difficult to use Difficult text entry Enter and correct text
- 16. Findings Existing desktop search tool difficult to use Difficult text entry Enter and correct text Annoying context switch and flow interruption Switch among apps and context
- 17. Findings Existing desktop search tool difficult to use Difficult text entry Enter and correct text Annoying context switch and flow interruption Switch among apps and context Tedious information gathering Difficult to link to documents and gather pieces of information
- 18. Notebook Control
- 19. In Situ Search
- 21. Breadcrumb
- 23. Search Panel
- 26. Result List Result List Focus Result
- 27. File and Timeline Filters Filters
- 28. Gestures for Filters Tap - Select Cross – Multi-select Disjunctive cross – Multi-select Cross & Turn Right – Exclude
- 29. Work Flow & Occlusion Selection Correction Search
- 30. Work Flow & Occlusion
- 31. Work Flow & Occlusion
- 32. Crossing Widgets
- 34. Examine Document
- 35. Document & Tracking Menu
- 37. First Round Observations Gesture discovery problem
- 38. First Round Observations Gesture discovery problem Context preservation & occlusion problem
- 39. First Round Observations Gesture discovery problem Context preservation & occlusion problem Recognition problem for out-of-vocabulary word UIST List?, Dist?, Gist?, Mist?, Hist? ?
- 40. Second Round Observations Highlighter hints Smart positioning of search panel Personalized indexer
- 41. Summary In situ search experience Optimum workflow / maximum flexibility Enable rich tradeoffs Gather content Minimize search screen real estate Span application boundaries Tailored to pen input
- 42. Future Directions Pull in materials for searches from notes
- 43. Future Directions Pull in materials for searches from notes Support vector-based searches
- 44. Questions?
Editor's Notes
- Now we have a basic idea of the system, let me tell you the story behind its design In the early design stage, we wanted to convince ourselves whether users see values for ink-based search, as well as how they feel about the existing approaches. We conducted a paper prototype studies with 5 researchers. This is a screenshot a study in progress.
- The process goes as follows: they are first asked to write down a page of notes, then comments on if there are terms or phrases they like to search for. The feedback is consistently positive. They all found search for terms and phrase within their notes valuable. We then ask them to perform the search on these terms using an existing desktop search tool called Phlat (which is based on windows desktop search).
- Once they have done that, we ask them to propose improvements on the current search interfaces and suggest wanted features. Here are the screen shot of various interfaces proposed during the study.
- What we have learned from this study is that, first, existing desktop search tool (which literally designed for desktop computers with keyboard and mice) are very difficult to use. One major problem is that text entry through the tip interface is difficult and tedious.
- And it’s annoying to switch between the search application and the active note taking application. Both the tedious text entry process, and the application switches disrupt the train of thoughts, which is very important for active note taking.
- Finally, the lack of support of easy gathering and linking the relevant information together in the existing application making the active note taking process inconvenient to deal with. Based on these findings, we developed a prototype to overcome the forth mentioned limitations, and try to provide an optimal experience for searching while active note taking. Earlier, we saw the video of how the system works as whole, now I will go through the system step-by-step with you, and discuss the individual features in more details.
- The search tool live within an active notebook application, we call it the notebook control (example application are window’s journal or Scriblet) The application can have a number of objects, and support a number of gestural commands.
- Now we have a basic idea of the container application, let’s look at how we can perform the search tasks. As you can see, we are active note taking on the subject of InkSeine, and trying to gather some related work. We remember there is a paper called Bumptop last year that’s so what interesting, so we want to look it up. To do that, we draw an open ended lasso around the word: BumpTop
- Immediately, a small icon appears at the end of the lasso. This icon is called the breadcrumb and it’s an indication that I am interested to perform a search on the selected word. However, I don’t have to dive into the search right away, for example, if my train of thoughts has a number of related work I am thinking about, I can continue write down all the other papers I can think of. The breadcrumb serves as a reminder, information storage, and a placeholder for further actions.
- Breadcrumbs provides information about the searches they represent.
- By drawing a mark start on the breadcrumb and toward the south, we trigger the personal search, and brings up the search panel
- The search panel is composed of different functional units.
- On the top, right under the breadcrumb is the query editing and revising area. Since ink can sometimes be mis-interpreted, we need a place to allow the user to correct the ink. Also, user could change their mind and revise the query, the query area also allow the user to add, delete, and edit the query terms.
- Once user are sure about their search query, they want to look at the result, which is located right below the query box. It contains multiple columns of information on the research results.
- Point to any of the record will make it the focus result with additional information, such as text snippets, thumbnail, and hyperlinks to offer actions for that document.
- Based on the lessons we learned from earlier search interfaces such as phlat, two types of filters are the most useful for people to narrow down their search result for personal items. The two types are the file type filter (which is located vertically on the right), and the timeline filter, which is located at the bottom. Both filters support a number of convenient gestures for pen based interface.
- The search panel is composed of different functional units.
- When a document opens, we provide the user with a tracking menu to afford pen interaction with legacy mouse-and-keyboard applications. It supports scrolling documents with a scroll ring gesture, and a capture tool for grabbing clippings from a document. The earlier video shows how it works.