NuSys is a proposed system to enhance how creative groups brainstorm and develop ideas using an electronic whiteboard. It would allow groups to easily share content from various sources onto the whiteboard workspace. NuSys would record discussions, annotations, and links between content to capture information networks. It would also support automatic generation of presentations from the whiteboard content, such as meeting summaries and slide shows. One use case described a study group using NuSys on a large display to organize and link course materials, images, and discussions to map relationships between concepts.

1. NuSys: Collaborative Insight Extraction on an Electronic Whiteboard
Pen and Touch Computing Center
Brown University
Providence RI, 02912
June 23, 2015
1 Abstract
Gaining insights – the act of acquiring a deep and intuitive understanding of a thing or concept – is often achieved
through discussions in small work groups. Meetings or informal brainstorming sessions between individuals
where questions and ideas are bounced back and forth are a common way to develop, work out and present new
knowledge. We identify scenarios where such insight extraction is typically found and propose to build NuSys, a
system intended to enhance how creative groups brainstorm and develop ideas when an electronic whiteboard
(EW) replaces a physical whiteboard as the focal point of small team collaborations, and provides more flexible
means of importing, organizing and presenting content than the combined use of a set of state-of-the-art
productivity tools.
2 Introduction
2.1 Overview
Previous electronic whiteboard (EW) research primarily considers either overcoming the technical shortcomings
of physical whiteboards, such as limited space and immovable content, or simple schemes of sharing information
between personal devices and the whiteboard. Our research will build on this work to address aiding a
comprehensive, small-group ideation workflow. With NuSys, workgroups will effortlessly share and organize
heterogeneous materials, e.g. Office documents, multimedia, and HTML content, from their laptops and tablets
onto a central EW workspace. By recording their discussions, pen markups and metadata annotations, and
by recognizing explicit and implicit data linking, NuSys will capture rich information networks that can be
transformed and visualized to illustrate emergent patterns and to retrace the genesis of ideas. Our development
will be driven by multiple scenarios, especially an educational research scenario in which small teams of students
can share content from their native tablet and laptop applications, such as Word or Acrobat, onto a large
interactive display, such as the 84” Microsoft Surface Hub. While organizing and augmenting these materials
within NuSys, they will be able to generate a variety of structured layouts to help gain perspective over their
materials and to retrace the history of their workspace, including contemporaneous discussions. Moreover, by
analyzing the structure of the whiteboard content, NuSys will support the automatic generation of targeted
presentations, such as meeting summaries, slide presentations, and review guides.
2.2 Insight Extraction Model
Insight extraction is an iterative process that typically consists of the following three different phases (Figure 1):
1. Information Gathering: Before researching a topic, relevant information is gathered from different sources.
Such sources include PDF files, HTML document found on the Internet, images, videos, rich-text files
and presentations such as Microsoft Office and Open Office documents, spreadsheets, emails as well as
structured data in the form of databases, tables etc.
2. Sensemaking / Insight Extraction: Once relevant material has been collected, information workers go
through a thinking process, analyzing and interpreting the content of their documents, writing notes and
1

2. Information
Gathering
Sensemaking /
Insight
Extraction
Presentation
Figure 1: 3-phase model of a insight extraction process.
building relationships and hierarchies in order to gain new insights and support or refute hypotheses based
on evidence found in the collected data.
3. Presentation: At any point during the Sensemaking / Insight Extraction phase, information workers are
often required to present interim findings to colleagues and supervisors or to compile a summary/handout
of their results which is used for further brainstorming and reviewing.
With NuSys, we will lower the technical barrier for each of these phases and ease transitions between them by
facilitating the following ideas:
• Implementing plugins for existing productivity tools and webbrowsers enables us to enhance the Information
Gathering phase in the following three ways: First, having a part of NuSys reside in third party software
gives us the ability to improve the process of selecting and transferring content, for example, copying a
paragraph or image of a website from a web-browser to a workspace in NuSys. Second, since plugins
usually have access to lower-level structures of a document (e.g. the Document Object Model (DOM) in
browsers) and additional metadata such as a browsing history, NuSys can store and use the context of
imported content in order to perform automatic (weighted) tagging/labeling - a task often neglected by
users but crucial for efficient search, visualization and clustering. Third, knowing the internal structure of
a document allows us to create find-grained, bi-directional links between fragments imported into NuSys
and their original source.
• NuSys will support users in the Sensemaking/Insight Extraction phase by providing a collaborative
workspace using a whiteboard metaphor. The workspace allows users to organize material gathered during
the Information Gathering phase by arranging, grouping, linking and annotating content in a free-form
fashion. Furthermore, it offers means to search, visualize and automatically cluster related documents and
fragments of information by inspecting their original context, taking the automatically created tags/labels
and the explicit and implicit links created by the user into account. To further support the information
workers during the Sensemaking/Insight Extraction phase, NuSys allows users to backtrack changes and
provide access to an activity history in order for them to keep track of their progress and playback the
evolution of their insights and ideas.
• The simplest way of presenting interim results and insights gained during the Sensemaking / Insight
Extraction phase is to manually navigate through the workspace. However, information workers often
want to present their findings in a more targeted format such as a slide presentation, a website, a
printable summary or review guides. By inspecting hierarchies and relationships created by the user in
the Sensemaking / Insight Extraction phase as well as by using metadata and the original context of
documents, NuSys will offer the ability to automatically generate such targeted presentations.
2

3. 3 Use Cases
3.1 Study Group
Three students – Alice, Bob, and Claire – take the same art history course and meet every week to discuss and
review course material. Using our system, which is running on a Microsoft Surface Hub in their department,
Alice starts by importing slides from a lecture into the workspace by selecting the file on her tablet and flicking
it onto an unbounded, manipulable 2D workspace on the Hub. The students highlight important parts of the
slides and add notes from their discussion directly on the workspace using a pen. Claire then proposes to go
over a selection of van Gogh’s artworks that were discussed in class. They find some high-resolution images
of the artworks on the internet, select them, along with accompanying text, with lassoing gestures, and flick
the selections onto the workspace. On the workspace, Bob then establishes relationships between artworks and
relevant slides by drawing links from sections of the slides to the associated images, and then annotating them
with brief explanations and/or media. These and all other metadata accompanying all objects in the system will
allow the students to search the workspace. The group records their ongoing discussion using the integrated
camera and microphone. The links the group creates form a navigable visual hypergraph of relationships that
map out relevance between content, and build a picture of the abstract idea space. As the amount of content
grows large, they decide to organize the workspace by grouping and nesting the material using intuitive pen/touch
gestures. A textual search quickly filters content into a grouping of relevant objects, which is then saved to
the workspace as a composite object. NuSys also attempts to extract metadata tags such as the date created,
country, and subject by analyzing the original contexts of several artworks. These tags are then used to suggest
suitable groupings. The students are later tasked with creating a presentation on their findings; though they
have their material stored in NuSys, they disagree on how they came to certain conclusions. To clarify those
points, they use NuSys’ ability to backtrack and visualize the construction of these content areas to revisit their
previous thinking. Finally, the students use NuSys to export their findings in a presentable format to show in
class.
4 Related Works
Prior research has examined how information workers use traditional whiteboards as ”thinking tools” and what
kinds of improvements should be made in order to support the thinking process of individuals and groups.
Commonly noted constraints of the whiteboard include the inability to rearrange content and a restricted amount
of space [8]. WorkTop [4], which resolves some of these constraints was designed to enhance the ability of
students and scholars at all levels to more efficiently perform fundamental scholarly tasks while at the same time
providing new opportunities for collaborative learning. Like NuSys, WorkTop tries to reduce the technological
barriers to capturing, displaying and linking heterogeneous documents or document fragments within a unified
workspace. Searchable keywords can be applied to entire documents or specific regions and WorkTop also allows
for fine-grained, typed, and bi-directional hyperlinking between different contents, inspired by CodeBubbles [3].
Furthermore, it supports ink and rich-text annotations and provides a variety of structured workflow assistants,
such as templates for publishing collected materials as stylized multimedia web pages, search operations which
can be persisted as smart folders, and snapshots that capture the workspace state so that tasks can be resumed
at a later time.
The Sandbox [9], similarly to WorkTop, is a flexible and expressive thinking environment that supports both
ad-hoc and more formal analytical tasks. Fragments of content and documents can be laid out on a 2D workspace
and notes as well as annotated user-created links explain the relationships between collected material. The
Sandbox offers support for a variety of different document types, such as images, PDFs, videos and text. Content
can be imported via clip-board operations or through a companion application called TRIST, an information
scanning and analysis tool which is part of nSpace [7], a visual analytics system for analysts to collaboratively
produce insightful evidence-based reports.
NuSys will extend WorkTop’s and Sandbox’ set of workspace features, such as linking, nesting, annotating, with
interactions inspired by RichReview and LiquidText [10, 6]. Moreover, in contrast to WorkTop and The Sandbox,
Nusys focuses on collocated collaboration - study groups whose members conglomerate at the same physical
location e.g., a meeting room. Based on evidence found in [1], which suggests that intriguing possibilities for
sensemaking tools emerge by creating a virtual workspace with real physical space i.e. large high-resolution
displays, NuSys will be a EW application that runs on such devices (hub) and tightly integrates with its
3

4. counterpart on tablets and laptops (clients) in order to support push and pull operations of content between the
hub and clients, similar to the ones proposed by Code Space [2].
Furthermore, NuSys follows a different approach when importing content to a workspace: While existing visual
analytics and process capture tools largely ignore the semantic context of imported material, NuSys gains
access to that additional metadata through plugins which we will develop for existing productivity software and
web-browsers. On one hand, this supporting information allows us to implement automatic grouping/clustering
of documents and fragments of documents gathered on the workspace, similarly to Jigsaw [5]. On the other
hand, additional metadata allows for a better semantic understanding of collected content which we will exploit
to implement search capabilities and automated generation of targeted presentations.
NuSys also extends beyond the capabilities of many commercial standalone applications in order to support
complex and evolving long-term work. Microsoft OneNote, for instance, does not (yet) have a robust multi-user
experience due to its limited realtime features and lacks the ability to structure and navigate content via
fine-grained bidirectional hyperlinks. OneNote is also not able to capture the progression of the ideas and
insights. On the other end of the spectrum, Office Sway offers a toolkit for assembling presentations, but does
not specifically facilitate better information gathering and structuring. Something of a middle ground can be
found in Storyboard. It offers the ability to import and organize content, and rearrange the logical sequence to
efficiently produce some kind of simple, accessible narrative for consumption. However, as a consequence of its
efficiency, it appears to lack sophisticated tools for content exploration and process capture, and also cannot
generate the more sophisticated hypergraph upon which NuSys operates.
5 Method
5.1 Information Gathering
5.2 Collaborative Workspace
5.3 Targeted Presentations
6 Evaluation
Telemetry:
With NuSys, we expect to measure a significant increase in efficiency for creative groups trying to get from
an initial set of materials or ideas to a final deliverable. We will qualitatively and quantitatively evaluate our
system in user studies for classes in Art History and in Moving Panoramas, whose faculty have used the far
more basic forerunner system, TAG (Touch Art Gallery) in previous years.
7 Timeline
Experience with similar projects convinced us that concentrating our efforts on concrete prototyping goals will
yield results faster. We will aim to build prototypes quickly and then incrementally improve it based on external
feedback and the experience gained throughout the process. The plan below outlines our goals for a period of
two years.
Phase I (Months 1-8) Phase II (Months 9-16) Phase III (Months 17-24)
Implementation of plugin prototypes for
a web browser and a limited set of Mi-
crosoft Office tools to support the extrac-
tion of the original context of material
brought to the Hub as well as bidirec-
tional linking. Implementation of a prim-
itive collaborative workspace for organiz-
ing material. Basic export functionality
in form of a HTML summary. Prelimi-
nary user testing.
Refinement of plugin prototypes and sup-
port for more productivity tools. In-
tegration of search on workspace. Hy-
pergraph Visualization. Ability to back-
track changes
Focus on targeted presentation. Devel-
opment of algorithms which make use of
the hypergraph structure to create tai-
lored presentations. Extensive user test-
ing in Art History and Moving Panora-
mas classes offered at Brown University,
and possibly other venues.
4

5. References
[1] Christopher Andrews, Alex Endert, and Chris North. Space to think: large high-resolution displays for
sensemaking. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages
55–64. ACM, 2010.
[2] Andrew Bragdon, Rob DeLine, Ken Hinckley, and Meredith Ringel Morris. Code space: touch+ air gesture
hybrid interactions for supporting developer meetings. In Proceedings of the ACM International Conference
on Interactive Tabletops and Surfaces, pages 212–221. ACM, 2011.
[3] Andrew Bragdon, Robert Zeleznik, Steven P Reiss, Suman Karumuri, William Cheung, Joshua Kaplan,
Christopher Coleman, Ferdi Adeputra, and Joseph J LaViola Jr. Code bubbles: a working set-based interface
for code understanding and maintenance. In Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems, pages 2503–2512. ACM, 2010.
[4] Robert Zeleznik et al. WorkTop. http://cs.brown.edu/research/ptc/worktop/, 2012.
[5] John Stasko, Carsten G¨org, and Zhicheng Liu. Jigsaw: supporting investigative analysis through interactive
visualization. Information visualization, 7(2):118–132, 2008.
[6] Craig S Tashman and W Keith Edwards. Liquidtext: a flexible, multitouch environment to support
active reading. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages
3285–3294. ACM, 2011.
[7] uncharted. nSpace. https://uncharted.software/nspace/, 2012.
[8] Jagoda Walny, Sheelagh Carpendale, Nathalie Henry Riche, Gina Venolia, and Philip Fawcett. Visual
thinking in action: Visualizations as used on whiteboards. Visualization and Computer Graphics, IEEE
Transactions on, 17(12):2508–2517, 2011.
[9] William Wright, David Schroh, Pascale Proulx, Alex Skaburskis, and Brian Cort. The sandbox for analysis:
concepts and methods. In Proceedings of the SIGCHI conference on Human Factors in computing systems,
pages 801–810. ACM, 2006.
[10] Dongwook Yoon, Nicholas Chen, Fran¸cois Guimbreti`ere, and Abigail Sellen. Richreview: blending ink,
speech, and gesture to support collaborative document review. In Proceedings of the 27th annual ACM
symposium on User interface software and technology, pages 481–490. ACM, 2014.
5