Bridging the Paper-Digital Divide:
The iPaper Interactive Paper Framework
Beat Signer
Institute for Information Systems, ETH Zurich
8092 Zurich, Switzerland
http://vub.academia.edu/BeatSigner
USI Seminar, May 30, 2007

The Paperless Office (1975)
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 2

The "Paperless Office" (2003)
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 3

The Myth of the Paperless Office
 For decades, people have predicted the
office of the future as a paperless office
 documents generated, published and
distributed electronically
 documents read electronically
 What has happened to this
imminent revolution?
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 4

Affordances of Paper
 The physical properties of an object determine
how people use that object
 Properties of paper
 light, flexible, robust, porous, opaque, transparent, …
 Human actions
 grasping, folding, tearing, carrying, writing on, …
 Paper supports forms of collaboration and
interaction difficult to mimic in the digital world
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 5

E-Paper versus Digitally Augmented Paper
Electronic Paper (E-Paper)
Digitally Augmented Paper
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch

Outline
 Motivation
 Interactive paper framework (iPaper)
 Cross-media link server (iServer)
 Interactive paper applications
 functionality
 authoring
 Future work
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch

Augmented Paper Projects
Digital Desk, Wellner, 1991 Listen Reader, Back et al., 2001
Audio Notebook, Books with Voices
Stifelman et al., 2001 Klemmer et al., 2003 ButterflyNet, Yeh et al., 2005
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch

Digital Pen and Paper
 Developed by Anoto
 Different pen manufacturers
 Logitech, Maxel, Nokia, ...
 Pattern space
 60 million km2
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 9

Problems of Existing Applications
 Many projects focus on the input device, paper,
printing and other hardware technologies rather
than on the data integration and information
management aspects  isolated solutions
 The linking of paper tends to be based on
physical rather than information-centric concepts
 difficult to integrate new input / output devices
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 10

Our Approach
 General interactive framework
 different forms of paper-based interactions (enhanced
reading, enhanced writing, …)
 rapid prototyping as well as large scale publishing
 integration of new types of resources and services
 integration of new input devices
 The key to a highly integrated interactive paper
solution lies in the introduction of a platform for
general cross-media information management
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 11

Active Areas and Virtual Page Layers
image
video clip
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch

iPaper Architecture
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch

iServer Metamodel and Resource Plug-ins
link
Entities
Selectors Resources
Shape Page Query DB Object
Time Span / Movie Clip XPointer XHTML
Data/Metadata
Spatial Info Document
iPaper Application Database
Movie Clips HTML Documents
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch

iSERVER
iServer
Links link parameter
(1,*) (1,*) Has (0,*)
HasSource Links HasTarget Preferences
Preferences
(0,*) (0,*)
(0,*)
parameter entity (0,*) AccessibleTo (0,*) user
(0,*) (0,*) (0,*)
Properties HasProperties Entities (0,*)
(0,*)
Users HasMembers
(1,1)
InaccessibleTo
Layers partition
partition
(0,*)
layer individual group
(0,*) (0,*)
|Layers| OnLayer CreatedBy Individuals Groups
Users
layer
(1,1) Plug-ins
selector resource
Active (1,1) (0,*) (0,*)
Selectors RefersTo Resources Contains
Layers
(0,*)
Resources
iPAPER page resource activeComponent medium
(0,*) Active Com-
OnPage Pages Containers Media
(1,1) ponents
shape
(0,*)
Shapes HasShapes (1,1)
medium
document
Contains (0,*) Movies
partition (2,*) Documents
Pages
rectangle polygon ellipse circle complexShape medium medium
Complex
Rectangles Polygons Ellipses Circles Images Webpages
Shapes
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch

Active Content
 Execute code (services) on the client and server side
AC Manager
Active Component
ACS Manager A
Active Component Active Component
Stub B B
Request Sender
Client Device
Active Component
AC Names HTTP
C
iServer AC Names
Client Server
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch

Active Components
Button Image
Slider Sound
Slider2D Movie
DateSelector SQLClient
Browser XCMRequest
MapSelector ClientCtrlRequest
Rating PaperPointControl
Capture PaperPointAnnot
CaptureAndSpeak ...
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch

Active Components …
<?xml version="1.0" encoding="UTF-8" ?> PAPERPOINT_CONTROL
<iserver>
<activeComponent id="s1a" creator="beat"> Show slide 1
<name>Show slide 1</name>
<properties> command showSlide
<parameter>
<key>org.ximtec.iserver.ac:command</key> slide 1
<value>showSlide</value>
</parameter>
<parameter>
<key>org.ximtec.iserver.ac:slide</key>
<value>1</value>
</parameter>
</properties>
<identifier>PAPERPOINT_CONTROL</identifier>
</activeComponent>
...
</iserver>
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch

PaperPoint Presentation Tool
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch

XML-Based Authoring
... ...
<document id="handout" creator="beat"> <activeComponent id="fsa" creator="beat">
<name>Slide Handout</name> <name>First slide action</name>
<id>h1</id> <properties>
<size> <parameter>
<width>210</width> <key>org.ximtec.iserver.ac:command</key>
<height>297</height> <value>firstSlide</value>
</size> </parameter>
<content>handout.pdf</content> </properties>
</document> <identifier>PAPERPOINT_CONTROL</identifier>
<page id="p1" creator="beat" document="h1"> </activeComponent>
<name>Page 1, Slide Controller</name> ...
<number>1</number> <link id="fsl" creator="beat" sources="fs"
</page> targets="fsa">
... <name>Show the first slide</name>
<rectangle id="fs" creator="beat" </link>
layer="default" resource="p1"> ...
<name>First Slide</name>
<upperLeft>
<point><x>24</x><y>12</y></point>
</upperLeft>
<size>
<width>25</width><height>8</height>
</size>
</rectangle>
...
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch

Digital Link Authoring Tool
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch

PaperPoint Architecture
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch

Edinburgh Fringe Festival
 World's largest
international arts festival
 4 weeks
 ~1700 events
 ~27000 performances
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 23

EdFest Components
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 24

EdFest Booklet
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 25

Content Publishing Framework
iServer
cross-media link server
XML link
XCM iPublish definition
content publishing publishing framework
data
publish • paper link data
Structure Style • active areas on paper
• layers
• digital resources
import • user model
PDF
XML data XSL CSS
of festival
venues
and events
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch

Print-n-Link
 Citations build the basis
for a scientific web of
publications
 Use digital services to
search / retrieve cited
publications
 Automatic authoring
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch

Print-n-Link Architecture
Interactive Paper Architecture Desktop
(x,y)
Coordinates iSERVER
iPAPER Active Content Text-to-
Speech
iPaper iPaper Plug-in
Client Link DB
Publish
Positions and
Active Content
Definition
Document Retrieval
Print-n-Link Engine
Web Search
Citation Document Services
Retrieval Lookup
Print-n-Link: Weaving the Paper Web Semantic Mapper Printing
Moira C. Norrie Beat Signer Nadir Weibel
Dept. of Computer Science Dept. of Computer Science Dept. of Computer Science
ETH Zurich ETH Zurich ETH Zurich
8092 Zurich, Switzerland
norrie@inf.ethz.ch
8092 Zurich, Switzerland
signer@inf.ethz.ch
8092 Zurich, Switzerland
weibel@inf.ethz.ch
Publish Display
ABSTRACT Citation DB
Citations
engines and web browsers. The ease with which we can
Citations form the basis for a web of scientific publications. not only find documents on the web, but also follow links
Search engines, embedded hyperlinks and digital libraries to related documents is considered to be one of the main
all simplify the task of finding publications of interest on advantages of working with digital documents as opposed
the web and navigating to cited publications or web sites. to paper equivalents. Y many studies have shown that
et
However the actual reading of publications often takes place paper is still the preferred med ium when it comes to reading
on paper and frequently on the move. We present a system documents for reasons of comfort, mobility and also ease of
Print-n-Link that uses technologies for interactive paper to annotation.
enhance the reading process by enabling users to access dig- In this paper, we show how emerging technologies for in-
ital information and/or searches for cited documents from a teractive paper enable a bridge to be built between paper
printed version of a publication using a digital pen for inter- and digital document spaces that allows users to travel back
and forth between the two using the medium most appropri-
iDOC
action. A special virtual printer driver automatically gener-
ates links from paper to digital services during the printing ate to the current task and sit uation. Users can print doc-
process based on an analysis of PDF documents. Depend- uments for reading and, by activating hyperlinks on paper
ing on the user setting and interaction gesture, the system with a digital pen, they can not only access digital informa-
may retrieve metadata about the citation and inform the tion about related documents, but also initiate web searches
user through an audio channel or directly display the cited to retrieve copies of those documents. The retrieved docu-
document on the user’s screen. ments may in turn be printed for reading with automatically
generated embedded hyperlinks on the paper. Mobility is
supported by providing disconnected users on the move with
Categories and Subject Descriptors audio information about related documents and the option
D.2.11 [ Software Engineering ]: Software Architectures; of adding them to a request list to be processed when recon-
H.4.m [ Information Systems Applications ]: Miscella- nected.
neous; H.5.m [ Information Interfaces and Presenta- As an application, we take the everyday academic task of
tion ]: Miscellaneous reading scientific publications. Whether the reading activ-
Semantic Mapper
ity is for the purpose of knowledge discovery or reviewing,
readers regularly look up citations to other publications by
General Terms flicking to the reference list at the back of the paper. In some
Design, Algorithms, Experimentation cases, a search for the cited publication may follow and, if
found, often it is printed and added to a pile of documents to
Keywords be read. Citations are the established means of providing the
links that form the basis of the scientific web of publications
Publish
Interactive paper, document integration, digital library, ci- that spans the digital and paper worlds. Just as citations
tation management within digital documents can be used to automatically gen-
1. INTRODUCTION
erate hyperlinks to cited documents, we can automatically
generate links within paper documents through an analysis
Citation Printer
Nowadays we are all familiar with the notion of hyperlinks
as a means of creating interconnected document spaces that
of the PDF file when the document is printed.
Section 2 presents a more detailed look at the activity
of reading scientific publications and a discussion of related Document Print
Driver
can be easily searched and navigated with the help of search
work. Section 3 describes the digital pen and paper tech-
nologies and how these can be used to support interaction
Analyser
Pattern
between paper and digital services. We then present the
Permission to make digital or hard copies of all or part of this work for general functionality of our enhanced reading system in Sec-
personal or classroom use is granted without fee provided that copies are tion 4 and the overall architecture in Section 5. Details of
not made or distributed for profit or commercial advantage and that copies the virtual printer component used to analyse PDF docu-
bear this notice and the full citation on the first page. To copy otherwise, to ments for automatic link generation as well as printing are
republish, to post on servers or to redistribute to lists, requires prior specific
permission and/or a fee.
given in Section 6. Concluding remarks follow in Section 7.
DocEng’06, October 10–13, 2006, Amsterdam, The Netherlands.
Copyright 2006 ACM 1-59593-515-0/06/0010 ...$5.00.
Document DB
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch

Interactive Paper Applications
 Enhanced Reading  Enhanced Writing
 interactive paper maps  photo album
 interactive festival brochures  researcher's notebook
 educational materials  mammography
 auction brochures  search system
 scientific publications  …
 …
 Paper-Based Interfaces  Artists' Installations
 PaperPoint  Lost Cosmonaut
 digital libraries  Generosa Enterprise
 video analysis tool
 educational games  Interactive TableTops
 …
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch

Current Projects
 PaperWorks (FP6-516895)
 Interweaving Paper and Digital Documents
 King's College London, Acreo AB, Anoto, ETH Zurich,
Arjowiggins SAS, Brunel University, Malmo
University, Pearson Education
 DELOS
 Network of Excellence on Digital Libraries
 Paper-based user interfaces for image retrieval
(collaboration with Heiko Schuldt, University of Basel)
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch

Future Work
 Design guidelines for interactive paper interfaces
 functionalty (active components) and design
 Cross-media authoring and publishing
 analysis of existing publishing processes (Pearson)
 general document model for mapping between printed
and digital document instances
 Extensions of iServer
 collaborative cross-media information sharing
 new resource plug-ins
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch

Mapping between Printed and Digital
Instances
static physical rendering
dynamic logical document
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch

Conclusions
 Interactive paper framework (iPaper)
 information-centric approach supports different
technologies, functionalities and applications
 rapid prototyping (reusable active components)
 Extensible platform for cross-media information
management (iServer)
 Cross-media authoring / publishing tools
 manual, content-driven and automatic link authoring
USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch