General Framework for the Rapid Development
of Interactive Paper Applications
Moira C. Norrie, Beat Signer and Nadir Weibel
Institute for Information Systems, ETH Zurich
8092 Zurich, Switzerland
http://vub.academia.edu/BeatSigner




                                                November 4, 2006
Applications
 Enhanced Reading                                 Enhanced Writing
           interactive paper maps                            photo album
           interactive festival brochures                    researcher's notebook
           educational materials                             mammography
           auction brochures                                 recommender system
           scientific publications                           …
           …

 Paper-Based Interfaces                           Artists' Installations
           PaperPoint                                      Lost Cosmonaut
           educational games                               Generosa Enterprise
           digital libraries
           video analysis tool                    Interactive TableTops
           …

November 4, 2006                             CoPADD 2006                               2
Input Devices




November 4, 2006   CoPADD 2006   3
Interactive Paper Architecture


                                  Print-n-Link: Weaving the Paper Web

                        Moira C. Norrie
                 Dept. of Computer Science
                         ETH Zurich
                  8092 Zurich, Switzerland
                     norrie@inf.ethz.ch
                                                                            Beat Signer
                                                                   Dept. of Computer Science
                                                                           ETH Zurich
                                                                    8092 Zurich, Switzerland
                                                                      signer@inf.ethz.ch
                                                                                                                 Nadir Weibel
                                                                                                          Dept. of Computer Science
                                                                                                                  ETH Zurich
                                                                                                           8092 Zurich, Switzerland
                                                                                                             weibel@inf.ethz.ch
                                                                                                                                                                        HTTP Request
   ABSTRACT
   Citations form the basis for a web of scientific publications.
                                                                                       engines and web browsers. The ease with which we can
                                                                                       not only find documents on the web, but also follow links
                                                                                                                                                                        ID, Page, (x,y)
                                                                                                                                                         Encoded
   Search engines, embedded hyperlinks and digital libraries                           to related documents is considered to be one of the main
   all simplify the task of finding publications of interest on                         advantages of working with digital documents as opposed
   the web and navigating to cited publications or web sites.                          to paper equivalents. Y many studies have shown that
                                                                                                                  et
   However the actual reading of publications often takes place                        paper is still the preferred med ium when it comes to reading
   on paper and frequently on the move. We present a system                            documents for reasons of comfort, mobility and also ease of
   Print-n-Link that uses technologies for interactive paper to                        annotation.
   enhance the reading process by enabling users to access dig-                           In this paper, we show how emerging technologies for in-
   ital information and/or searches for cited documents from a                         teractive paper enable a bridge to be built between paper
   printed version of a publication using a digital pen for inter-                     and digital document spaces that allows users to travel back
                                                                                       and forth between the two using the medium most appropri-




                                                                                                                                                         Position
   action. A special virtual printer driver automatically gener-
                                                                                       ate to the current task and sit uation. Users can print doc-




                                                                                                                                                                                          XIMA
   ates links from paper to digital services during the printing
   process based on an analysis of PDF documents. Depend-                              uments for reading and, by activating hyperlinks on paper
   ing on the user setting and interaction gesture, the system                         with a digital pen, they can not only access digital informa-
   may retrieve metadata about the citation and inform the                             tion about related documents, but also initiate web searches
   user through an audio channel or directly display the cited                         to retrieve copies of those documents. The retrieved docu-
   document on the user’s screen.                                                      ments may in turn be printed for reading with automatically
                                                                                       generated embedded hyperlinks on the paper. Mobility is
                                                                                       supported by providing disconnected users on the move with
   Categories and Subject Descriptors                                                  audio information about related documents and the option
   D.2.11 [ Software Engineering    ]: Software Architectures;                         of adding them to a request list to be processed when recon-
   H.4.m [ Information Systems Applications        ]: Miscella-                        nected.
   neous; H.5.m [ Information Interfaces and Presenta-                                    As an application, we take the everyday academic task of
   tion ]: Miscellaneous                                                               reading scientific publications. Whether the reading activ-
                                                                                       ity is for the purpose of knowledge discovery or reviewing,




                                                                                                                                                                           Response
                                                                                       readers regularly look up citations to other publications by
   General Terms                                                                       flicking to the reference list at the back of the paper. In some
   Design, Algorithms, Experimentation                                                 cases, a search for the cited publication may follow and, if




                                                                                                                                                                                                        AC
                                                                                       found, often it is printed and added to a pile of documents to
   Keywords                                                                            be read. Citations are the established means of providing the
                                                                                       links that form the basis of the scientific web of publications
   Interactive paper, document integration, digital library, ci-                       that spans the digital and paper worlds. Just as citations
   tation management                                                                   within digital documents can be used to automatically gen-
                                                                                       erate hyperlinks to cited documents, we can automatically
   1.     INTRODUCTION                                                                 generate links within paper documents through an analysis
     Nowadays we are all familiar with the notion of hyperlinks                        of the PDF file when the document is printed.
   as a means of creating interconnected document spaces that                             Section 2 presents a more detailed look at the activity




                                                                                                                                                                                                     iPaper
   can be easily searched and navigated with the help of search                        of reading scientific publications and a discussion of related
                                                                                       work. Section 3 describes the digital pen and paper tech-
                                                                                       nologies and how these can be used to support interaction
                                                                                       between paper and digital services. We then present the
   Permission to make digital or hard copies of all or part of this work for           general functionality of our enhanced reading system in Sec-
   personal or classroom use is granted without fee provided that copies are           tion 4 and the overall architecture in Section 5. Details of
   not made or distributed for profit or commercial advantage and that copies           the virtual printer component used to analyse PDF docu-
   bear this notice and the full citation on the first page. To copy otherwise, to      ments for automatic link generation as well as printing are
   republish, to post on servers or to redistribute to lists, requires prior specific
                                                                                       given in Section 6. Concluding remarks follow in Section 7.




                                                                                                                                                                                                 Link DB
   permission and/or a fee.
   DocEng’06, October 10–13, 2006, Amsterdam, The Netherlands.
   Copyright 2006 ACM 1-59593-515-0/06/0010 ...$5.00.




     Interactive                                                                                                                                             iPaper Client and
     Document                                                                                                                                                  Output Device



November 4, 2006                                                                                                                                                         CoPADD 2006                      4
Input Devices
                                                                             <<interface>>
                                                                             InputHandler          0..*
                                       <<interface>>                                                                       Location
                         0..*        InputDeviceEvent                        +handle()
                                                                                                                       +id: String
                                   +getTimestamp() : long                                                              +page: int
                                                                                                                       +x: double
                                                                                                                       +y: double
                                                                                                          Notifies
                                  AbstractInputDeviceEvent            DefaultInputHandler          ...

                                                                                                                     TimestampedLocation
                   Creates
                                                                                                                     +timestamp: long

                                                                                                                           0..*
                                InductivePenEvent     NokiaPenEvent              ...
                                                                                                         1
                   1                                                                                                        Creates
                                                                                                <<interface>>
                                    <<interface>>                 1               0..*                                        1
                                                                                          InputDeviceEventListener
                                    InputDevice                       Notifies
                                                                                          +inputDeviceEvent()
                       +addInputDeviceEventListener()
                                                                                          +addInputHandler()
                       +removeInputDeviceEventListener()
                                                                                          +removeInputHandler()



                                AbstractInputDevice                                    AbstractInputDeviceEventListener




                    InductivePen          NokiaPen          ...   InductivePenEventListener                  NokiaPenEventListener      ...



November 4, 2006                                                        CoPADD 2006                                                           5
iServer Cross-Media Platform
                                                           link

                                       Entities



                               Selectors        Resources




                Shape          Page                      Query         DB Object
               Time Span /    Movie Clip                 XPointer          XHTML
                                                                       Data/Metadata
               Spatial Info                                              Document



        iPaper                                  Application Database
          Movie Clips                            HTML Documents


November 4, 2006                           CoPADD 2006                                 6
Interactive Paper Authoring
 iServer/iPaper interfaces
           Java API
           XML-based specification
           Web Service interface
           P2P interface

 Manual link authoring
 Content-driven publishing
 Automatic link authoring

November 4, 2006                      CoPADD 2006   7
iDoc Publishing Framework


                                                        CMS



                   iDOC
                    Semantic Mapper
                                      Document
                                      Metadata                 Printer
                       Plug-ins                                Driver

                                                 Document DB
                     Positions and                              Print
                     Active Content
                                      iSERVER
                                       iPAPER

November 4, 2006                          CoPADD 2006                    8
Active Components
                       Button                            Image

                       Slider                           Sound

                      Slider2D                          Movie

                    DateSelector                      SQLClient

                      Browser                         XCMRequest

                    MapSelector                    ClientCtrlRequest

                       Rating                      PaperPointControl

                      Capture                      PaperPointAnnot

                   CaptureAndSpeak                        ...

November 4, 2006                     CoPADD 2006                       9
Slider / Slider2D Active Component




November 4, 2006   CoPADD 2006       10
Conclusions
 General framework for interactive paper
 Cross-media information server
        extensibility (resource plug-ins)
        link service acessible through various APIs
        active content
        information sharing

 Constantly growing set of resource plug-ins and
     active components

November 4, 2006               CoPADD 2006             11

General Framework for the Rapid Development of Interactive Paper Applications

  • 1.
    General Framework forthe Rapid Development of Interactive Paper Applications Moira C. Norrie, Beat Signer and Nadir Weibel Institute for Information Systems, ETH Zurich 8092 Zurich, Switzerland http://vub.academia.edu/BeatSigner November 4, 2006
  • 2.
    Applications  Enhanced Reading  Enhanced Writing  interactive paper maps  photo album  interactive festival brochures  researcher's notebook  educational materials  mammography  auction brochures  recommender system  scientific publications  …  …  Paper-Based Interfaces  Artists' Installations  PaperPoint  Lost Cosmonaut  educational games  Generosa Enterprise  digital libraries  video analysis tool  Interactive TableTops  … November 4, 2006 CoPADD 2006 2
  • 3.
    Input Devices November 4,2006 CoPADD 2006 3
  • 4.
    Interactive Paper Architecture Print-n-Link: Weaving the Paper Web Moira C. Norrie Dept. of Computer Science ETH Zurich 8092 Zurich, Switzerland norrie@inf.ethz.ch Beat Signer Dept. of Computer Science ETH Zurich 8092 Zurich, Switzerland signer@inf.ethz.ch Nadir Weibel Dept. of Computer Science ETH Zurich 8092 Zurich, Switzerland weibel@inf.ethz.ch HTTP Request ABSTRACT Citations form the basis for a web of scientific publications. engines and web browsers. The ease with which we can not only find documents on the web, but also follow links ID, Page, (x,y) Encoded Search engines, embedded hyperlinks and digital libraries to related documents is considered to be one of the main all simplify the task of finding publications of interest on advantages of working with digital documents as opposed the web and navigating to cited publications or web sites. to paper equivalents. Y many studies have shown that et However the actual reading of publications often takes place paper is still the preferred med ium when it comes to reading on paper and frequently on the move. We present a system documents for reasons of comfort, mobility and also ease of Print-n-Link that uses technologies for interactive paper to annotation. enhance the reading process by enabling users to access dig- In this paper, we show how emerging technologies for in- ital information and/or searches for cited documents from a teractive paper enable a bridge to be built between paper printed version of a publication using a digital pen for inter- and digital document spaces that allows users to travel back and forth between the two using the medium most appropri- Position action. A special virtual printer driver automatically gener- ate to the current task and sit uation. Users can print doc- XIMA ates links from paper to digital services during the printing process based on an analysis of PDF documents. Depend- uments for reading and, by activating hyperlinks on paper ing on the user setting and interaction gesture, the system with a digital pen, they can not only access digital informa- may retrieve metadata about the citation and inform the tion about related documents, but also initiate web searches user through an audio channel or directly display the cited to retrieve copies of those documents. The retrieved docu- document on the user’s screen. ments may in turn be printed for reading with automatically generated embedded hyperlinks on the paper. Mobility is supported by providing disconnected users on the move with Categories and Subject Descriptors audio information about related documents and the option D.2.11 [ Software Engineering ]: Software Architectures; of adding them to a request list to be processed when recon- H.4.m [ Information Systems Applications ]: Miscella- nected. neous; H.5.m [ Information Interfaces and Presenta- As an application, we take the everyday academic task of tion ]: Miscellaneous reading scientific publications. Whether the reading activ- ity is for the purpose of knowledge discovery or reviewing, Response readers regularly look up citations to other publications by General Terms flicking to the reference list at the back of the paper. In some Design, Algorithms, Experimentation cases, a search for the cited publication may follow and, if AC found, often it is printed and added to a pile of documents to Keywords be read. Citations are the established means of providing the links that form the basis of the scientific web of publications Interactive paper, document integration, digital library, ci- that spans the digital and paper worlds. Just as citations tation management within digital documents can be used to automatically gen- erate hyperlinks to cited documents, we can automatically 1. INTRODUCTION generate links within paper documents through an analysis Nowadays we are all familiar with the notion of hyperlinks of the PDF file when the document is printed. as a means of creating interconnected document spaces that Section 2 presents a more detailed look at the activity iPaper can be easily searched and navigated with the help of search of reading scientific publications and a discussion of related work. Section 3 describes the digital pen and paper tech- nologies and how these can be used to support interaction between paper and digital services. We then present the Permission to make digital or hard copies of all or part of this work for general functionality of our enhanced reading system in Sec- personal or classroom use is granted without fee provided that copies are tion 4 and the overall architecture in Section 5. Details of not made or distributed for profit or commercial advantage and that copies the virtual printer component used to analyse PDF docu- bear this notice and the full citation on the first page. To copy otherwise, to ments for automatic link generation as well as printing are republish, to post on servers or to redistribute to lists, requires prior specific given in Section 6. Concluding remarks follow in Section 7. Link DB permission and/or a fee. DocEng’06, October 10–13, 2006, Amsterdam, The Netherlands. Copyright 2006 ACM 1-59593-515-0/06/0010 ...$5.00. Interactive iPaper Client and Document Output Device November 4, 2006 CoPADD 2006 4
  • 5.
    Input Devices <<interface>> InputHandler 0..* <<interface>> Location 0..* InputDeviceEvent +handle() +id: String +getTimestamp() : long +page: int +x: double +y: double Notifies AbstractInputDeviceEvent DefaultInputHandler ... TimestampedLocation Creates +timestamp: long 0..* InductivePenEvent NokiaPenEvent ... 1 1 Creates <<interface>> <<interface>> 1 0..* 1 InputDeviceEventListener InputDevice Notifies +inputDeviceEvent() +addInputDeviceEventListener() +addInputHandler() +removeInputDeviceEventListener() +removeInputHandler() AbstractInputDevice AbstractInputDeviceEventListener InductivePen NokiaPen ... InductivePenEventListener NokiaPenEventListener ... November 4, 2006 CoPADD 2006 5
  • 6.
    iServer Cross-Media Platform link Entities Selectors Resources Shape Page Query DB Object Time Span / Movie Clip XPointer XHTML Data/Metadata Spatial Info Document iPaper Application Database Movie Clips HTML Documents November 4, 2006 CoPADD 2006 6
  • 7.
    Interactive Paper Authoring iServer/iPaper interfaces  Java API  XML-based specification  Web Service interface  P2P interface  Manual link authoring  Content-driven publishing  Automatic link authoring November 4, 2006 CoPADD 2006 7
  • 8.
    iDoc Publishing Framework CMS iDOC Semantic Mapper Document Metadata Printer Plug-ins Driver Document DB Positions and Print Active Content iSERVER iPAPER November 4, 2006 CoPADD 2006 8
  • 9.
    Active Components Button Image Slider Sound Slider2D Movie DateSelector SQLClient Browser XCMRequest MapSelector ClientCtrlRequest Rating PaperPointControl Capture PaperPointAnnot CaptureAndSpeak ... November 4, 2006 CoPADD 2006 9
  • 10.
    Slider / Slider2DActive Component November 4, 2006 CoPADD 2006 10
  • 11.
    Conclusions  General frameworkfor interactive paper  Cross-media information server  extensibility (resource plug-ins)  link service acessible through various APIs  active content  information sharing  Constantly growing set of resource plug-ins and active components November 4, 2006 CoPADD 2006 11