Seminar presentation about cross-media information spaces given at EPFL.

1. 2 December 2005
Cross-Media Information Systems - Quo Vadis?
Prof. Beat Signer
http://www.beatsigner.com
Web & Information Systems Engineering Lab (WISE)
Department of Computer Science
Vrije Universiteit Brussel

2. 01001000100
0101001100010010100010010
0100100010010001000100001001000
100010001100001001000110010101010110
1100010010100010010010010100010101011001
101001100001100 101010100
00101000100 110101010101
00100 01010101000
1001 1111111111
111 0000000010
0 11111111
0101001011
010101010
111010101
001010010110
Fluid Cross-Media Information Spaces

3. Prof. Dr. Beat Signer
Interactive Paper, Cross-Media
Information Architectures
Dr. Bruno Dumas
Multimodal Interaction, Cross-
Media Information Architectures
Lode Hoste
Multimodal Interaction, Midas
and Mudra Framework
Brecht De Rooms
Implicit Context-aware Human-
Computer Interaction, HCII
CISA
Human-Machine &
Human-Information
Interaction
Information
Systems &
Management
Information
Visualisation
& Navigation
WEB & INFORMATION
SYSTEMS ENGINEERING
CROSS-MEDIA INFORMATION SPACES
AND ARCHITECTURES (CISA)

4. Sandra Trullemans
Personal Cross-Media
Information Management
Reinout Roels
MindXpres Next Generation
Presentation Tool
Ahmed A.O. Tayeh
Open Cross-Media Authoring,
Fluid Document Formats
WEB & INFORMATION
SYSTEMS ENGINEERING
CISA
Human-Machine &
Human-Information
Interaction
Information
Systems &
Management
Information
Visualisation
& Navigation
CROSS-MEDIA INFORMATION SPACES
AND ARCHITECTURES (CISA)

5. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012
PARC User Interface (Mid 70s)
 Desktop metaphor
 WIMP interaction
 "filing cabinets" with
hierarchical folders
 Paper paradigm
 "What You See Is What
You Get" (WYSIWYG)
 what about richer
document formats that
existed at that time?
Xerox Star 8010 [http://www.digibarn.com/collections/systems/xerox-8010/]

6. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012
Digital Documents as a Paper Simulator?
Vannevar Bush
Ted Nelson
Most people don't understand the logic of the
concept: "What You See Is What You Get" is
based on printing the document out ("get"
means "get WHEN YOU PRINT IT OUT"). And that
means a metaphysical shift: a document can
only consist of what can be printed! [...] No
overlays [...] – PAPER UNDER GLASS.
When data of any sort are placed in storage,
they are filed alphabetically or numerically,
and information is found (when it is) by
tracing it down from subclass to subclass. It
can be in only one place, unless duplicates
are used [...] The human mind does not work
that way. It operates by association.
As We May Think, Atlantic Monthly, July 1945
Geeks Bearing Gifts: How the Computer World Got This Way, Mindful Press 2009

7. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012
What is Wrong with Digital Documents?
 Existing document formats are based on the simulation
of paper affordances on desktop computers
 How to manage mixed-media "documents" in open and
fluid cross-media information spaces?
 on the data level
- context-sensitive adaptation
- cross-media transclusion
 on the visualisation and navigation level
- zoomable user interfaces
 on the cross-media interaction level
- fluid multimodal cross-media interfaces
 Remediation of the "paper simulation" approach
 WYSIWYG is only one out of many options!

8. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012
RSL Metamodel and iServer

9. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012
Links
Users
Layers
Plug-ins
Structure
Active
Content

10. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012
Bridging the Paper-Digital Divide

11. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012
Interactive Paper Applications
 Enhanced Reading
 interactive paper maps
 interactive festival brochures
 educational materials
 auction brochures
 scientific publications
 …
 Enhanced Writing
 photo album
 researcher's notebook
 mammography annotation
 query by sketching
 …
 Paper-based Interfaces
 PaperPoint
 document proof-editing
 digital libraries
 video analysis tool
 educational games
 Art Installations
 Lost Cosmonaut
 Generosa Enterprise
 Interactive Tabletops
 photo browser
 collaborative sketch tool

12. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012
Interactive Festival Guide (EdFest)

13. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012
The Lost Cosmonaut
 Writing as a collaborative
act of memory and story-
telling
 Reading, writing and
narrative as an act of
making sense
 Handwriting as a tool for
human-computer
interaction
In collaboration with Axel Vogelsang,
Artists in Labs Programme

14. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012
ArtVis

15. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012
ArtVis ...

16. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012
ArtVis ...
 Advanced visualisation
techniques in combination
with a TUI
 explore Web Gallery of Art
 faceted browsing
 Phidgets-based TUI
 RFID-tagged physical
objects
 Three main components
to explore, analyse and
browse the information

17. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012
RSL-based Associative File System
signer
My Pictures
Vancouver ER 2010
Granville
Island
Vancouver
Harbour
Public
Market
Keynote
J. Thorp
Olga and
Lamia
Dinner
ER 2010
multiple classification
structural link
navigational link
My Videos
Vancouver
Aquarium
ER 2010
Keynote
cross-media association
My Presentations
ER 2010 WISE 2009
cross-media
transclusion
single slide
transclusion
(via OOXML
RSL selector)

18. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012
RSL-based Associative File System ...

19. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012
What is Wrong with Slideware?
 Simulation of physical slides
 limited space due to the slide concept
 linear navigation from slide to slide
 difficult to reuse content and embed rich media types
 MindXpres addresses these issues
 unlimited canvas with zoomable user interface
 non-linear navigation and associative linking
 content-based approach with automatic visualisation

20. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012
MindXpres Presentation Tool
 HTML5-based extensible
core with three types of
JavaScript plug-ins
 components
- source code, graph, video, ...
 containers
- slide, image gallery, ...
 structures
- grid, graph, clustered, ...
 Presentation as RSL-
based overlay structure
 enables navigation across
different presentations

21. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012
MindXpres Presentation Tool ...

22. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012
MindXpres Presentation Tool ...
 Functional MindXpres prototype
 Ongoing work
 visual authoring tool
- alternative to XML-based authoring
 rich media plug-ins
- videos with subtitles, annotations etc.
- statistical data with alternative visualisations
- live sensor data
- ...
 audience-driven presentations
 collaborative annotations and information sharing
 multimodal interaction (e.g. integration of PaperPoint concepts)
 import of existing slideware formats

23. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012
Fluid Multimodal Cross-Media Interfaces
 Investigation of mobile and distributed document formats
 Mobile Cross-Media Applications (MobiCraNT)
 Passive or active information units (crosslets)
 processing and storage of sensor data
 binding of specific multimodal interactions to resources
 Implicit context-aware human-computer interaction (HCII)
 Personal Cross-Media Information Management

24. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012
 Midas/Mudra Framework
 declarative definition of multimodal
and multi-touch interactions
 rule-based language approach
 rapid protoyping and application development
 iGesture Workbench
 create and test gesture sets and algorithms
 different modalities: digital pen, Wii remote, …
 open source (www.igesture.org)
 SpeeG
 multimodal speech- and body gesture-based
text input
 real-time correction of recognised voice input
Multimodal and Multi-Touch Interaction

25. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012
Open Cross-Media Linking
 Four existing classes of
cross-media link systems
 A: single source and
single target type
 B: single source and
multiple target types
 C: multiple source and
single target type
 D: multiple source and
multiple target types
 Our proposed open cross-
media link approach covers
the whole space
# source types
1 2 3 4 5 ...
1
2
3
4
5
...
# target types
B1
B2
D2
open cross-media
linking
D1
A1... An C1 C2 C3

26. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012
Open Cross-Media Authoring
Cross-Media
Link Service
Client Application
Cross-Media
Browser & Editor
Visual Plug-ins
Resource Plug-in Repository
Data Plug-ins
Visual Plug-ins Data Plug-insVisual Plug-ins

27. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012
Open Cross-Media Authoring ...
 Integration of a new resource type includes
 development of data and visual plug-ins
 registration with the resource plug-in repository
 Cross-media browser loads plug-ins on demand
 internal or external (third party application) visualisation
 Third party applications delegate the visualisation of a
link target to the cross-media browser
 similar to "integration" of third-party applications in web browsers
 Non-monolithic link model and authoring tool
 resource-specific plug-ins

28. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012
Open Cross-Media Browser and Editor
Cross-media browser & editor Client application
showEntity(), getSelectedEntity()
showEntity()
C
R
U
D
R
S
L
Menu
Source Window Target Window

29. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012
Visual Plug-ins
 Cross-media browser deals only with RSL-specific data
but not with any media-specific details
 create/delete links (CRUD), user management, …
 Definition of link source and target anchors (resources or
selectors) handled by visual plug-ins
 separation of concerns
 Two basic purposes
 rendering of specific resource types (resources and selectors)
 creation and deletion of resources and selectors
 Well-defined visual plug-in API

30. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012
Conclusions
 Time to question existing document formats which are
based on the simulation of paper on desktop computers
 RSL-based representation of open and fluid cross-media
information spaces
 interactive paper solutions
 associative file system
 MindXpres presentation tool
 crosslet-based fluid multimodal interfaces
 open cross-media information browser and editor
 Interlinked structural components rather than a
document-centric representation of information
 data no longer "owned" by applications

31. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012
References
 Beat Signer, Fundamental Concepts for
Interactive Paper and Cross-Media Information
Spaces, ISBN 978-3-8370-2713-6, May 2008
 Vannevar Bush, As We May Think, Atlanic Monthly,
July 1945
 Beat Signer and Moira C. Norrie, As We May Link: A
General Metamodel for Hypermedia Systems,
Proceedings of ER 2007, Auckland, New Zealand,
November 2007

32. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012
References ...
 Beat Signer, What is Wrong with Digital
Documents? A Conceptual Model for Structural Cross-
Media Content Composition and Reuse, Proceedings of
ER 2010, Vancouver, Canada, November 2010
 Moira C. Norrie, Beat Signer, Michael Grossniklaus, Rudi
Belotti, Corsin Decurtins and Nadir Weibel, Context-
Aware Platform for Mobile Data Management, Journal on
Wireless Networks (WINET), Vol. 13, No. 6, Springer,
December 2007
 ArtVis
 http://wise.vub.ac.be/content/artvis-exploring-
information-through-advanced-visualisation-techniques

33. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012
References ...
 Axel Vogelsang and Beat Signer, The Lost
Cosmonaut: An Interactive Narrative Environment on
Basis of Digitally Enhanced Paper, Proceedings of
International Conference on Virtual Storytelling,
Strasbourg, France, December 2005
 Christophe Scholiers, Lode Hoste, Beat Signer and
Wolfgang De Meuter, Midas: A Declarative Multi-Touch
Interaction Framework, Proceedings of TEI 2011,
Funchal, Portugal, January 2011

34. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012
References ...
 Lode Hoste, Bruno Dumas and Beat Signer,
Mudra: A Unified Multimodal Interaction Framework,
Proceedings of ICMI 2011, Alicante, Spain, November
2011
 Beat Signer, Ueli Kurmann and Moira C. Norrie,
iGesture: A General Gesture Recognition Framework,
Proceedings of ICDAR 2007, Curitiba, Brazil, September
2007
 Lode Hoste, Bruno Dumas and Beat Signer, SpeeG: A
Multimodal Speech- and Gesture-based Text Input
Solution, Proceedings of AVI 2012, Capri Island, Italy,
May 2012

35. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012
References ...
 Beat Signer and Moira C. Norrie, A Model and
Architecture for Open Cross-Media Annotation and Link
Services, Information Systems 36(3), Elsevier, May 2011