Cross-Media Information Systems - Quo Vadis?

8,119 views

Published on

Seminar presentation about cross-media information spaces given at EPFL.

Published in: Science
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
8,119
On SlideShare
0
From Embeds
0
Number of Embeds
6,550
Actions
Shares
0
Downloads
30
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Cross-Media Information Systems - Quo Vadis?

  1. 1. 2 December 2005 Cross-Media Information Systems - Quo Vadis? Prof. Beat Signer http://www.beatsigner.com Web & Information Systems Engineering Lab (WISE) Department of Computer Science Vrije Universiteit Brussel
  2. 2. 01001000100 0101001100010010100010010 0100100010010001000100001001000 100010001100001001000110010101010110 1100010010100010010010010100010101011001 101001100001100 101010100 00101000100 110101010101 00100 01010101000 1001 1111111111 111 0000000010 0 11111111 0101001011 010101010 111010101 001010010110 Fluid Cross-Media Information Spaces
  3. 3. Prof. Dr. Beat Signer Interactive Paper, Cross-Media Information Architectures Dr. Bruno Dumas Multimodal Interaction, Cross- Media Information Architectures Lode Hoste Multimodal Interaction, Midas and Mudra Framework Brecht De Rooms Implicit Context-aware Human- Computer Interaction, HCII CISA Human-Machine & Human-Information Interaction Information Systems & Management Information Visualisation & Navigation WEB & INFORMATION SYSTEMS ENGINEERING CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA)
  4. 4. Sandra Trullemans Personal Cross-Media Information Management Reinout Roels MindXpres Next Generation Presentation Tool Ahmed A.O. Tayeh Open Cross-Media Authoring, Fluid Document Formats WEB & INFORMATION SYSTEMS ENGINEERING CISA Human-Machine & Human-Information Interaction Information Systems & Management Information Visualisation & Navigation CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA)
  5. 5. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012 PARC User Interface (Mid 70s)  Desktop metaphor  WIMP interaction  "filing cabinets" with hierarchical folders  Paper paradigm  "What You See Is What You Get" (WYSIWYG)  what about richer document formats that existed at that time? Xerox Star 8010 [http://www.digibarn.com/collections/systems/xerox-8010/]
  6. 6. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012 Digital Documents as a Paper Simulator? Vannevar Bush Ted Nelson Most people don't understand the logic of the concept: "What You See Is What You Get" is based on printing the document out ("get" means "get WHEN YOU PRINT IT OUT"). And that means a metaphysical shift: a document can only consist of what can be printed! [...] No overlays [...] – PAPER UNDER GLASS. When data of any sort are placed in storage, they are filed alphabetically or numerically, and information is found (when it is) by tracing it down from subclass to subclass. It can be in only one place, unless duplicates are used [...] The human mind does not work that way. It operates by association. As We May Think, Atlantic Monthly, July 1945 Geeks Bearing Gifts: How the Computer World Got This Way, Mindful Press 2009
  7. 7. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012 What is Wrong with Digital Documents?  Existing document formats are based on the simulation of paper affordances on desktop computers  How to manage mixed-media "documents" in open and fluid cross-media information spaces?  on the data level - context-sensitive adaptation - cross-media transclusion  on the visualisation and navigation level - zoomable user interfaces  on the cross-media interaction level - fluid multimodal cross-media interfaces  Remediation of the "paper simulation" approach  WYSIWYG is only one out of many options!
  8. 8. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012 RSL Metamodel and iServer
  9. 9. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012 Links Users Layers Plug-ins Structure Active Content
  10. 10. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012 Bridging the Paper-Digital Divide
  11. 11. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012 Interactive Paper Applications  Enhanced Reading  interactive paper maps  interactive festival brochures  educational materials  auction brochures  scientific publications  …  Enhanced Writing  photo album  researcher's notebook  mammography annotation  query by sketching  …  Paper-based Interfaces  PaperPoint  document proof-editing  digital libraries  video analysis tool  educational games  Art Installations  Lost Cosmonaut  Generosa Enterprise  Interactive Tabletops  photo browser  collaborative sketch tool
  12. 12. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012 Interactive Festival Guide (EdFest)
  13. 13. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012 The Lost Cosmonaut  Writing as a collaborative act of memory and story- telling  Reading, writing and narrative as an act of making sense  Handwriting as a tool for human-computer interaction In collaboration with Axel Vogelsang, Artists in Labs Programme
  14. 14. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012 ArtVis
  15. 15. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012 ArtVis ...
  16. 16. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012 ArtVis ...  Advanced visualisation techniques in combination with a TUI  explore Web Gallery of Art  faceted browsing  Phidgets-based TUI  RFID-tagged physical objects  Three main components to explore, analyse and browse the information
  17. 17. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012 RSL-based Associative File System signer My Pictures Vancouver ER 2010 Granville Island Vancouver Harbour Public Market Keynote J. Thorp Olga and Lamia Dinner ER 2010 multiple classification structural link navigational link My Videos Vancouver Aquarium ER 2010 Keynote cross-media association My Presentations ER 2010 WISE 2009 cross-media transclusion single slide transclusion (via OOXML RSL selector)
  18. 18. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012 RSL-based Associative File System ...
  19. 19. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012 What is Wrong with Slideware?  Simulation of physical slides  limited space due to the slide concept  linear navigation from slide to slide  difficult to reuse content and embed rich media types  MindXpres addresses these issues  unlimited canvas with zoomable user interface  non-linear navigation and associative linking  content-based approach with automatic visualisation
  20. 20. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012 MindXpres Presentation Tool  HTML5-based extensible core with three types of JavaScript plug-ins  components - source code, graph, video, ...  containers - slide, image gallery, ...  structures - grid, graph, clustered, ...  Presentation as RSL- based overlay structure  enables navigation across different presentations
  21. 21. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012 MindXpres Presentation Tool ...
  22. 22. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012 MindXpres Presentation Tool ...  Functional MindXpres prototype  Ongoing work  visual authoring tool - alternative to XML-based authoring  rich media plug-ins - videos with subtitles, annotations etc. - statistical data with alternative visualisations - live sensor data - ...  audience-driven presentations  collaborative annotations and information sharing  multimodal interaction (e.g. integration of PaperPoint concepts)  import of existing slideware formats
  23. 23. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012 Fluid Multimodal Cross-Media Interfaces  Investigation of mobile and distributed document formats  Mobile Cross-Media Applications (MobiCraNT)  Passive or active information units (crosslets)  processing and storage of sensor data  binding of specific multimodal interactions to resources  Implicit context-aware human-computer interaction (HCII)  Personal Cross-Media Information Management
  24. 24. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012  Midas/Mudra Framework  declarative definition of multimodal and multi-touch interactions  rule-based language approach  rapid protoyping and application development  iGesture Workbench  create and test gesture sets and algorithms  different modalities: digital pen, Wii remote, …  open source (www.igesture.org)  SpeeG  multimodal speech- and body gesture-based text input  real-time correction of recognised voice input Multimodal and Multi-Touch Interaction
  25. 25. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012 Open Cross-Media Linking  Four existing classes of cross-media link systems  A: single source and single target type  B: single source and multiple target types  C: multiple source and single target type  D: multiple source and multiple target types  Our proposed open cross- media link approach covers the whole space # source types 1 2 3 4 5 ... 1 2 3 4 5 ... # target types B1 B2 D2 open cross-media linking D1 A1... An C1 C2 C3
  26. 26. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012 Open Cross-Media Authoring Cross-Media Link Service Client Application Cross-Media Browser & Editor Visual Plug-ins Resource Plug-in Repository Data Plug-ins Visual Plug-ins Data Plug-insVisual Plug-ins
  27. 27. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012 Open Cross-Media Authoring ...  Integration of a new resource type includes  development of data and visual plug-ins  registration with the resource plug-in repository  Cross-media browser loads plug-ins on demand  internal or external (third party application) visualisation  Third party applications delegate the visualisation of a link target to the cross-media browser  similar to "integration" of third-party applications in web browsers  Non-monolithic link model and authoring tool  resource-specific plug-ins
  28. 28. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012 Open Cross-Media Browser and Editor Cross-media browser & editor Client application showEntity(), getSelectedEntity() showEntity() C R U D R S L Menu Source Window Target Window
  29. 29. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012 Visual Plug-ins  Cross-media browser deals only with RSL-specific data but not with any media-specific details  create/delete links (CRUD), user management, …  Definition of link source and target anchors (resources or selectors) handled by visual plug-ins  separation of concerns  Two basic purposes  rendering of specific resource types (resources and selectors)  creation and deletion of resources and selectors  Well-defined visual plug-in API
  30. 30. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012 Conclusions  Time to question existing document formats which are based on the simulation of paper on desktop computers  RSL-based representation of open and fluid cross-media information spaces  interactive paper solutions  associative file system  MindXpres presentation tool  crosslet-based fluid multimodal interfaces  open cross-media information browser and editor  Interlinked structural components rather than a document-centric representation of information  data no longer "owned" by applications
  31. 31. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012 References  Beat Signer, Fundamental Concepts for Interactive Paper and Cross-Media Information Spaces, ISBN 978-3-8370-2713-6, May 2008  Vannevar Bush, As We May Think, Atlanic Monthly, July 1945  Beat Signer and Moira C. Norrie, As We May Link: A General Metamodel for Hypermedia Systems, Proceedings of ER 2007, Auckland, New Zealand, November 2007
  32. 32. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012 References ...  Beat Signer, What is Wrong with Digital Documents? A Conceptual Model for Structural Cross- Media Content Composition and Reuse, Proceedings of ER 2010, Vancouver, Canada, November 2010  Moira C. Norrie, Beat Signer, Michael Grossniklaus, Rudi Belotti, Corsin Decurtins and Nadir Weibel, Context- Aware Platform for Mobile Data Management, Journal on Wireless Networks (WINET), Vol. 13, No. 6, Springer, December 2007  ArtVis  http://wise.vub.ac.be/content/artvis-exploring- information-through-advanced-visualisation-techniques
  33. 33. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012 References ...  Axel Vogelsang and Beat Signer, The Lost Cosmonaut: An Interactive Narrative Environment on Basis of Digitally Enhanced Paper, Proceedings of International Conference on Virtual Storytelling, Strasbourg, France, December 2005  Christophe Scholiers, Lode Hoste, Beat Signer and Wolfgang De Meuter, Midas: A Declarative Multi-Touch Interaction Framework, Proceedings of TEI 2011, Funchal, Portugal, January 2011
  34. 34. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012 References ...  Lode Hoste, Bruno Dumas and Beat Signer, Mudra: A Unified Multimodal Interaction Framework, Proceedings of ICMI 2011, Alicante, Spain, November 2011  Beat Signer, Ueli Kurmann and Moira C. Norrie, iGesture: A General Gesture Recognition Framework, Proceedings of ICDAR 2007, Curitiba, Brazil, September 2007  Lode Hoste, Bruno Dumas and Beat Signer, SpeeG: A Multimodal Speech- and Gesture-based Text Input Solution, Proceedings of AVI 2012, Capri Island, Italy, May 2012
  35. 35. Beat Signer - Department of Computer Science - bsigner@vub.ac.beNovember 2, 2012 References ...  Beat Signer and Moira C. Norrie, A Model and Architecture for Open Cross-Media Annotation and Link Services, Information Systems 36(3), Elsevier, May 2011

×