Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Introducing Tangible Holograms for Data Physicalisation and Big Data Exploration

1,524 views

Published on

Seminar talk delivered in the Global Information Systems lab at ETH Zurich.

Published in: Science
  • Be the first to comment

  • Be the first to like this

Introducing Tangible Holograms for Data Physicalisation and Big Data Exploration

  1. 1. 2 December 2005 Introducing Tangible Holograms for Data Physicalisation and Big Data Exploration Prof. Beat Signer http://www.beatsigner.com Web & Information Systems Engineering Lab Department of Computer Science Vrije Universiteit Brussel WEB & INFORMATION SYSTEMS ENGINEERING
  2. 2. 01001000100 0101001100010010100010010 0100100010010001000100001001000 100010001100001001000110010101010110 1100010010100010010010010100010101011001 101001100001100 101010100 00101000100 110101010101 00100 01010101000 1001 1111111111 111 0000000010 0 11111111 0101001011 010101010 111010101 001010010110 Fluid Cross-Media Information Spaces
  3. 3. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 3September 23, 2017 Prof. Dr. Beat Signer Interactive Paper, Cross-Media Information Architectures Audrey Sanctorum User-defined Cross-Device and Cross-Media Interaction Reinout Roels MindXpres: Extensible Content- driven Presentation Tool CISA Human-Machine & Human-Information Interaction Information Systems & Management Information Visualisation & Navigation WEB & INFORMATION SYSTEMS ENGINEERING CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA) Payam Ebrahimi Data Physicalisation
  4. 4. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 4September 23, 2017 Dr. Ahmed A.O. Tayeh Open Cross-Media Authoring, Fluid Document Formats WEB & INFORMATION SYSTEMS ENGINEERING CISA Human-Machine & Human-Information Interaction Information Systems & Management Information Visualisation & Navigation CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA) Kushal Soni Smart Environments, Implicit Human-Computer Interaction Jan Maushagen Learning Analytics, Adaptive Persuasive ICT Tools Sandra Trullemans Personal Cross-Media Information Management
  5. 5. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 5September 23, 2017 WEB & INFORMATION SYSTEMS ENGINEERING CISA Human-Machine & Human-Information Interaction Information Systems & Management Information Visualisation & Navigation CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA) Lars Van Holsbeeke Smart Environments, Implicit Human-Computer Interaction
  6. 6. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 6September 23, 2017 Cross-Media Information Spaces and Architectures Information Systems & Management Information Visualisation & Navigation Information Visualisation nd NavigationHuman-Machine & Human-Information Interaction CISA RSL [4] MindXpres [7] OC2 [5] associate file system [2] data-driven storytelling [10] XIMA [12] cross-device interaction [8] iServer/iPaper [3] ViDaX [15] data physicalisation [30] map-based interaction [17] non-linear storytelling [25] PaperProof [27]iGesture [21] Query by Sketch [28] PaperSketch [29] Context Modelling Toolkit [9] Midas [19] SpeeG2 [22] PaperPoint [24] Print-n-Link [16] digital libraries [11] Mudra [20] mixed reality [26] ArtVis [14] open cross-media linking [1] PimVis [13] DocTr [6] EdFest [23] source code visualisation [18]
  7. 7. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 7September 23, 2017 ArtVis: TUI-based Information Exploration  Advanced visualisation techniques in combination with a TUI  explore Web Gallery of Art  faceted browsing  phidgets-based TUI  RFID-tagged physical objects  Three main components to explore, analyse and browse the information Bram Moerman
  8. 8. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 8September 23, 2017 ArtVis Setup Bram Moerman
  9. 9. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 9September 23, 2017 ArtVis Visualisation Bram Moerman
  10. 10. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 10September 23, 2017 Tangible Bits (Ishii & Ullmer, 1997)  Beyond GUIs ("painted bits")  Tangible User Interfaces (TUIs) augment the physical space by coupling digital information to everyday objects and environments  physical instantiation of GUI elements in TUI  Key concepts  interactive surfaces  unidirectional coupling of atoms and bits  ambient media (inspired by Life Wire)  Tangible bits mainly act as input controls for digital information and services (output on computer screens) Hiroshi Ishii Brygg Ullmer
  11. 11. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 11September 23, 2017 Radical Atoms (Ishii et al., 2012)  Vision taking a leap beyond Tangible Bits  assuming a hypothetical generation of materials that can dynamically change their form and appearance  vision-driven design research  Radical Atoms is  a computationally transformable and reconfigurable material that is bidirectionally coupled with an underlying digital model (bits) - changes to digital model are reflected in physical representation and vice versa  a future material that can transform its shape, conform to constraints and inform the users of its affordances  a vision for the future of human-material interaction and Material User Interfaces (MUIs) in which all digital information has a physical manifestation so that we can directly interact with it
  12. 12. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 12September 23, 2017 Radical Atoms …  Transform  interface can transform its shape to modify the model and reflect changes in the computational model  Conform  interface has to conform to some physical laws and user constraints (e.g. for safety)  Inform  user has to be informed about changing interface affordances
  13. 13. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 13September 23, 2017 Interactions with Radical Atoms  Direct touch and gestural interaction  gestures coupled with direct touch create an interaction appropriate for Radical Atoms since users are able to rapidly reform dynamic materials at all scales  Context-aware transformation  context-aware transformations of the tool/interface - e.g. screwdriver adapting to the type of screw it is operating on  Shape-memory clay: Perfect Red  Perfect Red is a fictional material that can be sculpted like and responds according to rules inspired by CAD operations - e.g. if we split a piece in two even halves, then the operations performed on one part are mirrored on the other part
  14. 14. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 14September 23, 2017 TRANSFORM TRANSFORM, MIT  Dynamic shape display that can physically render 3D content  tangible interaction with digital content - geospatial data - 3D modelling - …  TRANSFORM display can also interact with the physical world around it  remote users can be displayed physically  Step towards MIT’s vision of Radical Atoms
  15. 15. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 15September 23, 2017 Video: TRANSFORM
  16. 16. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 16September 23, 2017 Data Physicalisation (Jansen et al., 2015)  Physical objects used for output  often static encoding of digital information  physical visualisations can outperform their virtual counterparts  How can we achieve dynamic data physicalisation with dynamic affordances  e.g. 3D visualisation with additional dimensions represented by physical variables such as temperature, texture, weight or plasticity  TRANSFORM is one possible approach; tangible holograms are another experimental mobile solution - can be used for multidimensional collaborative exploration and analysis of big data sets
  17. 17. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 17September 23, 2017 Mixed Reality "The perceived congruence concerns both visual as well as tactile perception. A perfect hologram will appear embodied but trying to touch it will destroy the illusion." Microsoft HoloLens Jansen, 2014 [34]
  18. 18. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 18September 23, 2017 Tangible Hologram (TangHo) Setup Timothy J. Curtin
  19. 19. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 19September 23, 2017 Tangible Graph Hologram
  20. 20. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 20September 23, 2017 Collaborative TangHo Interaction
  21. 21. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 21September 23, 2017 Robotic Arm Prototype  Lego Mindstorms-based 6DOF arm prototype  motors can be locked or set to float mode - useful for bidirectional I/O  replaceable sphere for different types of non-visual feedback  Challenges  inverse kinematics (final system will be body mounted) - use robotics toolbox (MATLAB)  limited HoloLens hand tracking  3D printing of final arm Timothy J. Curtin
  22. 22. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 22September 23, 2017 TangHo Prototype Timothy J. Curtin
  23. 23. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 23September 23, 2017 TangHo Interactions Timothy J. Curtin
  24. 24. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 24September 23, 2017 Data Physicalisation Framework  Inspired by research agenda of Jansen et al. [34]  Understanding the design space  how do users map data to physicalisations  Understanding the perceptual effectiveness  e.g. what are the just-noticeable differences that a physical variable can convey?  Framework for dynamic data physicalisation  data processing with device-independent output - TextDIO: text for static or dynamic reports - PhysDIO: data points, data variables and physical variables  text and visualisation drivers for representation on specific devices - e.g. tangible hologram prototype
  25. 25. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 25September 23, 2017 Conclusions  Tangible holograms for data physicalisation and simulation of radical atoms  physical augmentation of highly configurable virtual objects rather than programming of physical material (programmable matter)  mobile solution that is no longer bound to a specific place such as a the TRANSFORM table  modular solution that can be configured for specific domains (e.g. different spheres for various physical variables)  Data physicalisation framework for investigating new opportunities in dynamic data physicalisation  new data physicalisation design guidelines (as in InfoVis)  new evaluation techniques  …
  26. 26. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 26September 23, 2017 References  B. Signer, Fundamental Concepts for Interactive Paper and Cross-Media Information Spaces, ISBN 978-3-8370-2713-6, May 2008  ArtVis Project  http://www.beatsigner.com/flyers/ArtVis.pdf  Tangible Holograms  http://beatsigner.com/tangibleHolograms.html  TRANSFORM  https://www.youtube.com/watch?v=lCARHatJQJA
  27. 27. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 27September 23, 2017 References ... [1] B. Signer and M.C. Norrie, A Model and Architecture for Open Cross-Media Annotation and Link Services, Information Systems 36(3), Elsevier, May 2011  http://beatsigner.com/publications/signer_IS2011.pdf [2] B. Signer, What is Wrong with Digital Documents? A Conceptual Model for Structural Cross-Media Content Composition and Reuse, Proceedings of the 29th International Conference on Conceptual Modeling (ER 2010), Vancouver, Canada, November 2010  http://beatsigner.com/publications/signer_ER2010.pdf
  28. 28. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 28September 23, 2017 References ... [3] B. Signer, Fundamental Concepts for Interactive Paper and Cross-Media Information Spaces, Dissertation ETH No. 16218, Zurich, Switzerland, 2005  http://beatsigner.com/publications/signer2006.pdf [4] B. Signer and M.C. Norrie, As We May Link: A General Metamodel for Hypermedia Systems, Proceedings of the 26th International Conference on Conceptual Modeling (ER 2007), Auckland, New Zealand, November 2007  http://beatsigner.com/publications/signer_ER2010.pdf
  29. 29. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 29September 23, 2017 References ... [5] S. Trullemans and B. Signer, Towards a Con- ceptual Framework and Metamodel for Context-Aware Personal Cross-Media Information Management Systems, Proceedings of ER 2014, 33rd International Conference on Conceptual Modelling, Atlanta, USA, October, 2014  http://beatsigner.com/publications/trullemans_ER2014.pdf [6] S. Trullemans, A. Vercruysse and B. Signer, DocTr: A Unifying Framework for Tracking Physical Docu- ments and Organisational Structures, Proceedings of EICS 2016, Brussels, Belgium, June 2016  http://beatsigner.com/publications/trullemans_EICS2016.pdf
  30. 30. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 30September 23, 2017 References ... [7] R. Roels and B. Signer, MindXpres: An Exten- sible Content-driven Cross-Media Presentation Platform, Proceedings of WISE 2014, 15th Interna- tional Conference on Web Information System Engineering, Thessaloniki, Greece, October, 2014  http://beatsigner.com/publications/roels_WISE2014.pdf [8] A. Sanctorum and B. Signer, Towards User-defined Cross-Device Interaction, Proceedings of DUI 2016, 5th Workshop on Distributed User Interfaces, Lugano, Switzerland, June 2016  http://beatsigner.com/publications/sanctorum_DUI2016.pdf
  31. 31. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 31September 23, 2017 References ... [9] S. Trullemans and B. Signer, A Multi-layered Context Modelling Approach for End Users, Expert Users and Programmers, Proceedings of the Interna- tional Workshop on Smart Ecosystems cReation by Visual dEsign (SERVE 2016), Bari, Italy, June 2016  http://beatsigner.com/publications/trullemans_SERVE2016.pdf [10] R. Roels, Y. Baeten and B. Signer, An Interactive Data Visualisation Approach for Next Generation Presentation Tools: Towards Rich Presentation-based Data Exploration and Storytelling, Proceedings of CSEDU 2016, Rome, Italy, April, 2016  http://beatsigner.com/publications/roels_CSEDU2016.pdf
  32. 32. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 32September 23, 2017 References ... [11] M.C. Norrie, B. Signer and N. Weibel, Interactive Paper as a Reading Medium in Digital Libraries, Pro- ceedings of ECDL 2008, 12th European Conference on Research and Advanced Technology for Digital Libraries, Aarhus, Denmark, September 2008  http://beatsigner.com/publications/norrie_ECDL2008.pdf [12] B. Signer, M.C. Norrie, P. Geissbuehler and D. Heiniger, Telephone Interface for Avalanche War- nings based on Information Server for Adaptable Con- tent Delivery, Proceedings of Pervasive 2002, Zurich, Switzerland, August 2002  http://beatsigner.com/publications/signer_Pervasive2002.pdf
  33. 33. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 33September 23, 2017 References ... [13] S. Trullemans, A. Sanctorum and B. Signer, PimVis: Exploring and Re-finding Documents in Cross- Media Information Spaces, Proceedings of the Inter- national Working Conference on Advanced Visual Interfaces (AVI 2016), Bari, Italy, June 2016  http://beatsigner.com/publications/trullemans_AVI2016.pdf [14] B. Dumas, B. Moerman, S. Trullemans and B. Signer, ArtVis: Combining Advanced Visualisation and Tan- gible Interaction for the Exploration, Analysis and Browsing of Digital Artwork Collections, Proceedings of AVI 2014, Como, Italy, May 2014  http://beatsigner.com/publications/dumas_AVI2014.pdf
  34. 34. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 34September 23, 2017 References ... [15] B. Dumas, T. Broché, L. Hoste and B. Signer, ViDaX: An Interactive Semantic Data Visualisation and Exploration Tool, Proceedings of AVI 2012, Internatio- nal Working Conference on Advanced Visual Inter- faces, Capri Island, Italy, May 2012  http://beatsigner.com/publications/dumas_AVI2012.pdf [16] Moira C. Norrie, Beat Signer and Nadir Weibel, Print-n-Link: Weaving the Paper Web, Proceedings of DocEng 2006, ACM Symposium on Document Engineering, Amsterdam, The Netherlands, October 2006  http://beatsigner.com/publications/norrie_DocEng2006.pdf
  35. 35. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 35September 23, 2017 References ... [17] M. Grossniklaus, M.C. Norrie, B. Signer and N. Weibel ,Putting Location-Based Services on the Map, Proceedings of W2GIS 2006, Hong Kong, China, December 2006  http://beatsigner.com/publications/grossniklaus_W2GIS2006.pdf [18] R. Roels, P. Mestereaga and B. Signer, An Interactive Source Code Visualisation Plug-in for the MindXpres Presentation Platform, Communications in Computer and Information Science (CCIS), 583, 2016  http://beatsigner.com/publications/roels_CCIS2016.pdf
  36. 36. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 36September 23, 2017 References ... [19] C. Scholiers, L. Hoste, B. Signer and W. De Meuter, Midas: A Declarative Multi-Touch Interaction Framework, Proceedings of TEI 2011, 5th Interna- tional Conference on Tangible, Embedded and Em- bodied Interaction, Funchal, Portugal, January 2006  http://beatsigner.com/publications/scholliers_TEI2011.pdf [20] L. Hoste, B. Dumas and B. Signer, Mudra: A Unified Multimodal Interaction Framework, Proceedings of ICMI 2011, 13th International Conference on Multi- modal Interaction (ICMI 2011), Alicante, Spain, November 2011  http://beatsigner.com/publications/scholliers_TEI2011.pdf
  37. 37. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 37September 23, 2017 References ... [21] B. Signer, U. Kurmann and M.C. Norrie, iGesture: A General Gesture Recognition Framework, Proceedings of the 9th International Conference on Document Analysis and Recognition (ICDAR 2007), Curitiba, Brazil, September 2007  http://beatsigner.com/publications/signer_ICDAR2007.pdf [22] L. Hoste and B. Signer, SpeeG2: A Speech- and Gesture-based Interface for Efficient Controller-free Text Entry, Proceedings of ICMI 2013, 15th Inter- national Conference on Multimodal Interaction, Sydney, Australia, December 2013  http://beatsigner.com/publications/hoste_ICMI2013.pdf
  38. 38. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 38September 23, 2017 References ... [23] B. Signer, M. Grossniklaus and M.C. Norrie, Interactive Paper as a Mobile Client for a Multi- Channel Web Information System, World Wide Web Journal (WWW), Vol. 10, No. 4, Springer, December 2007  http://beatsigner.com/publications/signer_WWWJ2007.pdf [24] B. Signer and M.C. Norrie, PaperPoint: A Paper- based Presentation and Interactive Paper Prototyping Tool, Proceedings of TEI 2007, First International Conference on Tangible and Embedded Interaction, Baton Rouge, USA, February 2007  http://beatsigner.com/publications/signer_TEI2007.pdf
  39. 39. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 39September 23, 2017 References ... [25] A. Vogelsang and B. Signer, The Lost Cosmo- naut: An Interactive Narrative Environment on Basis of Digitally Enhanced Paper, Proceedings of the Inter- national Conference on Virtual Storytelling 2005, Strasbourg, France, December 2005  http://beatsigner.com/publications/vogelsang_ICVS2005.pdf [26] L. Hoste and B. Signer, Expressive Control of Indirect Augmented Reality During Live Music Performances, Proceedings of NIME 2013, 13th International Confe- rence on New Interfaces for Musical Expression, Daejeon, Korea Republic, May 2013  http://beatsigner.com/publications/hoste_NIME2013.pdf
  40. 40. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 40September 23, 2017 References ... [27] N. Weibel, A. Ispas, B. Signer and M.C. Norrie, PaperProof: A Paper-Digital Proof-Editing System, Proceedings of CHI 2008, 26th International Confe- rence on Human Factors in Computing Systems (Interactivity Track), Florence, Italy, April 2008  http://beatsigner.com/publications/weibel_CHI2008.pdf [28] A. Ispas, H. Schuldt, M.C. Norrie and B. Signer, Towards Query by Sketch, Michael Springmann, Second DELOS Conference on Digital Libraries, Pisa, Italy, December 2007  http://beatsigner.com/publications/springmann_DELOS2007.pdf
  41. 41. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 41September 23, 2017 References ... [29] N. Weibel, B. Signer, M.C. Norrie, H. Hofstetter, H.-C. Jetter and H. Reiterer, PaperSketch: A Paper- Digital Collaborative Remote Sketching Tool, Proceed- ings of IUI 2011, International Conference on Intelli- gent User Interfaces, Palo Alto, USA, February 2011  http://beatsigner.com/publications/weibel_IUI2011.pdf [30] B. Signer and T.J. Curtin, Tangible Holograms: Towards Mobile Physical Augmentation of Virtual Objects, Technical Report WISE Lab, WISE-2017-01, March 2017  http://beatsigner.com/publications/signer_arXiv2017.pdf
  42. 42. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 42September 23, 2017 References ... [31] H. Ishii and B. Ullmer, Tangible Bits: Towards Seamless Interfaces between People, Bits and Atoms, Proceedings of CHI 1997, ACM Conference on Human Factors in Computing Systems, Atlanta, USA, March 1997  http://dx.doi.org/10.1145/258549.258715 [32] H. Ishii, D. Lakatos, L. Bonanni and J.-B. Labrune, Radical Atoms: Beyond Tangible Bits, Toward Transformable Materials, interactions, 19(1), January 2012  http://dx.doi.org/10.1145/2065327.2065337
  43. 43. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 43September 23, 2017 References ... [33] Y. Jansen et al., Opportunities and Challenges for Data Physicalizations, Proceedings of CHI 2015, ACM Conference on Human Factors in Computing Systems, Seoul, Korea, April 2015  https://doi.org/10.1145/2702123.2702180 [34] Y. Jansen, Physical and Tangible Information Visualization, 2014, Ph.D. Dissertation, Université Paris-Sud  https://tel.archives-ouvertes.fr/tel-00981521

×