Introducing Tangible Holograms for Data Physicalisation and Big Data Exploration

2 December 2005
Introducing Tangible Holograms for
Data Physicalisation and Big Data Exploration
Prof. Beat Signer
http://www.beatsigner.com
Web & Information Systems Engineering Lab
Department of Computer Science
Vrije Universiteit Brussel
WEB & INFORMATION
SYSTEMS ENGINEERING

01001000100
0101001100010010100010010
0100100010010001000100001001000
100010001100001001000110010101010110
1100010010100010010010010100010101011001
101001100001100 101010100
00101000100 110101010101
00100 01010101000
1001 1111111111
111 0000000010
0 11111111
0101001011
010101010
111010101
001010010110
Fluid Cross-Media Information Spaces

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 3September 23, 2017
Prof. Dr. Beat Signer
Interactive Paper, Cross-Media
Information Architectures
Audrey Sanctorum
User-defined Cross-Device and
Cross-Media Interaction
Reinout Roels
MindXpres: Extensible Content-
driven Presentation Tool
CISA
Human-Machine &
Human-Information
Interaction
Information
Systems &
Management
Information
Visualisation
& Navigation
WEB & INFORMATION
SYSTEMS ENGINEERING
CROSS-MEDIA INFORMATION SPACES
AND ARCHITECTURES (CISA)
Payam Ebrahimi
Data Physicalisation

Dr. Ahmed A.O. Tayeh
Open Cross-Media Authoring,
Fluid Document Formats
WEB & INFORMATION
SYSTEMS ENGINEERING
CISA
Human-Machine &
Human-Information
Interaction
Information
Systems &
Management
Information
Visualisation
& Navigation
Kushal Soni
Smart Environments, Implicit
Human-Computer Interaction
Jan Maushagen
Learning Analytics, Adaptive
Persuasive ICT Tools
Sandra Trullemans
Personal Cross-Media
Information Management

WEB & INFORMATION
SYSTEMS ENGINEERING
CISA
Human-Machine &
Human-Information
Interaction
Information
Systems &
Management
Information
Visualisation
& Navigation
Lars Van Holsbeeke
Smart Environments, Implicit
Human-Computer Interaction

Cross-Media Information Spaces and Architectures
Information
Systems &
Management
Information
Visualisation
& Navigation
Information
Visualisation
nd NavigationHuman-Machine &
Human-Information
Interaction
CISA
RSL [4]
MindXpres [7]
OC2 [5]
associate file system [2]
data-driven storytelling [10]
XIMA [12]
cross-device interaction [8]
iServer/iPaper [3]
ViDaX [15]
data physicalisation [30]
map-based interaction [17]
non-linear storytelling [25]
PaperProof [27]iGesture [21]
Query by Sketch [28]
PaperSketch [29]
Context Modelling Toolkit [9]
Midas [19]
SpeeG2 [22]
PaperPoint [24]
Print-n-Link [16]
digital libraries [11]
Mudra [20] mixed reality [26]
ArtVis [14]
open cross-media linking [1]
PimVis [13]
DocTr [6]
EdFest [23]
source code visualisation [18]

ArtVis: TUI-based Information Exploration
 Advanced visualisation
techniques in combination
with a TUI
 explore Web Gallery of Art
 faceted browsing
 phidgets-based TUI
 RFID-tagged physical
objects
 Three main components
to explore, analyse and
browse the information
Bram Moerman

ArtVis Setup
Bram Moerman

ArtVis Visualisation
Bram Moerman

Tangible Bits (Ishii & Ullmer, 1997)
 Beyond GUIs ("painted bits")
 Tangible User Interfaces (TUIs)
augment the physical space by
coupling digital information to
everyday objects and environments
 physical instantiation of GUI
elements in TUI
 Key concepts
 interactive surfaces
 unidirectional coupling of atoms and bits
 ambient media (inspired by Life Wire)
 Tangible bits mainly act as input controls for digital
information and services (output on computer screens)
Hiroshi Ishii Brygg Ullmer

Radical Atoms (Ishii et al., 2012)
 Vision taking a leap beyond Tangible Bits
 assuming a hypothetical generation of materials that can
dynamically change their form and appearance
 vision-driven design research
 Radical Atoms is
 a computationally transformable and reconfigurable material that
is bidirectionally coupled with an underlying digital model (bits)
- changes to digital model are reflected in physical representation and vice versa
 a future material that can transform its shape, conform to
constraints and inform the users of its affordances
 a vision for the future of human-material interaction and Material
User Interfaces (MUIs) in which all digital information has a
physical manifestation so that we can directly interact with it

Radical Atoms …
 Transform
 interface can transform its
shape to modify the model
and reflect changes in the
computational model
 Conform
 interface has to conform to
some physical laws and user
constraints (e.g. for safety)
 Inform
 user has to be informed
about changing interface
affordances

Interactions with Radical Atoms
 Direct touch and gestural interaction
 gestures coupled with direct touch create an interaction
appropriate for Radical Atoms since users are able to rapidly
reform dynamic materials at all scales
 Context-aware transformation
 context-aware transformations of the tool/interface
- e.g. screwdriver adapting to the type of screw it is operating on
 Shape-memory clay: Perfect Red
 Perfect Red is a fictional material that can be sculpted like and
responds according to rules inspired by CAD operations
- e.g. if we split a piece in two even halves, then the operations performed on
one part are mirrored on the other part

TRANSFORM
TRANSFORM, MIT
 Dynamic shape display
that can physically render
3D content
 tangible interaction with
digital content
- geospatial data
- 3D modelling
- …
 TRANSFORM display can also
interact with the physical world around it
 remote users can be displayed physically
 Step towards MIT’s vision of Radical Atoms

Video: TRANSFORM

Data Physicalisation (Jansen et al., 2015)
 Physical objects used for output
 often static encoding of digital
information
 physical visualisations can outperform
their virtual counterparts
 How can we achieve dynamic data
physicalisation with dynamic affordances
 e.g. 3D visualisation with additional dimensions represented
by physical variables such as temperature, texture, weight or
plasticity
 TRANSFORM is one possible approach; tangible holograms are
another experimental mobile solution
- can be used for multidimensional collaborative exploration and analysis of
big data sets

Mixed Reality
"The perceived congruence concerns both visual as well
as tactile perception. A perfect hologram will appear
embodied but trying to touch it will destroy the illusion."
Microsoft HoloLens
Jansen, 2014 [34]

Tangible Hologram (TangHo) Setup
Timothy J. Curtin

Tangible Graph Hologram

Collaborative TangHo Interaction

Robotic Arm Prototype
 Lego Mindstorms-based
6DOF arm prototype
 motors can be locked or set to
float mode
- useful for bidirectional I/O
 replaceable sphere for different
types of non-visual feedback
 Challenges
 inverse kinematics (final system
will be body mounted)
- use robotics toolbox (MATLAB)
 limited HoloLens hand tracking
 3D printing of final arm
Timothy J. Curtin

TangHo Prototype
Timothy J. Curtin

TangHo Interactions
Timothy J. Curtin

Data Physicalisation Framework
 Inspired by research agenda of Jansen et al. [34]
 Understanding the design space
 how do users map data to physicalisations
 Understanding the perceptual effectiveness
 e.g. what are the just-noticeable differences that a physical
variable can convey?
 Framework for dynamic data physicalisation
 data processing with device-independent output
- TextDIO: text for static or dynamic reports
- PhysDIO: data points, data variables and physical variables
 text and visualisation drivers for representation on specific
devices
- e.g. tangible hologram prototype

Conclusions
 Tangible holograms for data physicalisation and
simulation of radical atoms
 physical augmentation of highly configurable virtual objects rather
than programming of physical material (programmable matter)
 mobile solution that is no longer bound to a specific place such as
a the TRANSFORM table
 modular solution that can be configured for specific domains
(e.g. different spheres for various physical variables)
 Data physicalisation framework for investigating new
opportunities in dynamic data physicalisation
 new data physicalisation design guidelines (as in InfoVis)
 new evaluation techniques
 …

References
 B. Signer, Fundamental Concepts for
Interactive Paper and Cross-Media Information
Spaces, ISBN 978-3-8370-2713-6, May 2008
 ArtVis Project
 http://www.beatsigner.com/flyers/ArtVis.pdf
 Tangible Holograms
 http://beatsigner.com/tangibleHolograms.html
 TRANSFORM
 https://www.youtube.com/watch?v=lCARHatJQJA

References ...
[1] B. Signer and M.C. Norrie, A Model and
Architecture for Open Cross-Media Annotation and
Link Services, Information Systems 36(3), Elsevier,
May 2011
 http://beatsigner.com/publications/signer_IS2011.pdf
[2] B. Signer, What is Wrong with Digital Documents?
A Conceptual Model for Structural Cross-Media
Content Composition and Reuse, Proceedings of the
29th International Conference on Conceptual Modeling
(ER 2010), Vancouver, Canada, November 2010
 http://beatsigner.com/publications/signer_ER2010.pdf

References ...
[3] B. Signer, Fundamental Concepts for Interactive
Paper and Cross-Media Information Spaces,
Dissertation ETH No. 16218, Zurich, Switzerland,
2005
 http://beatsigner.com/publications/signer2006.pdf
[4] B. Signer and M.C. Norrie, As We May Link: A General
Metamodel for Hypermedia Systems, Proceedings of
the 26th International Conference on Conceptual
Modeling (ER 2007), Auckland, New Zealand,
November 2007
 http://beatsigner.com/publications/signer_ER2010.pdf

References ...
[5] S. Trullemans and B. Signer, Towards a Con-
ceptual Framework and Metamodel for Context-Aware
Personal Cross-Media Information Management
Systems, Proceedings of ER 2014, 33rd International
Conference on Conceptual Modelling, Atlanta, USA,
October, 2014
 http://beatsigner.com/publications/trullemans_ER2014.pdf
[6] S. Trullemans, A. Vercruysse and B. Signer, DocTr:
A Unifying Framework for Tracking Physical Docu-
ments and Organisational Structures, Proceedings of
EICS 2016, Brussels, Belgium, June 2016
 http://beatsigner.com/publications/trullemans_EICS2016.pdf

References ...
[7] R. Roels and B. Signer, MindXpres: An Exten-
sible Content-driven Cross-Media Presentation
Platform, Proceedings of WISE 2014, 15th Interna-
tional Conference on Web Information System
Engineering, Thessaloniki, Greece, October, 2014
 http://beatsigner.com/publications/roels_WISE2014.pdf
[8] A. Sanctorum and B. Signer, Towards User-defined
Cross-Device Interaction, Proceedings of DUI 2016,
5th Workshop on Distributed User Interfaces, Lugano,
Switzerland, June 2016
 http://beatsigner.com/publications/sanctorum_DUI2016.pdf

References ...
[9] S. Trullemans and B. Signer, A Multi-layered
Context Modelling Approach for End Users, Expert
Users and Programmers, Proceedings of the Interna-
tional Workshop on Smart Ecosystems cReation by
Visual dEsign (SERVE 2016), Bari, Italy, June 2016
 http://beatsigner.com/publications/trullemans_SERVE2016.pdf
[10] R. Roels, Y. Baeten and B. Signer, An Interactive
Data Visualisation Approach for Next Generation
Presentation Tools: Towards Rich Presentation-based
Data Exploration and Storytelling, Proceedings of
CSEDU 2016, Rome, Italy, April, 2016
 http://beatsigner.com/publications/roels_CSEDU2016.pdf

References ...
[11] M.C. Norrie, B. Signer and N. Weibel, Interactive
Paper as a Reading Medium in Digital Libraries, Pro-
ceedings of ECDL 2008, 12th European Conference
on Research and Advanced Technology for Digital
Libraries, Aarhus, Denmark, September 2008
 http://beatsigner.com/publications/norrie_ECDL2008.pdf
[12] B. Signer, M.C. Norrie, P. Geissbuehler and
D. Heiniger, Telephone Interface for Avalanche War-
nings based on Information Server for Adaptable Con-
tent Delivery, Proceedings of Pervasive 2002, Zurich,
Switzerland, August 2002
 http://beatsigner.com/publications/signer_Pervasive2002.pdf

References ...
[13] S. Trullemans, A. Sanctorum and B. Signer,
PimVis: Exploring and Re-finding Documents in Cross-
Media Information Spaces, Proceedings of the Inter-
national Working Conference on Advanced Visual
Interfaces (AVI 2016), Bari, Italy, June 2016
 http://beatsigner.com/publications/trullemans_AVI2016.pdf
[14] B. Dumas, B. Moerman, S. Trullemans and B. Signer,
ArtVis: Combining Advanced Visualisation and Tan-
gible Interaction for the Exploration, Analysis and
Browsing of Digital Artwork Collections, Proceedings of
AVI 2014, Como, Italy, May 2014
 http://beatsigner.com/publications/dumas_AVI2014.pdf

References ...
[15] B. Dumas, T. Broché, L. Hoste and B. Signer,
ViDaX: An Interactive Semantic Data Visualisation and
Exploration Tool, Proceedings of AVI 2012, Internatio-
nal Working Conference on Advanced Visual Inter-
faces, Capri Island, Italy, May 2012
 http://beatsigner.com/publications/dumas_AVI2012.pdf
[16] Moira C. Norrie, Beat Signer and Nadir Weibel,
Print-n-Link: Weaving the Paper Web, Proceedings of
DocEng 2006, ACM Symposium on Document
Engineering, Amsterdam, The Netherlands,
October 2006
 http://beatsigner.com/publications/norrie_DocEng2006.pdf

References ...
[17] M. Grossniklaus, M.C. Norrie, B. Signer and
N. Weibel ,Putting Location-Based Services on the
Map, Proceedings of W2GIS 2006, Hong Kong, China,
December 2006
 http://beatsigner.com/publications/grossniklaus_W2GIS2006.pdf
[18] R. Roels, P. Mestereaga and B. Signer, An Interactive
Source Code Visualisation Plug-in for the MindXpres
Presentation Platform, Communications in Computer
and Information Science (CCIS), 583, 2016
 http://beatsigner.com/publications/roels_CCIS2016.pdf

References ...
[19] C. Scholiers, L. Hoste, B. Signer and W. De
Meuter, Midas: A Declarative Multi-Touch Interaction
Framework, Proceedings of TEI 2011, 5th Interna-
tional Conference on Tangible, Embedded and Em-
bodied Interaction, Funchal, Portugal, January 2006
 http://beatsigner.com/publications/scholliers_TEI2011.pdf
[20] L. Hoste, B. Dumas and B. Signer, Mudra: A Unified
Multimodal Interaction Framework, Proceedings of
ICMI 2011, 13th International Conference on Multi-
modal Interaction (ICMI 2011), Alicante, Spain,
November 2011
 http://beatsigner.com/publications/scholliers_TEI2011.pdf

References ...
[21] B. Signer, U. Kurmann and M.C. Norrie,
iGesture: A General Gesture Recognition Framework,
Proceedings of the 9th International Conference on
Document Analysis and Recognition (ICDAR 2007),
Curitiba, Brazil, September 2007
 http://beatsigner.com/publications/signer_ICDAR2007.pdf
[22] L. Hoste and B. Signer, SpeeG2: A Speech- and
Gesture-based Interface for Efficient Controller-free
Text Entry, Proceedings of ICMI 2013, 15th Inter-
national Conference on Multimodal Interaction,
Sydney, Australia, December 2013
 http://beatsigner.com/publications/hoste_ICMI2013.pdf

References ...
[23] B. Signer, M. Grossniklaus and M.C. Norrie,
Interactive Paper as a Mobile Client for a Multi-
Channel Web Information System, World Wide Web
Journal (WWW), Vol. 10, No. 4, Springer,
December 2007
 http://beatsigner.com/publications/signer_WWWJ2007.pdf
[24] B. Signer and M.C. Norrie, PaperPoint: A Paper-
based Presentation and Interactive Paper Prototyping
Tool, Proceedings of TEI 2007, First International
Conference on Tangible and Embedded Interaction,
Baton Rouge, USA, February 2007
 http://beatsigner.com/publications/signer_TEI2007.pdf

References ...
[25] A. Vogelsang and B. Signer, The Lost Cosmo-
naut: An Interactive Narrative Environment on Basis of
Digitally Enhanced Paper, Proceedings of the Inter-
national Conference on Virtual Storytelling 2005,
Strasbourg, France, December 2005
 http://beatsigner.com/publications/vogelsang_ICVS2005.pdf
[26] L. Hoste and B. Signer, Expressive Control of Indirect
Augmented Reality During Live Music Performances,
Proceedings of NIME 2013, 13th International Confe-
rence on New Interfaces for Musical Expression,
Daejeon, Korea Republic, May 2013
 http://beatsigner.com/publications/hoste_NIME2013.pdf

References ...
[27] N. Weibel, A. Ispas, B. Signer and M.C. Norrie,
PaperProof: A Paper-Digital Proof-Editing System,
Proceedings of CHI 2008, 26th International Confe-
rence on Human Factors in Computing Systems
(Interactivity Track), Florence, Italy, April 2008
 http://beatsigner.com/publications/weibel_CHI2008.pdf
[28] A. Ispas, H. Schuldt, M.C. Norrie and B. Signer,
Towards Query by Sketch, Michael Springmann,
Second DELOS Conference on Digital Libraries, Pisa,
Italy, December 2007
 http://beatsigner.com/publications/springmann_DELOS2007.pdf

References ...
[29] N. Weibel, B. Signer, M.C. Norrie, H. Hofstetter,
H.-C. Jetter and H. Reiterer, PaperSketch: A Paper-
Digital Collaborative Remote Sketching Tool, Proceed-
ings of IUI 2011, International Conference on Intelli-
gent User Interfaces, Palo Alto, USA, February 2011
 http://beatsigner.com/publications/weibel_IUI2011.pdf
[30] B. Signer and T.J. Curtin, Tangible Holograms:
Towards Mobile Physical Augmentation of Virtual
Objects, Technical Report WISE Lab, WISE-2017-01,
March 2017
 http://beatsigner.com/publications/signer_arXiv2017.pdf

References ...
[31] H. Ishii and B. Ullmer, Tangible Bits: Towards
Seamless Interfaces between People, Bits and Atoms,
Proceedings of CHI 1997, ACM Conference on Human
Factors in Computing Systems, Atlanta, USA,
March 1997
 http://dx.doi.org/10.1145/258549.258715
[32] H. Ishii, D. Lakatos, L. Bonanni and J.-B. Labrune,
Radical Atoms: Beyond Tangible Bits, Toward
Transformable Materials, interactions, 19(1),
January 2012
 http://dx.doi.org/10.1145/2065327.2065337

References ...
[33] Y. Jansen et al., Opportunities and Challenges
for Data Physicalizations, Proceedings of CHI 2015,
ACM Conference on Human Factors in Computing
Systems, Seoul, Korea, April 2015
 https://doi.org/10.1145/2702123.2702180
[34] Y. Jansen, Physical and Tangible Information
Visualization, 2014, Ph.D. Dissertation, Université
Paris-Sud
 https://tel.archives-ouvertes.fr/tel-00981521

Introducing Tangible Holograms for Data Physicalisation and Big Data Exploration

More Related Content

What's hot

Similar to Introducing Tangible Holograms for Data Physicalisation and Big Data Exploration

More from Beat Signer

Recently uploaded

Introducing Tangible Holograms for Data Physicalisation and Big Data Exploration