Lecture for the Cooperation Technology course at IDI-NTNU

1. Physical and embodied
user interfaces
Simone Mora
simonem@idi.ntnu.no
IDI, Sept. 17th, 2012

2. Agenda
• Definition of physical user interface
• Brief History
• Frameworks
• Research Showcase
• Conclusions

3. What is a physical user interface?
“An interface that rely on embodied interaction,
tangible manipulation, physical representation of
data and embeddedness in real space”
(Hornecker&Buur, 2006)

4. What is a physical user interface?
“An interface containing at least one interaction
technique not dependent on classical 2D widgets
such as menu and icons”
(Van Dam, 1997)

5. History
• The emergence of physical interfaces is closely
related to the adoption of ubiquitous computing
technologies
• Instead of creating “Virtual Words” or building on
the WIMP (Windows, Icons, Mouses, Pointers)
paradigm, computation should enrich the real
world with information from the digital domain

6. History
Marble Answering Machine, Durrell Bishop, 1992, RCA
Video: https://vimeo.com/19930744

7. Frameworks

8. Frameworks
• GUIs are really flexible since they can represent a
variety of information by decoupling representation
from control
• The key idea of physical computing is to give physical
form to digital information. However that makes the
tangible part more specific and less flexible.
• So, the challenge is: how can you mix both?

9. MCRit (Ishii et al.)
MVC Model MCRit Model
(Graphical UIs) (Tangible UIs)
Ullmer, B., Ishii, H. & Jacob, R.J.K., 2005. Token+constraint systems for tangible interaction with digital information.
Transactions on Computer-Human Interaction (TOCHI).

10. MCRit (Ishii et al.)
GUI
CityScape sw. TUI
Project Urp

11. Reality Based Interaction (Jacob et al.)
• Descriptive framework
• Many new interaction styles are designed to take
advantage of users’ skills and experience of
interacting with the real non-digital world
• Interaction with information becomes from a
segregated activity that takes places at a desk to
a fluid free form activity that takes place into the
real world

12. Reality Based Interaction (Jacob et al.)
• Jacob et al. identified four themes of interaction
with the real world that can be leveraged by
physical interfaces:
Jacob, R. et al., 2008. Reality-based interaction: a framework for post-WIMP interfaces. Proceeding of the twenty-sixth annual SIGCHI conference on
Human factors in computing systems, pp.201–210.

13. Research showcase

14. Application Domain
Technology

15. Main Themes (technology)
• Tangibles (graspables)
• Tabletops & Interactive surfaces
• Augmented Reality
• Whole-body interaction
• Ambient interfaces
Illustrations adapted from: Shaer, O. 2009. Tangible User Interfaces: Past, Present, and Future Directions, 2009 and http://dub.washington.edu/pubs

16. Application Domains
• Learning and Training*
• Social Communication*
• Problem Solving and Planning*
• Tangible Programming*
• Entertainment, Play, and Edutainment*
• Information Visualization*
• Music and Performance*
* Collaborative

17. Project: Sifteo Cubes
Category: Tangibles
Domain: Learning, Play
Authors: Sifteo Inc., 2009 (former MIT Media Lab project)

18. Project: Chronotape
Category: Tangibles
Domain: Social Interaction
Authors: University of Bristol
Key Publication: Bennett et al. ChronoTape: tangible timelines for family history. In TEI '12: Proceedings of the Sixth
International Conference on Tangible, Embedded and Embodied Interaction.

19. Project: Lambent shopping trolley
Category: Tangibles
Domain: Information Visualization
Authors: Open University, 2012
Key Pubblication: Kalnikaite, V. et al., 2011. How to nudge in Situ: designing lambent devices to deliver salient
information in supermarkets. In UbiComp '11: Proceedings of the 13th international conference on Ubiquitous
computing.

20. Project: Reactable
Category: Tabletops and Interactive surfaces
Domain: Music and Performance
Authors: Reactable Inc - 2006

21. Project: URB
Category: Tabletops and Interactive surfaces
Domain: Problem Solving and Planning
Authors: MIT Media Lab, 1999
Key publication: Underkoffler, J. & Ishii, H., 1999. Urp: a luminous-tangible workbench for urban planning and design.
In CHI '99: Proceedings of the SIGCHI conference on Human factors in computing systems.

22. Project: The Sixth Sense
Category: Whole-Body Interaction
Domain: Social Communication, Information Visualization
Authors: MIT Media Lab, 2009
Key Publication: P. Mistry, P. Maes. SixthSense – A Wearable Gestural Interface. In the Proceedings of SIGGRAPH Asia

23. Project: WATCHiT
Category: Whole-body Interaction
Domain: Training, Problem Solving
Authors: IDI-NTNU, 2012
Key publication: Daniel Cernea et al., Tangible and Wearable User Interfaces for Supporting Collaboration among
Emergency Workers. In proc. to the 18th CRIWG Conference on collaboration and technology.

24. Project: CroMAR
Category: Augmented Reality
Domain: Training
Authors: IDI-NTNU, 2011
Key Publication: Mora, S., Boron, A. & Divitini, M.
CroMAR: Mobile Augmented Reality for Supporting
Reflection on Crowd Management, IGI Global.

25. Project: Human PacMan
Category: Augmented Reality
Domain: Play, Enternatinment
Authors: Mixed Reality Lab Singapore, 2004
Key publication: Adrian David Cheok et al.; Human pacman: a wide area socio-physical interactive
entertainment system in mixed reality. In CHI '04 extended abstracts on Human factors in computing systems

26. Video:
http://www.mixedrealitylab.org/media/all
%20videos/HP/human_pacman.mpg

27. ambientROOM, MIT Media Lab, 1997
Ishii and Ullmer. Tangible bits: towards
seamless interfaces between people, bits
and atoms.
Project: AmbientROOM
Category: Ambient Interfaces
Domain: Information Visualization
Authors: MIT Media Lab, 1998
Key publication: Ishii, H. et al., 1998. ambientROOM: integrating ambient media with architectural space. In CHI 98
conference summary on Human factors in computing systems. New York, NY, USA: ACM, pp. 173–174.

28. Video: https://vimeo.com/48815734

29. Project: Collaborative Mood Awareness
Category: Ambient Interfaces
Domain: Information Visualization
Authors: IDI-NTNU, 2011
Key publication: Mora, S. et al.. Supporting Mood Awareness in Collaborative Settings. Proceedings of the 7th
International Conference on Collaborative Computing (CollaborateCom)

30. Physical Interfaces - Pros
• May increase usability
• May increase enjoyment
• Allows for (user-side) multitasking
• Allows for interaction in the periphery
• Support face-to-face social interaction
(Hornecker&Buur, 2006)
• Can leverage users’ knowledge and skills of
interaction with the everyday, non-digital, work
(Jacob et al., 2008)

31. Physical Interfaces - Pros
• (Almost) unexplored area of research, you can
be a pioneer!

32. Physical Interfaces - Challenges
• Requires not only designing the digital (software)
but also the physical (hardware) and the
interrelations between the two (affordances)
• Physical interaction is socially and culturally
situated (Hornecker, 2011)

33. Further readings

34. Thanks!
Simone Mora
simonem@idi.ntnu.no