Embedded & Tangible Interaction Design


Published on

A one-hour lecture I prepared for fulfilment of a MSc-level Advanced Human Computer Interaction class

Published in: Design, Technology, Business

Embedded & Tangible Interaction Design

  1. 1. Embedded & Tangible Interaction<br />David Shaw<br />“We live in a complex world, filled with myriad objects, tools, toys, and people. Our lives are spent in diverse interaction with this environment. <br />Yet, for the most part, our computing takes place sitting in front of, and staring at, a single glowing screen attached to an array of buttons and a mouse.”<br />Wellner, Mackay and Gold (1993)<br />
  2. 2. Introduction<br />Talk is about Embedded & Tangible Interaction<br />What it is<br />Related fields<br />History<br />The necessities to facilitate the technology<br />Problems and challenges<br />
  3. 3. About<br />Tangible and Embedded interfaces allow us to move beyond being limited to mouse and keyboard input to interact with a computer<br />We are quickly moving towards a post-WIMP revolution <br />Novel interaction devices are becoming commonplace<br />Simultaneously, computers are increasing being embedded in everyday objects and environments <br />
  4. 4. What is Embedded Technology?<br />“Embedded means enclosed; these chips and software are not considered computers. They are unseen parts of everyday things.”<br />Malcolm McCullough, Digital Ground, 2004<br />
  5. 5. What is Tangible Interaction?<br />Tangible Interaction encompasses user interfaces and interaction that emphasize<br />Tangibility and materiality of the interface<br />Physical embodiment of data<br />Whole-body interaction<br />The embedding of the interface and the users’ interaction in real spaces and contexts.<br />Eva Hornecker<br />
  6. 6. What is Tangible Interaction?<br />Tangible computing covers<br />Distributing computation over many specialised and networked devices in the environment<br />Augmenting the everyday world computationally so that it is able to react to the user<br />Interaction by manipulating physical objects<br />
  7. 7. What is Tangible Interaction?<br />Tangible computing shares these characteristics<br />No single focus or interaction<br />No enforced sequentially or modal interaction<br />Interface objects make intentional use of affordances <br />
  8. 8. What is Tangible Interaction?<br />Classifications of Tangible User Interfaces (TUIs)<br />Interactive Surfaces<br />Tangible objects can be placed onto a surface and interpreted by the system<br />Constructive Assembly<br />Modular and connectable elements attached to each other<br />Token & Constraint<br />Token represents an item, can be moved<br />Constraints provide structure to limit positioning and give tactile guidance<br />
  9. 9. Related fields<br />Tangible and Embedded Interaction Design is an interdisciplinary field that draws influence from: <br />Ubiquitous Computing<br />The Internet of Things<br />Industrial Design<br />Actuation and Sensor based technology<br />Robotics and Mechanics<br />
  10. 10. Technology That Disappears<br />“We have been very good at putting computers into the environment, but we have been very bad at getting them out of the way.”<br />“The most profound technologies are those that disappear”<br />“They weave themselves into the fabric of everyday life until they are indistinguishable from it”<br />Weiser (1991)<br />The computer will “take on the appearance of the task; it can disappear behind a facade.”<br />Norman (1990)<br />
  11. 11. 3rd Phase of Computing<br />Mainframe > PC > Ubiquitous<br />Lähdemäki (2007)<br />
  12. 12. History of the Technology<br />Emerged alongside Ubiquitous Computing as a research field philosophically opposed to Virtual Reality (VR) <br />Approach to “retain the richness and situatedness of physical interaction” whilst simultaneously “embedding computing in existing environments”<br />“Humans are of and in the everyday world”<br />Shaer and Hornecker (2009)<br />
  13. 13. Example Projects<br />
  14. 14. Example projects<br />Marble Answering Machine (Bishop, 1992)<br />Phonecall represented by coloured marbles<br />Drop marble to play message or call back<br />Graphic from Shaer and Hornecker (2009)<br />
  15. 15. Example projects<br />Graspable User Interface (Fitzmaurice, Ishii, Buxton, 1995)<br />Uses wooden blocks as handles to manipulate digital objects, early form of multi-touch<br />Blocks are placed on monitors<br />
  16. 16. Example projects<br />Tangible Bits (Ishii and Ulmer, 1997)<br />The entire world as an interface<br />Connect data between physical artifacts and surfaces<br />Move from ‘graspable’ to ‘tangible’<br />Three key concepts;<br />Interactive surfaces<br />Coupling of bits with graspable physical objects<br />Ambient media for background awareness<br />Ishii identifies the abacus as the ultimate tangible interaction metaphor <br />
  17. 17. Example projects<br />LiveWire (Jeremijenko)<br />Piece of string dangling from the ceiling<br />Visualisation of network traffic<br />Pioneer project was influence for ambient display<br />
  18. 18. Example projects<br />Intelligent Physical Modeling Systems (Frazer)<br />Intelligent cubes that know the position of its surrounding neighbours<br />Siftables (Merrill & Kalanithi)<br />1.5” cubes that sense motion & each other<br />http://www.youtube.com/watch?v=ZgF2rRzTg8Q<br />
  19. 19. Example projects<br />URP (Underkoffler and Ishii. 1999)<br />A TUI for urban planning<br />Combines physical models with interactive simulation<br />Can project / model wind flow, sunlight simulation, building materials <br />Graspable tokens<br />Collaborative<br />
  20. 20. Example projects<br />Tern<br />Tern is a tangible programming language for education<br />Program actions for robots<br />Uses interlocking wooden block which represent actions<br />Shape of blocks creates a physical syntax<br />
  21. 21. Example projects<br />reacTable<br />Tangible music interface<br />Each token has has a function<br />Dynamically attract using proximity<br />“The foremost goal was to design an attractive, intuitive and non-intimidating musical instrument for multi-user electronic music performance.”<br />http://www.youtube.com/watch?v=Ni_x_74VKU0<br />
  22. 22. Surface Technology<br />Microsoft Surface (2007)<br />Multi-touch is arguably the most commercially successful application of horizontal surfaces<br />Implicit capability of table interfaces is to support physical items on them<br />The Surface adds digital information to everyday physical objects, allowing digital entities to coexist as fully digital non-physical form and as shared digital-physical form<br />
  23. 23. Arduino<br />Open source physical computing platform<br />Simple I/O board that can be used as a stand-alone device or connecting to software on a computer<br />Add-on modules, shields, that provide additional functionality <br />
  24. 24. Benefits of Tangible UI<br />Tangible User Interfaces (TUI) have many benefits<br />Facilitating the kinds of collaborative activities that are not possible or poorly supported by single user technologies<br />Appropriate for those who have lost their sight or have difficulty with motor control<br />Andrew Cyrus Smith, Interactions 09/10 – 2010<br />Enhance learning - physical learning environments engage all sense and thereby support the child development. Lego Mindstorms and Topobo<br />Support ambient awareness<br />Can use tags to trigger digital information<br />
  25. 25. Necessities to facilitate this technology <br />McCullough (2004) suggests 10 essential building blocks to computing beyond the desktop<br />
  26. 26. 1. Sites and devices are embedded with microprocessors<br />“Less than a quarter of the chips produced by Intel, the largest manufacturer, are put into desktop or laptop computer motherboards”<br />“The rest are embedded into things you carry about, drive, or wear; or are embedded into physical locations.”<br />More than 95% of devices containing microchips do not present themselves to their users as computers.<br />
  27. 27. 1. Sites and devices are embedded with microprocessors<br />Practical economies of engineering do not always warrant providing a full service network operating system; devices can communicate at lower levels without that kind overhead.<br />With connectivity, embedded systems can communicate their status and receive ongoing instruction to and from their surroundings.<br />
  28. 28. 2. Sensors detect action<br />“If technologies are to keep out of the way, they need to see us coming.”<br />“If computationally embedded environments are to be useful yet unobtrusive, they have to recognise what is happening in them.”<br />Examples of sensors<br />Accelerometer<br />Tilt sensor<br />Pressure sensor<br />Light sensor<br />Microphone<br />
  29. 29. 2. Sensors detect action<br />Sensors have become the ‘key enabling technology’ for computing<br />A sensor responds to a change in state<br />Continuous sensor field<br />Wirelessly interlinked sensors<br />Passing or ‘hopping’ message directly amongst themselves<br />Compare to a typical setup – <br />LAN -> Dedicated Network -> Hardwired<br />
  30. 30. 3. Communication links form ad hoc networks of devices<br />Pervasive computing depends on unplanned communication<br />Not all linked objects will benefit from a full-featured web browser. More will run slimmer set of communications<br />Decentralised networking<br />
  31. 31. 4. Tags identify actors<br />Contextual awareness begins from an ability to recognise who or what is present<br />Recognition is easy with the use of tags<br />Smart badges<br />RFID Tags<br />Proximity detection<br />Passive, Active, or Battery Assisted Passive (BAP)<br />
  32. 32. 5. Actuators close the loop<br />A device to automatically control a system via motion<br />Open / close windows & doors, turn lights on / off, produce sound, motion or haptic feedback<br />Bridges and dams can detect and identify deterioration, and signal for upkeep before failure occurs <br />
  33. 33. 6. Controls make it participatory<br />Smart systems need to be operable where it is appropriate<br />This means providing an override facility <br />
  34. 34. An example of a smart system<br />Outdoor wind sensor detects wind speed and direction<br />Indoor temperature sensor monitors room temperature<br />Building Management System calculates that it can save energy by shutting off air conditioning system and opening windows<br />Actuators physically open windows to allow air to flow into the building<br />Staff may override system and close windows or turn AC system on, if they wish <br />
  35. 35. 7. Display spreads out<br />Before Gutenberg, text was reproduced using woodblock printing technique<br />The Gutenberg press revolutionised the type industry and his printing methods spread rapidly across the world<br />Today, the world thinks nothing of text. It is practically everywhere we look. On every conceivable surface<br />Embedded interaction will do the same for computing <br />
  36. 36. 8. Fixed locations track mobile positions<br />“Let’s put GPS in necklaces and dog collars. Everything that moves should have GPS.”<br />KanwarChadha, CEO at SiRF<br />“This kind of stuff has enormous potential for abuse by the authorities, or by anyone who can break into the information.”<br />Emily Whitfield, spokesperson for the <br />American Civil Liberties Union<br />Practical applications – GPS, Google Maps, Augmented Reality, Social Networks <br />
  37. 37. Proximity<br />“When you walk up to your computer, does the screensaver stop and the working windows reveal themselves?”<br />Bill Buxton<br />Important for context-aware properties of embedded interaction<br />
  38. 38. Proximity<br />There are four proxemic zones (Hall, 1966)<br />Intimate (< 1.5 feet)<br />Personal (1.5 to 4 feet)<br />Social (4 to 12 feet)<br />Public (12 to 25 feet)<br />Each have expectations of engagement and behaviour<br />
  39. 39. 9. Software models situations<br />System may begin to model a physically proximate area by polling local ad hoc links between known tags and devices<br />As hardware becomes less expensive, more diverse, and more plentiful, software becomes more challenging<br />“Who is here, and what are they doing?”<br />
  40. 40. 10. Tuning overcomes rigidity<br />Much of the place-centred character of situated interaction design comes from the fact that any fixed collection of devices has to be integrated<br />Question arise pertaining to protocols, distributed object programming systems<br />The challenge of embedded interaction design is how can we make these interactions meaningful.<br />
  41. 41. Problems and challenges<br /><ul><li>Privacy
  42. 42. Security
  43. 43. What if it breaks?
  44. 44. Social acceptance
  45. 45. Information overload</li></li></ul><li>Privacy<br />“Hundreds of computers in every room, all capable of sensing people near them and linked by high-speed networks, have the potential to make totalitarianism up to now seem like sheerest anarchy.”<br />Weiser<br />“Shifts in technology require us to rethink our attitude towards privacy, as suddenly our abilities to see, hear, detect, record, find and manipulate others and their lives is greatly enhanced.”<br />Langheinrich<br />
  46. 46. How Important is Privacy?<br />Sorry, Slide removed for privacy issues<br />Ironic, I know…<br />
  47. 47. How Important is Privacy?<br />Sorry, Slide removed for privacy issues<br />Ironic, I know…<br />
  48. 48. Security<br />Privacy and Security are two different concepts<br />Implementation of security does not ensure privacy<br />Data collection and processing are core components of ubiquitous computing, and therefore embedded interactions<br />
  49. 49. Privacy & Security Scenario<br />Intelligent fridge scenario<br />Knows what products you regularly buy and sources offers and coupons<br />Re-orders food when your stock levels are low<br />What if it gets hacked?<br />‘Hacker’can capture usage data<br />Can infer information<br />
  50. 50. What if it breaks?<br />What if <br />
  51. 51. What if it breaks?<br />Critical systems<br />Health systems<br />Flight systems<br />
  52. 52. Social Acceptance<br /><ul><li>Will the general public embrace an embedded / ubicomp future?
  53. 53. If a system is invisible, a user should still know it is there</li></li></ul><li>Information Overload<br />“Ubiquitous systems must not introduce undue complications into ordinary operations”<br />“You should be able to open a window, place a book upon a shelf, or boil a kettle of water without being asked if you ‘really want to do so.’”<br />Greenfield<br />More technology, more problems<br />Passwords for fridge doors?<br />Pay to use doors?<br />
  54. 54. References<br />Buxton, W. (1997) Living in augmented reality: Ubiquitous media and reactive environments. Video Mediated Communication. K. Finn, A. Sellen, and S. Wilber (eds.). Erlbaum, Hillsdale, N.J, 1997. <br />Greenfield, A. (2006). Everywhere: The Dawning Age of Ubiquitous Computing, New Riders, Berkeley, CA, USA<br />Hall, E.T. (1966) The Hidden Dimension. Doubleday, New York,1966.<br />Ishii, H. & Ullmer, B. (1997). Tangible Bits: Towards Seamless Interfaces between People, Bits and Atoms. Proc. CHI 1997, ACM Press (1997), p. 234-241.<br />McCullough (2004). Digital Ground, MIT Press, London, England<br />Norman, D. (1990). The Design of Everyday Things. Doubleday/Currency, New York.<br />Shaer, O., and Hornecker, E., (2009) “Tangible User Interfaces: Past, Present, and Future Directions” Foundations and Trends in Human-Computer Interaction, Vol. 3 Nos 1-2<br />Underkoffler, J. and Ishii, H. (1999) , “Urp: A luminous-tangible workbench for urban planning and design,” in Proceedings of CHI ’99, pp. 386–393, NY: ACM,1999.<br />Vogel, D. and Balakrishnan, R. (2004) Interactive public ambient displays: transitioning from implicit to explicit, public to personal, interaction with multiple users. Proc. of the 17th Annual ACM Symposium on User Interface Software and Technology. (Santa Fe, NM, Oct. 24-27). ACM, New York, 2004,137-146. <br />Weiser, M. (1991). The Computer for the 21st Century. Scientific American. Sept, 94-104.<br />Wellner, P., Mackay, W., and Gold, R., (1993) “Computer-augmented environments.Back to the real world,” Communications of the ACM, vol. 36, no. 7, pp. 24–26,1993.<br />