Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Direct manipulation is broken: O'Reilly Design Conference Jan 2016


Published on

Why connected products/the internet of things asks consumers to think like programmers, and the UX challenges this creates. With acknowledgement to Alan Blackwell of Cambridge University

Published in: Design
  • Be the first to comment

Direct manipulation is broken: O'Reilly Design Conference Jan 2016

  1. 1. Direct manipulation is broken Why the IoT asks consumers to think like programmers, and the UX challenges this creates Claire Rowland / @clurr O’Reilly Design Conference 21st January 2016
  2. 2. “This is more than a UX book; it covers all of the critical design and technology issues around making great connected products.” DAVID ROSE, ENCHANTED OBJECTS
 “As a grizzled veteran of several campaigns within the matter-battle of the Internet ofThings, I was pleasantly surprised to find the number of times this book made me pause, think, and rethink my own work (and that of others).A very valuable addition to the canon of design thinking in this emerging area.” MATT JONES, GOOGLE
 “Connected products are a unique beast. Designing this type of product well takes lots of iterations & user testing.This book takes you through the paces.We keep one on the shelf at Highway1 for our IoT companies.”
  3. 3. “The IoT breaks direct manipulation. Using connected devices is conceptually more like programming.”
 Alan Blackwell Professor of Interdisciplinary Design, Computer Laboratory, University of Cambridge Very roughly paraphrased
  4. 4. “The IoT breaks direct manipulation. Using connected devices is conceptually more like programming.”
  5. 5. Direct manipulation “A human–computer interaction style which involves continuous representation of objects of interest and rapid, reversible, and incremental actions and feedback.” Ben Shneiderman “The future of interactive systems and the emergence of direct manipulation". Behaviour & Information Technology
  6. 6. Why direct manipulation works - Visibility. By looking, the user can tell the state of the device/system and the alternatives for action. - A good conceptual model. The designer provides a good conceptual model for the user, with consistency in the presentation of operations and results and a coherent, consistent system image. - Good mappings. It is possible to determine the relationships between actions and results, between controls and their effects, and between the system state and what is visible. - Feedback. The user receives full and continuous feedback about the results of actions. 
 Chapter 2,The Design of Everyday Things
  7. 7. A key condition of direct manipulation is that the user’s actions and the results happen 
 at the same time, in the same situation
  8. 8. Remote control Image: British Gas
  9. 9. Automation Scheduling things to happen in future, like turning on lights to welcome you home, or turning off unnecessary appliances to save energy overnight. Image:; Martin Cathrae via Flickr
  10. 10. Interoperable, coordinating products Video: shot atTarget Open House Things in your world will work together to meet and maybe even anticipate your needs. A sleepsuit that detects your baby waking up could trigger a morning routine. An “intelligent” security system could draw on information from your calendar, to detect possible intrusions.
  11. 11. All these scenarios break direct manipulation because
 they displace user actions from system responses
  12. 12. Remote control: displacement in space You might: - Set the security alarm when there’s still someone at home - Turn on lights when they’re trying to sleep - Turn the heating off when someone needs it - …or turn on an electrical appliance when there’s a gas leak in the home Image:Audetat via WikiCommons
  13. 13. Risks of displacement in space - The action may be inappropriate or unsafe at that time (or just annoying to someone else) - Users may not see the consequences of their actions - Users may not realise that the action they thought they took didn’t work as planned Images:
  14. 14. Is that 7pm where I am now, or at home?
  15. 15. Automation: displacement in time - That “unnecessary” appliance that gets turned off might be the tumble dryer running overnight to ensure a shirt is ready for an important meeting. - Your alarm goes off because your calendar says you’re not ‘supposed’ to be home and infers you might be an intruder. - Your baby wakes up super early and you don’t want to start the day, you want to GO BACKTO SLEEP Google Image search for baby waking early
  16. 16. Risks of displacement in time - Users need to anticipate their own future needs: which humans are not very good at. - Users may forget that they have set up an action, and then be negatively affected by the execution of that action. - They need to anticipate all the changing conditions to which the system might have to respond. interrupted-by-sprinklers-as-fans-criticise-no-beer-lack-of-wi-fi-and-poor-10478187.html
  17. 17. Flexible, multi-purpose hardware Automated security lighting schedule? Monitoring energy consumption of an appliance? Remote control of your kid’s Xbox? Image: Belkin Functionality can be modified in a remote software layer
  18. 18. Smart plug that ran an automated schedule for a lamp is moved to hair straighteners… and is now turning those on and off when you’re not at home Image:; London Fire Brigade Flexible, multi-purpose hardware: displacement in function
  19. 19. Risk of displacement of function - The behaviours of the device that were appropriate for its original use are ported over to a new use, and are now actively dangerous Image: Shropshire Insurance
  20. 20. But hey, that’s why we need to make things smarter, isn’t it?
  21. 21. - You could have a gas sensor that alerts you and turns off the electricity if it detects a leak. - You could come up with a way of temporarily excluding the tumble dryer from your energy saving program. - You could program the smart plug to identify changes in energy consumption that might indicate that it’s being used with a different device, and disable the schedule. - You could disable the alarm with two factor voice authentication: not just keycode or phone, but voice. - You could have that baby’s room macro run only after 6am. Image:;
  22. 22. Indeed you could. But it’s very hard to mitigate every possible problem. Number of edge cases you’d have to anticipate Few Tons Discrete product Many interconnecting products
  23. 23. “…the cumulative complexity of a bunch of simple things—regardless of how delightful, simple and desirable they may be—will soon exceed the ability of humans to cope.” Bill Buxton Principal researcher, Microsoft Research important-design-jobs-of-the-future
  24. 24. In Norman’s terms, the design upshot can be: - Poor visibility. It’s not always possible to tell, by looking, the state of the device or the actions that are possible with it. - A complex conceptual model.The conceptual model of a connected product is way more complex. Need to understand parts, what each does and how they interrelate. Separation of operations and results can make it harder to identify consistencies. - Obscured mappings. It is harder to determine the relationships between actions and results, between controls and their effects, and between the system state and what is visible. - Poor feedback.The user may get only partial, delayed, incorrect or intermittent feedback about the results of actions.
  25. 25. “The IoT breaks direct manipulation. Using connected devices is conceptually more like programming.”
  26. 26. Since when did turning lights on and off count as programming? Images Keil C51 C Compiler (Phaedrus Systems), Philips
  27. 27. Let’s break down a home automation task…
  28. 28. I want a nice green lawn Understand and frame your own problem. = Requirements definition Image:Adam Kerfoot Robverts via Flickr
  29. 29. “I need a controller, and some sprinklers, and I need to plan around weather, water use restrictions, and social events”. = Specification Image: Blossom
  30. 30. “Now… how do I get this thing to do what I want?” Image: Rachio Iro = Implementation
  31. 31. “Oops.What went wrong?” =Testing and debugging Image:
  32. 32. This is software engineering! Users must: •Anticipate their own future needs and plan a solution •Configure the system to meet these needs using an abstract representation •Deal with unforeseen conditions that may make the system behave in unexpected ways •All the while… considering future conditions that might cause unexpected effects or conflicts Requirements definition Specification Implementation Testing/debugging
  33. 33. You say that like it’s dirty.
  34. 34. Empowering people to program is great
  35. 35. But it’s a lot of work for a consumer product Product Tool In areas where they don’t have expert knowledge or are short on time, consumers tend to look for products which come preconfigured to solve a specific problem, rather than multi-purpose tools which require the user to define and solve their own problem.
  36. 36. It’s a time and attention issue Image: XKCD
  37. 37. OK. But that still doesn’t explain how using a simple remote control is like programming
  38. 38. Users need to engage with the system model Internet Local WiFi
  39. 39. Programming can be hard, and painful, however good at it you are
  40. 40. Experts *expect* to make mistakes (and are often paid to do so) I expected it wouldn’t be perfect the first time around and there is always an exception.You learn about them after you start using them. Expert home automation user (and IT security professional)
  41. 41. Image: Zach Luz ( …users don’t (and are never paid to do so) - Things not working isn’t part of the normal path to success for a consumer product… but this will happen - It’s especially painful at home: which is supposed to be our haven from the world - System designers need to do all they can to mitigate potential damage
  42. 42. The logic of even apparently quite simple things can be complex… and inconsistent if (time == 1800) then set light =on if (time == 2200) then set light =off while (mode ==scheduled) then { while (time >=1800 && time <=2159) then set target temperature =72F while (time >=2200 && time >=0659) then set target temperature =65F while (time >=0700 && time >=0859) then set target temperature =72F while (time >=0900 && time >=1759) then set target temperature =58F }
  43. 43. What do we do about it? Or, how can we make controlling complex systems easier?
  44. 44. Home Maestro default.aspx?id=70594 This is not (just) about new types of programming interface They might help with the implementation part of the problem. But they don’t help with the problems of predicting your needs, planning and debugging, forming a conceptual model or knowing what’s happening at a remote location.
  45. 45. Images: Lowes Better feedback: was my command executed or not?
  46. 46. Better visibility/feedback: remote context - Am I about to do something that might be unwise? - Has something happened that might not be what I want? Carrier 12:00 PM !! Jen is at home !! ARM SYSTEM Home security alarm
  47. 47. Better visibility: why did that thing happen/not happen? - “Why didn’t the dishwasher run this morning?” (perhaps the smart electricity grid turned off the supply to that nonessential device at a time of peak demand) - “Why is the hall light flashing on and off?” (perhaps it is receiving conflicting instructions from security and energy saving apps)
  48. 48. Error prevention - Flag up potential consequences: sandbox simulations to check for rule conflicts - Try to prevent the user from doing dangerous things Your new lighting rule conflicts with your security rule I’m pretty sure that’s no longer a lamp I’m powering
  49. 49. What about autonomous systems? Scott Jenson: Home automation is an easyhard problem …common sense is hard to get right
  50. 50. What about us? Will we adapt to a programming-like world?
  51. 51. [Ben Evans] My grandfather could probably have told you how many electric motors he owned.There was one in the car, one in the fridge, one in his drill and so on. My father, when I was a child, might have struggled to list all the motors he owned (how many, exactly, are in a car?) but could have told you how many devices were in the house that had a chip in. Today, I have no idea how many devices I own with a chip, but I could tell you how many have a network connection.And I doubt my children will know that, in their turn. Benedict Evans Andreessen Horowitz
  52. 52. If you’ve enjoyed this, the book covers: - How the technology of IoT affects UX - Product and design strategy for connected devices - Industrial design - Interface and interaction design for embedded devices - Cross-device interactions and interusability - Interoperability - Responsible IoT design - Designing with data - Prototyping and user research methods for connected products @clurr |