Biases and The Law of Unintended Consequences

Editor's Notes

1. Portland Japanese Gardens in Portland, Oregon.
2. I’m delighted to be here speaking at the Cambridge Usability Group UX unconference. Our best wishes are with Jane… One theme of today is about having an impact as a designer. That sounds a bit like making sure that users have an experience. I mean users are going to have an experience right? It might be good or bad, but there will be one! When it comes to having an impact as a designer – you want that impact to be a good one. So you make decisions that you hope will improve rather than degrade an end-users experience. All your decisions are going to have some impact. To make good decisions, you need to judge outcomes. And to help assess likely outcomes, it helps to understand the way people behave and react, given what they know. The thing is, people often don’t know themselves why they do things, and frequently can’t put it into words. So understanding ‘what users do’ can get a little tricky. This talk explores why people often appear to act rather irrational, and how that might be biased by their environment or by what we do as designers. And how our decisions ultimately impact users in unexpected ways.
3. There’s a popular article about biases by Buster Benson, who categorised biases into four areas: Those where there’s… Too much information in the world - In which case we use things that change or stand out in order to prioritise. Too much confusion or ambiguity in the world - So we fill in gaps with substitutions using what we know or think we know. We’re constrained by time and information - So we prefer simpler, low risk, reversible solutions with short term gratification. Err. The last one? We can’t remember everything - Which means we cluster information with generalizations, stereotypes, key moments.
4. I’ve found this breakdown really helpful. Being aware of these human behaviours, and understanding why people are the way they are, can help me... 1) consider who the audience is 2) think about ways to approach a particular problem, 3) consider what language or terminology to use 4) think about ways to describe or reframe a problem 5) be aware of my own biases affecting what I do.
5. I started planning this talk with a little seed - just 7 letters - WYSIATI. No - it’s not some religious sub-cult of the illuminati. It stands for “what you see is all there is”. What you see in your mind is a combination of what you’re given and what you think you know at that instant in time.
6. This acronym comes from Daniel Kahneman - a nobel prize winner in Economic Sciences, who wrote a book called Thinking Fast and Slow
7. Kahneman’s theory is that when the mind makes decisions, it deals primarily with Known Knowns, phenomena it has already observed. It rarely considers Known Unknowns, phenomena that it knows to be relevant but about which it has no information. And the mind appears oblivious to the possibility of Unknown Unknowns - unknown phenomena of unknown relevance.
8. Kahneman explains that humans fail to take into account complexity, and that our understanding of the world consists of a small and un-representative set of observations. And the mind generally doesn’t account for the role of chance, falsely assuming that a future event will mirror a past event.
9. Kahneman uses the concept of What You See Is All There Is to explain overconfidence. The overconfidence effect is one of the many well-established biases in which a person's confidence in judgements is reliably greater than the objective accuracy of those judgements.  Especially when confidence is relatively high.
10. Overconfidence is a natural human bias – and actually it covers a collection of biases in human behaviour. overestimation of one's actual performance I imagine many people here are probably familiar with inaccurate project estimates? - overplacement of one's performance relative to others For example: 93% of American drivers rate themselves as better than the median! - overprecision - expressing unwarranted certainty in the accuracy of one's beliefs. 
11. If human confidence had perfect calibration... judgments with 100% confidence would be correct 100% of the time, judgments with 90% confidence correct 90% of the time, and so on.
12. But what research shows is that confidence systematically exceeds accuracy ...so long as you’re answering hard questions about an unfamiliar topic.
13. Overconfidence has been called the most ‘pervasive and potentially catastrophic’ of all the cognitive biases to which human beings fall victim. [Read examples?] It sounds like ‘A Bad Thing’ doesn’t it?
14. Still on the plus side, another bias which this covers is a more helpful one... Over-optimism - which causes a person to believe that they are at a lesser risk of experiencing a negative event compared to others. So I know at least that any detrimental outcomes from overconfidence will be more likely to happen to you lot than to me. Yes, guess what - I’m biased!
15. People are biased. But not everyone is biased in the same way! Different people are biased by different things, - because they have different lives, - different experiences, - different cultures. It’s diverse. It’s the real world. And it’s messy - as I will get to later.
16. Everything might be messy but, everything is connected. (any Dirk Gently fans here?) As product designers, a lot of what we do and what we say has an impact on what other people see. And because What You See Is All There Is, that means that what we do also indirectly affects what other people do. Or don’t do. And so on. - there’s a chain of events.
17. That hierarchy of connectedness is a problem, because it’s unintentional and non-obvious... It makes it really hard to predict how your actions will propagate forward to the future. And it makes it really hard to work backwards into the past as to why something happened.
18. One simple way to examine cause and effect is the 5 whys method. In the 5 whys method the goal is to determine the root cause of a defect or problem by keep asking "Why?". Each answer forms the basis of the next Why question.
19. But not all problems have a single root cause. To uncover multiple root causes, the ‘5 whys’ method should be repeated asking a different sequence of questions each time. Preferably with varying people asking the questions (because hey guess what – biases!) In that way you can uncover a multitude of issues which can typically be clustered into common categories, like product, people, process, place, price, performance.
20. You can represent the results as a kind of tree using an Ishikawa diagram. Ishikawa because that’s who created it, but it’s also known as a fishbone diagram. Sometimes it gets called Fishikawa diagram! This is a healthcare example around what’s called an ‘unplanned extubation’. Hospital patients of any age may have a tube inserted into the airways to help them breath. But for some reason the tube gets removed unintentionally, and this can be for a variety of reasons. This chart shows an Ishikawa diagram after a discovery exercise which uncovered a large number of causes. You can see how you could take some of these further. For example, why was the wrong type of tape used? Why was there a long time between re-taping?
21. That’s just one way to look back at root causes. Obviously it’s possible to use the outcomes as a driver towards preventing situations from happening again. But could we be better at stopping situations from happening in the first place? Wouldn’t it be nice to be better at predicting the future?
22. Hindsight is a wonderful thing. Using hindsight is way easier than making predictions.
23. At least - it’s way easier than making accurate predictions. Sooo, rather than trying to predict the future – try using hindsight earlier. By testing early. Testing whatever you have.
24. You don’t just have to test the products and services that you’re building.
25. You can test at any stage on the journey towards a product
26. You can test ideas & concepts; drawings & prototypes; tools & processes; language & terminology.
27. Even so - sometimes you just can’t predict how a product will work once it’s released into the real world - because now you’re at the mercy of different people in different environments who just might not use something the way you intended. You can’t be there watching over their shoulder, and helping them at every step, so it’s natural that some percentage of users will do something you hadn’t thought of. So you have to accept that people will misuse your product
28. Here’s another 7 letter acronym. Often the way things go wrong is that for any given user at any given time... WPSIWYI - What People See Isn’t What You Intended. Here’s a couple of real world examples.
29. Imagine a stick with a needle cleverly hidden in one end. When you poke someone with the stick the needle pops out and stabs them. What could go wrong?
30. This is an EpiPen. It’s used when someone is having a life threatening allergic reaction or anaphylaxis. This is actually an older version of the EpiPen.
31. I’m not going to tell you exactly how to use this one because they vary slightly and if you ever need to use one, ALWAYS READ THE INSTRUCTIONS. Generally though, you pull off a safety release, grab it in your fist, release it into the thigh, and hold for a specified number of seconds. This could save someone’s life. It’s a stressful situation. Now - what a surprisingly large number of people do, is pull off what they think is a safety cap over the needle. (Like the lid of a pen!) They grab hold of it like they do with a syringe - with their thumb over the other end - and then ram it against the patient’s thigh. Thus pushing the needle into their thumb.
32. Don’t do that! It bloody hurts and won’t help the patient one bit. People do this even in non-stressful situations - such as when demonstrating to others with an out-of-date pen instead of a needle-free trainer device.
33. Despite the fact that people do get training, sometimes it’s parents, carers, work colleagues, or family that have to do this. People are too flustered to remember instructions, or too panicked to follow them. Despite new versions of the EpiPen having big orange messaging and arrows saying ‘Needle End’ - it still happens way too often, and I believe almost a billion dollars went in to redesigning the EpiPen! In one study 36% of paediatricians used a trainer pen in a way that would have injected themselves if it was real.
34. Fortunately, after training this was reduced to 7%. 1 in 14 people, even after training, would still have stabbed themselves. How many people in this room is that?
35. Here’s another example. Imagine, you leave a robot at home with your pet all day. What could go wrong?
36. Enter - The Roomba Robot vacuum cleaners like Roomba move automatically around your home cleaning the floor while you’re out. Brilliant. I thought I couldn’t live without one. But I manage somehow.
37. Now some Roomba owners also have pets, (or pet owners also have Roombas), and the Roomba is great at picking up shed hair. That’s not hair from your shed. I mean hair that’s come from pets.
38. Problem is that animals don’t necessarily poop where they’re meant to. So you can imagine the scenario. And Roomba has wheels with a really good tread on them! I told you earlier - the real world it’s messy.
39. When this was reported, some people bizarrely said it happened to them like 5 or 10 times
40. Roomba responded with ‘Quite honestly, we see this a lot’. Is it a design error or a usage error. Note I didn’t say ‘user’ error. Was this foreseen? - who knows? But Roomba said they were looking at developing a ‘faeces recognition camera’. Now there’s a job!
41. The point here is that it’s hard to predict how different people behave, and how things work in the real world when you can’t be there. A lot of thought has to go into expecting the unexpected. Testing the spaces between the intended uses. A good test team should be doing that - not just testing what they see, but prodding all the spaces between and around what they see and what they know. If you create a product that can be misused then it will probably get misused - perhaps because people… - didn’t read instructions - forget how to use something in conjunction with something else - panic in the heat of the moment, - or simply use the default settings without tuning the product to their individual needs. - Or...any number of diverse reasons 
42. What you can’t do is easily change other people’s behaviour. So rather than try and change people, can we try and change their environment? I’ll cover two ways we often do that, and why they’re problematic. Firstly, 1) We create tools.
43. And people luuurve tools! Makers of tools often promote them on the basis that they save time, effort, and can save money in the long run. It’s certainly better than saying they pander to people’s lazy human behaviour. Incidentally, a quick side note here. By lazy here, then using the word lazy, is itself kind of lazy. What I mean is - people being put off due to situational constraints and even unseen barriers such as anxiety, confusion, and self doubt. I really recommend reading an article by Devon Price called ‘Laziness does not exist’.
44. So anyway, much of what we do is based around giving people the tools to do something themselves. For customers it’s efficient. It’s empowering. And It’s a good business model. We make you a better YOU, with our iSmartTool™
45. Here’s a couple of simple tools that pander to our natural biases: One click ordering at Amazon. Removes all the barriers. So easy a child could do it? Unfortunately mine do! Skip Intro on Netflix. People went mad with praise when Netflix introduced the button to skip intros. And auto-skip is brilliant when you’re binge-watching the latest Marvel series.
46. Tools empower people. They put you in control. Now I am the boss at Netflixing.
47. Secondly - we create patterns.
48. A pattern is a solution to a problem in a given context, and an approach that can be re-used over and over again in that context.
49. And people luuuurve patterns People home in on patterns and generalizations. Patterns are our subconscious way of simplifying information. People are good at spotting patterns - quite often even when none was intended. And people behave in patterns - or rather they form habits. And very easily at that!
50. Tools and patterns – they sound like good things, right? So, what’s wrong with them.
51. With patterns, thanks to the biases in human behaviour, people are prone to associating new information with existing patterns rather than creating new ones. You come across a problem and you just apply the nearest matching solution that worked last time.
52. More to the point, maybe you mis-apply an existing solution to the new problem. Or to a new audience. In cases like that, it might be that you just used an anti-pattern! An anti-pattern appears to be beneficial at first but then creates unintended consequences. An anti-pattern is also one that actually has a demonstrably better solution that could be used repeatedly. Maybe it’s something that you’ve got into the habit of using. Maybe it’s one of your go-to tools.
53. Here’s a couple of examples of anti-patterns - in interface design. Offering hints in the user interface to let users know how to use the product. Sounds helpful, but much better to change the interface to behave the way people expect it to.
54. Listing options (usually in alphabetical order) seems helpful. But has some unexpected side effects It increases the odds of something unintentionally being left selected by default. One company found that people were omitting country selection on a form, and consequently finding that a large proportion of their customers were apparently living in Afghanistan. Better to detect the local country and put that first, or even better, to avoid a list of several hundred items altogether and use auto-completion as people start typing the country
55. And don’t get me started on passwords. Or CAPTCHAs. Or F.A.Q.s. Or…
56. Anti-patterns come about because of our own biases towards opting for quick, simple, understood patterns to achieve an outcome, without really considering the real world use. And they often come about from blindly following trends! What looks like success or something that works elsewhere gets copied without enough consideration for its audience appropriateness. Anti-patterns aren’t just for designers - they’re everywhere: Software engineering, documentation, and organizations, they all have anti-patterns. And there’s behavioural anti-patterns. Such as the need for quick action, which often instils the mentality that any action is better than no action.
57. Now what’s wrong with tools? Firstly – People! People DON’T - like to take time to learn to use new tools which is a problem in a fast moving world of technology - People don’t usually have many tools for the same thing. - They don’t always use the right tool for the right job. Remember - we’re *lazy* and busy. It’s often whatever’s easiest. - People don’t always get to choose their own tools Sometimes tools are thrust upon us. Company policy, restrictions, security, compliance, and of course cost.
58. People DO - have favourite go-to tools and not so many of them at that - They do have a tendency to use the default settings. Because *WYSIATI* - And people do form habits around their tools Tools and patterns – they’re heavily intertwined.
59. Well this isn’t all exactly the fault of the people using tools! The real issue is that people are just too human! It’s safe to say we’re stuck with that! So, we need to design tools for people, not just tools for tasks.
60. Secondly - the tools themselves can be a problem. What you do, what you can do, the way that you do it, and what happens next, is biased by the tools you use.
61. Yes, tools empower the user and can make them feel like they have superpowers. They augment our intelligence and our capabilities. But tools are not neutral. They contain bias (hidden or otherwise). And they influence and drive trends.
62. It’s very easy to let the tool drive what you do. Using default settings and values, front-of-house features, wizards. Those defaults: who chooses them? Who are they chosen for? Tools are often built in a one-size-fits-all fashion. They solve one thing in a general purpose way. You inevitably have to stack your tools, and then connectivity and communication between tools becomes a factor - or a constraint.
63. When you combine people and tools, to do similar jobs, the choice of tools means that what you get may diverge into a variety of creative solutions. Or… what you get may converge towards a singularity. Divergence: When different people use different tools for the same job, you get different results. Convergence: But when different people use the same tool for similar jobs, you get similar results.
64. There used to be a handful of frameworks for building websites. As a developer you could tell within seconds of visiting a website which framework had been used to build it. Then there was an explosion of tools to create similar web content, and at the same time the world went minimal. Oh and flat. So we ended up with a sea of similar looking sites each trying to stand out - and failing because with the same tools and the same mindset, there are few opportunities to stand out from the crowd. These are websites of design agencies who do beautifully simple (and apparently epic) design. So beware the impact that will come from tools you use as a designer. We may shape our tools, but our tools also shape us and what we do. That affects the tools that we build for others. As the chain goes on, choices at the top of the heap trickle down to impact the masses.
65. This was a quote overheard from a former CTO of Autodesk. Autodesk make architectural and structural design software. Their CTO said “I can walk through most buildings and tell you which version of AutoCad was used to design it.” I find that such a sad interpretation of the state of architecture - that the skylines of the worlds greatest cities might be changing in line with new features that made the cut for the latest version of the software that may or may not have been installed on a designer’s PC.
66. So let’s try and wrap all this up. People are biased... - But it’s just human behaviour. Accept it, but be mindful of it. The real world is messy, complicated and unpredictable... - But it’s just entropy. Accept it, but be mindful of it. You can’t predict the future, but you can empathise with cause and effect. Learn and improve by keep asking why. Remembering: -    Nothing has one cause -    Nothing has one consequence
67. Test ideas, prototypes, processes and tools as well as the things you build from them. Tools are good, and don’t be afraid of patterns, BUT Be willing to challenge existing ideas
68. Question your tools in the best interests of the user.     Watch out for and record anti-patterns and consequences of habit forming.     Ask if what you’re doing is defined by your tools. Whatever you do in your work, our environments change, and what we know is changing all the time. Yet, it’s often easy to let tools, patterns and processes remain static. We need to constantly be on our guard and be asking whether the decisions we made, or the solutions we used in the past are still valid today. Changes don’t have to be big or risky or high impact. They can be incremental, low risk, reversible. Just how people like it!
69. And finally - Think like a gardener!    
70. Designing and building a product is often associated with architecture, and web design borrows a lot from that field. But… …I think it’s a poor metaphor for what we want to happen.
71. This is George R.R. Martin. He wrote a few fantasy stories which have become mildly popular. (You may have heard of Game of Thrones? ) I’d like to read a quote from him which I rather like:
72. “I think there are two types of writers, the architects and the gardeners. The architects plan everything ahead of time, like an architect building a house. They know how many rooms are going to be in the house, what kind of roof they're going to have, where the wires are going to run, what kind of plumbing there's going to be. They have the whole thing designed and blueprinted out before they even nail the first board up. The gardeners dig a hole, drop in a seed and water it. They kind of know what seed it is, they know if they planted a fantasy seed or mystery seed or whatever. But as the plant comes up and they water it, they don't know how many branches it's going to have, they find out as it grows. And I'm much more a gardener than an architect.”
73. In closing… As designers we’d like to maximise beneficial impact. We’d like to make good decisions not default decisions. Which often requires change. In order to be able to change, we need to be able to change. And to feel able to change. The less we know at the beginning of a project, the more we need to try and build in flexibility to change direction. A readiness to adapt to the newly incoming information in our work. Not just in what we’re trying to make, but the way that we’re going about it. I’m often frustrated by the glacial pace at which people and organisations seem able to adapt to change. We need our processes to be more modular and our tools to be more interchangeable. We need to be able to apply more critical thinking when deciding what are the right tools to use at what time. We need our tools to be less all-in-one, and yet we seem to always be moving towards tools or systems that do more. We live in a time where data is massively more valuable than the tools and services we actually use. But the fact is we need data transfer to be more open, our data to be more in our control, so as to enable more considered choices, rather than being tied in to support someone else’s business model.
74. And above all, we need to curate all this in a way that continually re-focuses on the outcomes for the end-users and their own changing environments. Not on what’s dictated to us, or what’s fashionable and trending. And not tied down by the same old tools, static processes, and questionable patterns.
75. So there we are. I started with a seed. I nurtured it like a gardner, and it grew into this talk. What You See Is All There Is.
76. If you are interested in knowing more about some of these topics, I can share a list of books, articles, and podcasts, some of which I mentioned earlier. Thank you.