Getting User Research Right
•Download as PPTX, PDF•
7 likes•861 views
Getting user research right is hard. It is easy to get it wrong. Here are some mistakes and patterns to avoid and a way to get results.
More Related Content
What's hot
What's hot (20)
Viewers also liked
Similar to Getting User Research Right
Similar to Getting User Research Right (20)
Recently uploaded
Recently uploaded (20)
Getting User Research Right
- 2. Dr. Edward F. Staiano Ph.D
- 3. ….wireless personal communications network designed to permit any type of telephone transmission—voice, data, fax, or paging—to reach its destination - anywhere on earth.
- 5. CASH!!!!
- 11. !!!
- 13. LOL Sounds like they should have done some customer research
- 14. 23,000 customers surveyed 42 countries
- 15. It is possible to do a load of user research and still get it wrong.
- 17. How not to blindly follow your users off a cliff
- 20. “we want better search!”
- 24. You took on Google?!?! BWAHAHAHAHAAH
- 25. “we want better search!”
- 26. “we want better search!” = “we want better results!”
- 29. Above all else, understand your users workflow.
- 34. To understand workflow, we need to be in the problem space.
- 35. 1. Find some users.
- 37. Sod that. Just do a survey.
- 39. Existing Customers Potential Customers Ex Customers
- 40. 1. Find some users. 1. “tell me about a time when”
- 41. Listen.
- 42. Sounds like too much work. Just ask them what features they want.
- 44. 1. Find some users. 1. “tell me about a time when” 1. Let them lead you to the problems they have
- 45. “if you had a magic wand, what would you change about the process you just talked me through?”
- 46. 1. Find some users. 1. “tell me about a time when” 1. Let them lead you to the problems they have
- 48. outcomes
- 49. pain points
- 50. cadence
- 51. workflow
- 53. Write up call notes and ready for next call Confirm customer is happy and end call Walk through solution with customer, solve problem Confirm symptoms with customer Search for symptoms to problem to find potential fixes Triage and define problem Answer call, get customer details
- 54. Write up call notes and ready for next call Confirm customer is happy and end call Walk through solution with customer, solve problem Confirm symptoms with customer Search for symptoms to problem to find potential fixes Triage and define problem Answer call, get customer details
- 55. Write up call notes and ready for next call Confirm customer is happy and end call Walk through solution with customer, solve problem Confirm symptoms with customer Search for symptoms to problem to find potential fixes Triage and define problem Answer call, get customer details
- 56. Write up call notes and ready for next call Confirm customer is happy and end call Walk through solution with customer, solve problem Confirm symptoms with customer Search for symptoms to problem to find potential fixes Triage and define problem Answer call, get customer details
- 57. Write up call notes and ready for next call Confirm customer is happy and end call Walk through solution with customer, solve problem Confirm symptoms with customer Search for symptoms to problem to find potential fixes Triage and define problem Answer call, get customer details
- 58. Write up call notes and ready for next call Confirm customer is happy and end call Walk through solution with customer, solve problem Confirm symptoms with customer Search for symptoms to problem to find potential fixes Triage and define problem Answer call, get customer details
- 59. Write up call notes and ready for next call Confirm customer is happy and end call Walk through solution with customer, solve problem Confirm symptoms with customer Search for symptoms to problem to find potential fixes Triage and define problem Answer call, get customer details
- 60. Write up call notes and ready for next call Confirm customer is happy and end call Walk through solution with customer, solve problem Confirm symptoms with customer Search for symptoms to problem to find potential fixes Triage and define problem Answer call, get customer details
- 61. Write up call notes and ready for next call Confirm customer is happy and end call Walk through solution with customer, solve problem Confirm symptoms with customer Search for symptoms to problem to find potential fixes Triage and define problem Answer call, get customer details
- 62. Write up call notes and ready for next call Confirm customer is happy and end call Walk through solution with customer, solve problem Confirm symptoms with customer Search for symptoms to problem to find potential fixes Triage and define problem Answer call, get customer details - Search generally starts with google – very rare that user uses internal knowledge bases - User is very adept at search operators to refine searches fast – has list of ‘go to’ sites - Often maintains a repository of articles in a word doc on their computer as ‘short cuts
- 63. Write up call notes and ready for next call User Story: What is the overall task the user is trying to achieve? Confirm customer is happy and end call Walk through solution with customer, solve problem Confirm symptoms with customer Search for symptoms to problem to find potential fixes Triage and define problem Answer call, get customer details
- 64. When a customer calls me with a problem about their phone that I do not know the answer to Write up call notes and ready for next call Confirm customer is happy and end call Walk through solution with customer, solve problem Confirm symptoms with customer Search for symptoms to problem to find potential fixes User Story: Triage and define problem Answer call, get customer details
- 65. When a customer calls me with a problem about their phone that I do not know the answer to I want to find a correct and comprehensive answer to that problem as fast as possible Write up call notes and ready for next call Confirm customer is happy and end call Walk through solution with customer, solve problem Confirm symptoms with customer Search for symptoms to problem to find potential fixes User Story: Triage and define problem Answer call, get customer details
- 66. When a customer calls me with a problem about their phone that I do not know the answer to I want to find a correct and comprehensive answer to that problem as fast as possible So that I can fix the customer’s problem, make them happy and keep my call times down Write up call notes and ready for next call Confirm customer is happy and end call Walk through solution with customer, solve problem Confirm symptoms with customer Search for symptoms to problem to find potential fixes User Story: Triage and define problem Answer call, get customer details
- 67. When a customer calls me with a problem about their phone that I do not know the answer to I want to find a correct and comprehensive answer to that problem as fast as possible So that I can fix the customer’s problem, make them happy and keep my call times down Write up call notes and ready for next call Confirm customer is happy and end call Walk through solution with customer, solve problem Confirm symptoms with customer Search for symptoms to problem to find potential fixes User Story: Triage and define problem Answer call, get customer details Outcomes / Deliverables: If the user is successful in this task, what will be the end results?
- 68. When a customer calls me with a problem about their phone that I do not know the answer to I want to find a correct and comprehensive answer to that problem as fast as possible So that I can fix the customer’s problem, make them happy and keep my call times down Write up call notes and ready for next call Confirm customer is happy and end call Walk through solution with customer, solve problem Confirm symptoms with customer Search for symptoms to problem to find potential fixes User Story: Triage and define problem Answer call, get customer details Outcomes / Deliverables: • A happy customer with a working phone • Increased knowledge • A short phone call within my targets • A happy manager
- 69. Well that cannot possibly work. Everybody has a different workflow!! Are you going to document each one?!
- 70. Workflows differ by many things.
- 71. Workflows differ by many things. Job and Level Manager vs. IC
- 72. Workflows differ by many things. Job and Level Skill / Experience vs. Manager IC vs. Beginner Expert
- 73. Workflows differ by many things. Job and Level Skill / Experience Geography Manager IC Beginner Expert Lives in NYC Lives in rural WA vs. vs. vs.
- 74. Above all else, understand your users workflow.
- 77. Pragmatic Marketing Rules Only build solutions for problems that are urgent, pervasive and that the market will pay to solve.
- 83. Smug guy I don’t.
- 89. It is possible to do a load of user research and still get it wrong.
- 90. Above all else, understand your users workflow.
- 91. 1. Find some users. 1. “tell me about a time when” 1. Let them lead you to the problems they have
- 92. When a customer calls me with a problem about their phone that I do not know the answer to I want to find a correct and comprehensive answer to that problem as fast as possible So that I can fix the customer’s problem, make them happy and keep my call times down Write up call notes and ready for next call Confirm customer is happy and end call Walk through solution with customer, solve problem Confirm symptoms with customer Search for symptoms to problem to find potential fixes User Story: Triage and define problem Answer call, get customer details Outcomes / Deliverables: • A happy customer with a working phone • Increased knowledge • A short phone call within my targets • A happy manager
Editor's Notes
- Intro story
- Back in the dark ages of the 90’s, before iphone – this guy – Dr. Edward Stanio, had a company with a bold vision
- Over the next 2 years they launched these satellites into space – A HUGE technical achievement!!!!
- Over the next 2 years they launched these satellites into space – A HUGE technical achievement!!!!
- November 1st 1998 Al Gore made the inaugral call on the network to a great gradson of Alexanger Graham Bell – thus launching one of the most ambitious companies ever conceived
- By What went wrong?
- So what went wrong? Well firstly the costs of these phones was extremely high – over a thousand dollars for a handset, up to $8 a minute. For reference in yr. 2000 – Nextel was offering calls at 20c a minute The service did not work inside buildings – the phones required line if sight with a satellite The service did not work in moving vehicles!!!
- So this is the point where the smug product manager stands up and tells you all that iridium could have avoided all of that, if only they had done some market research and validated their business model with users……
- Here is the thing – they did!! They spoke to 20k users, across 4 continents……. SHIT!! That’s not how it is supposed to work? Eric Ries told us if we talk to our customers we can’t possible fail? Right? We are all screwed!!!
- Despite what the books and the blogs tell you.
- It is possible, like Iridium did, to blindly follow your customers off a cliff.
- And what I am going to talk about today is ‘Getting User Research right. How not to follow your users off a cliff” Or really some patterns that as product managers, we can follow to make sure the user research we do is intentional, valid and representative of our customers. Because honestly, often, it is not! As product managers we often fall into this role from engineering, or marketing. There is little formal training around our job and discipline, we have to figure stuff out for ourselves. Part of the fun right? A lot of this stuff Iearned from smart people I work with – but a lot of this I have learned from getting stuff wrong!!!
- Not gonna tell you that iridium could have saved 5bn by talking to customers Am sure there were many who noticed the writing was on the wall But I will tell you if you don’t talk to you customer in the right way, you really do run the risk of following them down a black hole Am going to give you another example – this one is mine, from a few years ago. I was product manager on a a piece of software that helped agents in call centers answer customers problems. The idea was that the agent would take the call, use our softwar to find the answer to the customers problem, and help the customer fix it! Simple!
- So we went and found some users! Here are our call center agents!!! Look how happy and smiling they are that they have to sit all day with a stupid bit of metal on their face and talk to irate customers!
- We went out and spoke to some users and potential users and they were all saying the same thing – - We asked – what do you want to see in your tools? What doesn’t work for you? And they all said….! “we can’t find what we are looking for!”, “we want better search!”
- So we went and built this. A unified search interface that will let me dig into the vast expanse of knowledge about a particular device.
- Accessed by a search box. We were expecting to see an increase in user engagement, which we had defined as ‘number of sessions performed per agent’. And whilst we saw a small increase, it was nowhere near the numbers we were expecting. And it didn’t work like we thought. So this is perplexing – they told us they wanted better search, why aren’t they using it? So we dug deeper. Well it turns out they were already using a search tool.
- Actually a pretty good search one! We ended up building a feature that competed with Google.
- that is not a great idea By the way, this is smug sheep, he will be popping in and out for the rest of the presentation I like to think he represents my inner pessimist. Because the sheep has a point - we failed to get at the root of the problem. What we hard was:
- What they meant was:
- We want better results! We got our research wrong, and we followed our users off a cliff.
- Once we realized this we changed out focus, and instead spent our time making novel and valuable articles and tools. Tools like this, that allow someone to walk through an example problem as if they had the phone in their hand/
- And it worked. So we are going to talk about some things you can do, to avoid this from happening. But the number 1 rule.
- Above all else. Understand your users workflow. Before you build anything. What does you user do today to complete this task? This is by far and away the most important one of them all. Customers are terrible at telling you want they want. You need to dig and probe to understand completely how your users currently work before you even think about coming up with a solution. If we would have understood initially that these users workflow involves using google to search, we would have made different decisions. If Iridium had understood that their customers workflow meant they made phone calls inside buildings, would they have made different decisions
- Now to do this, we need to be aware of how we are approaching a situation. Because there are 2 very different ways we can do this. There is a concept of problem space vs. solution space that I think defines this very well. And really what these 2 concepts mean is the mindset from which you approach a situation. Approaching a situation from the problem space, means being in learning mode, understanding the situation, the people, the needs, the desires. But being incredibly disciplined about not trying to fix any of it. Being in the solution space is the opposite. Once a problem has been discovered and defined, what are the things we can do to fix that problem?
- In the problem space we talk about things like customer needs, market problems, and customer desires.
- The solution space is where we talk about features and products.
- And when we are understanding a users workflow we need to be really disciplined about staying in the problem space mindset. And actually, it is pretty hard for us as product managers to do this. Because we like coming up with ideas, and solving issues. We are smart right? We can fix this? But when we are learning about our customers, we need to be focused on the problem not the solution.
- To understand workflow, we need to be in the problem space. We must be really really intentional about this…. And actually, we are going to stay in this mode for the rest of the presentation.
- So how do you do this? You got to find some users to talk to. Obviously they need to want to talk to you. And obviously they need to be users who are in your target market.
- Then do this. You have to actually talk them. Unescapable
- The sheep wants you to do a survey but…..
- NO SURVEYS!! Why? Because we are in problem space here – this is about exploring, learning. And this is really hard to do when you can’t ask questions back!! How many of us have sent out a survey, got an answer and been like ‘shit – I wish I could ask that person why they said that!!! Right?’ For the mode of learning we are in right now, surveys are inappropriate. You have to talk them. Unescapable. In person or on screenshare. Get out there!! Have fun!!
- Make sure you get a good mix of current, existing and potential customers.
- This is my favorite interview questtion: “tell me about a time when you solved a customers problem…..” “tell me about a time when you made a phone call…..”
- This question does so many good things. It encourages them to tell a story. It gets the interviewee into a wonderful state of mind where they are sharing their story and sharing their emotion, you can get so much insight from it. It sets the tone of the interview – you as the listener, them as the master Make sure you dig into the things you don’t understand. And at the end, repeat back what you have been told and ask them to
- Pleeeeeese don’t do this. You can’t just ask people what tey want. This is how to blindly follow users off a cliff. Even if you they are right, unless you understand why they want it, you will build it wrong anyway.
- I saw this on twitter this week entitled the PRODUCT DEATH CYCLE!!!!
- Let them lead you to the problems they have. Because you as you talk to them, the problems are going to come out. “and then I do this really annoying a slow tool to do….” “and after that I have to send this to this person but that never works because….”
- And at the end of each one. My second favorite interview quesiton. “if you had a magic wand, what would you change?” Ask “why”
- Let them lead you to the problems they have. Because you as you talk to them, the problems are going to come out. “and then I do this really annoying a slow tool to do….” “and after that I have to send this to this person but that never works because….”
- This is what one of our walls looks like in the office right now. A bunch of stickie notes, a different one for each interviewee. A different section for each workflow.
- So what is the end product here? What do we do with all this user data, all these sticky notes – where does this end up?
- Well it probably ends up looking a little bit like this. A set of steps the user goes through in order to solve their problem. So for our call center agent example:
- This will frame the context of the workflow
- Well it probably ends up looking a little bit like this.
- Well it probably ends up looking a little bit like this.
- Well it probably ends up looking a little bit like this.
- Well it probably ends up looking a little bit like this.
- If we can get to this point, with any user workflow we want to improve, we are in a really good place. And when we do jump into solution space this will help us, this will help design, engineering – because we truly understand how our users work. PAUSE
- Well. This is where you come in. As a product manager you have to use your skill to figure out which workflows are common to a group of users or a market, and which are individual.
- Because workflows differ by many thing. Different types of users have different ways of solving problems. This is fine, just understand how those differences play out.
- The way my boss approaches a task is different from how I do it. Well mainly because he just delegates it to me!! But it is a different workflow!
- What we noticed with our call center agents is that the experts who had been working there a long time actually had a shared social system of knowledge transfer. Using things like shared folders and private email aliases. They had become so jaded over time with how bad their tools were that they had replaced them with a system they had created themselves. Where as the beginners would rely solely on the official tools provided to them.
- Someone who lives in NYC has a very different workflow for ordering a cab than someone who live in rural washington. So we need to be aware of this. And often, the problem we focus on might only apply to a certain segment of our market. Uber do not service rural washington, they do service NYC. So we need to be acutely aware of these differences.
- Before you build anything, understand your customer’s workflow. And by the time you have done this, it will start to becomes obvious which problems you might end up trying to solve. Because you are going to end up discovering a bunch of problems.
- This is a problem map we put together at Moz earlier this year around some research we did into how people are building content for the web.
- This is from Pragmatic Marketing, who are sponsors of this event and also have an excellent set of courses and training on product management, thoroughly recommend them. They have their 10 rules – this is number 8
- Only build solutions for the problems that are urgent, pervasive and that the market will pay to solve Only focus on the high value things
- So if we go back to our problem map
- You can see we have mapped out urgency along the y-axis here
- And then pervasiveness here.
- And it is pretty obvious now that this is where you end up focusing. The most pervasive, most urgent problems. And now, only now, should we start shifting our minds from problem space to solution space, and starting to think about what we can actually build.
- I am pleased to say I have a happy ending for you!!1 Because everyone likes a happy ending!!!
- Apart from this guy – he likes sad and tragic endings, preferebaly with dire consequences for the human race
- But I have a happy ending!!! Remember I told that Iridium were going to burn all their satellites? Well they didn’t
- In fact they are still around…… An investment group bought out their assets at a dirt cheap price, and with a little help from the DoD, built iridium v2 – which is by all accounts a pretty successful company….. They are currently trading on NASDAQ at about 8 dollars, and are worth a little under a billion dollars. Not quite the company that Dr. Edward Stanianio dreamt of 20 years ago, but decently succesfull. Lets look closer:
- Emergency Response! Now that’s a smart market to target! Users have a very painful and urgent need for communication, and a workflow that could mean the matter between life and death
- Maritime Tracking and Navigation! THIS!! Even Verizon doesn’t have service in the middle of the atlantic
- And my favorite of all – earlier this year Iridium built a network of 10 modems at the Barneo ice camp and the very top of the planet. Now the north pole has fucking wifi internet !!
- Despite what the books and the blogs tell you.
- Do this by really digging into who they are and what they do
- If we can get to this point, with any user workflow we want to improve, we are in a really good place. And when we do jump into solution space this will help us, this will help design, engineering – because we truly understand how our users work. PAUSE
- Focus on the most urgent and pervasive problems.