Uploaded byLaura Klein
Build Better Products: How to Identify and Validate Assumptions
The document outlines the principles of the Lean Startup methodology, emphasizing the importance of identifying and validating assumptions related to problems, solutions, and implementations before developing a product. It provides a framework for entrepreneurs to categorize and prioritize their assumptions and to formulate hypothesis statements for testing. Through examples like Dropbox, it illustrates how to articulate and validate these assumptions to reduce risk in product development.
More Related Content
Similar to Build Better Products: How to Identify and Validate Assumptions
Build Better Products: How to Identify and Validate Assumptions
- 1. Build Better Products How to Identify & Validate Assumptions
- 2. Laura Klein laura@usersknow.com @lauraklein www.usersknow.com
- 3. Research (If you’relucky) Define Ideate Prototype Test Waterfall Design Develop
- 4. The Problem Meh Research Define Ideate Prototype Test Design Develop
- 5. Illustration by KateRutter
- 6. Image Used WithoutPermission of Any Sort
- 7. Lean Startup isNOT about… spending no money releasing a crappy product just throwing something against the wall to see if it sticks a/b testing your way to success changing your business model constantly
- 8. Lean is aboutlearning
- 9. Why is itbetter? By using Lean Startup, we identify assumptions before we base our whole product strategy on them.
- 10. So, what isan assumption, anyway?
- 11. Types of Assumptions Problem Solution Implementation
- 12. Types of Assumptions Problem Is there a problem? Solution Is this the right way to solve it? Implementation Can you build & sell it before running out of money?
- 14. Problem “People whohave pets are upset by cost.” Illustration by Kate Rutter
- 15. Solution “People needa marketplace for pet jobs.” Illustration by Kate Rutter
- 16. Implementation “Pets canwork!” Illustration by Kate Rutter
- 17.
- 18. What’s The Problem? Do people have trouble accessing files across multiple devices?
- 19. Problem Assumptions PeopleHave Multiple Devices People Want to Access the Same Files Across Those Devices This Is Hard to Do
- 20.
- 21. What’s The Solution? Are people willing to download multiple applications and trust storing files in the cloud?
- 22. Solution Assumptions PeopleWill Download DropBox on Multiple Devices People Will Trust the Service with Important Data People Understand How DropBox Folders Work
- 23.
- 24. What’s The Implementation? Can we sync files across multiple devices quickly and accurately enough?
- 25. Implementation Assumptions WeCan Build a Product that Syncs Files Across Multiple Operating Systems The Product Will Sync Quickly and Accurately
- 26.
- 27. Here’s The Template For Product to succeed, it is necessary that ___________________.
- 28. 10 assumptions, 1per sticky
- 29.
- 30. Divide into “WillKill” and other Will Kill Other
- 31. What Makes anAssumption Risky?
- 32. So, What’s Yours? Devastating Unlikely Likely Annoying
- 33. Building is NEVERyour Riskiest Assumption
- 34. Ok, Almost Never Illustration by Kate Rutter
- 35. Targets Change Illustrationby Kate Rutter
- 36. Now, Identify YourBiggest Risk. Turn it into a Hypothesis Statement. Will Kill
- 37. A hypothesis statement should contain: 1. What you believe 2. How you will know if you were right
- 38. Problem Hypothesis Example: We believe people like customer type have a need for (or problem doing) need/action/behavior. We will know we have validated this when we see quantitative/measurable outcome or qualitative/observable outcome. * *This text was almost entirely stolen from Janice Fraser.
- 39. Solution Hypothesis Example: We believe people like customer type will solve their problem by solution behavior. We will know we have validated this when we see quantitative/measurable outcome or qualitative/observable outcome.
- 40. Implementation Hypothesis Example: We believe our company can provide solution by implementation method. We will know we have validated this when we see quantitative/measurable outcome or qualitative/observable outcome.
- 41. Now You TryIt Write Down: 1. What you believe 2. How you will know if you were right
- 42.
- 43. Various Types ofTests
- 45. Identify the BestMethod of Validation Problem Solution Implementation Landing Page Audience Building Concierge Wizard of Oz Fake Door Selling
- 46. Questions? laura@usersknow.com @lauraklein usersknow.blogspot.com
Editor's Notes
- #2 Hi. How is everybody doing today? Before we get started, I want to know, how many of you are in charge of making product decisions? And how many of you are totally confident that all of those decisions that you’re making are right every single time? Anyone? Ok. Get out. For everyone else, I’m going to share a technique for making better decisions. We’re going to do this by identifying the assumptions you’re making during your product development process. By the end of the talk, hopefully you’ll know how to figure out which dangerous ideas are the most likely to destroy your product. Everyone ready? Great, let’s do this.
- #3 In case you’re wondering who the hell I am, this is me. I’m Laura Klein. I wrote a book called UX for Lean Startups. I’ve been designing and building products for almost 20 years, and I’ve worked with a lot of product teams who were making a lot of decisions. And I’ve seen a lot of products fail. I’ve also seen some succeed. Not as many. Most importantly, I’ve spent some time studying the differences between the processes that produce those products.
- #4 In the past, most teams built products like this. This may look familiar. It’s a normal, user centered, waterfall design process. It’s been used to create a lot of products, especially in large organizations. In waterfall, we do some user research (well, sometimes), we identify a problem, we ideate to come up with some possible design solutions, then we create wireframes or a prototype, maybe do some prototype testing, create a nice visual design, and then throw it all over the wall to the engineers. Overall, it’s not really a terrible process. If done right, it can create very usable products. I did it for years. Many companies still make successful products this way.
- #5 But there is a big problem with it. You see, it can (and often does) create really usable products that aren’t at all USEFUL. In other words, a lot of the time, you get to the end of this process, and you find out that nobody is interested in the solution you’ve built to solve a problem. Or you find out that, while you’re solving A problem, you’re not solving the user’s most important problem. And, of course, the worst part of it is, you often don’t find out that nobody wants your product until the product has already been built, and in many cases this can take a very long time.
- #6 Need an example? Ok. Anybody remember Webvan? For those of you who weren’t around in the 90s, Webvan was a startup that delivered groceries to you. It sort of famously failed. But nobody thinks that Webvan failed because of usability, right? The problem wasn’t that people COULDN’T buy groceries online from webvan. God knows I bought a few. And the problem wasn’t that they couldn’t build a website for people to buy groceries. They absolutely did that. The problem wasn’t even that they couldn’t deliver the groceries to people. Hell, they built out a whole system of warehouses and distribution centers in order to support delivering groceries at a truly massive scale. Of course, THAT was the problem. The fact is, they simply couldn’t get enough people at that point in history to order groceries from Webvan to support the company at the scale they were planning. And the bigger problem, the MUCH bigger problem, was that Webvan spent in the neighborhood of $400 million dollars figuring that out. $400. Million. Dollars. That’s starting to be real money. So, how could they have figured out that they had a problem sooner? Preferably $399 million dollars sooner.
- #7 This is one of the ways they could have done it, if this book had existed in the 90s, which it didn’t because I think Eric was in high school at the time. Figuring out early whether people have the problem you’re solving and want the your solution to the problem is what The Lean Startup is meant to do. Now, you’ve probably heard a lot about The Lean Startup. And unless you heard it from Eric or read it in one of the books from the O’Reilly Lean Series, I’d like you to forget most of the stuff you think you know. Because a lot of the stuff people say about Lean Startup is absolute rubbish.
- #8 For example, these are all Lean Startup myths.Things like that Lean Startup is about not taking money or making a crappy product or optimizing button color. That’s all nonsense. Those are all things that people did before Lean Startup came along, only now they justify them by saying “MVP!” or “Pivot!” Doing those things doesn’t make you Lean. In fact, they often make you fail.
- #9 I’m sure you’ll hear this a million times at this particular conference, but it bears repeating. Lean Startup is about learning. At every step of the way. Every time you build something, you measure it and learn from it. And you do it iteratively so that you’re constantly getting lots of different kinds of real feedback about whether your idea is any good.
- #10 So why is Lean Startup better than the old waterfall methodology? I mean, as I mentioned before, waterfall hasn’t just produced Webvan. It’s produced some pretty good stuff, too. Mostly, Lean Startup is about finding out if you’re wrong early. So that means you find out you’re wrong before you spend $400 million dollars building something that nobody wants. One of the ways Lean Startups do this is by consciously identifying the assumptions that we’re making and testing them to see if those assumptions are solid.
- #11 But let’s back up for just a second and be incredibly pedantic. What is an assumption? An assumption is, obviously, something that we just believe is true, generally with no evidence. It’s something we take for granted. If you’re building a house, you might assume that the ground you’re building it on won’t open up and swallow the house once you’ve built it. Which is not a bad thing to verify, when you think about it. For example, when building a house, you might want to test to make sure it’s not being built on sand or over an abandoned mine.
- #12 So what kinds of assumptions are we making when developing products? Well, we make assumptions about a lot of things when developing a product. We make assumptions about the problem we’re solving and the person we’re solving it for. We make assumptions that our solution is the best one. We make assumptions that we can actually build and monetize our solution.
- #13 In fact, unsuccessful companies often make key assumptions about these sorts of things. They assume that there is a problem. They assume that their solution is the best way to solve it. And they just assume that they can build the product and sell it before running out of money. And that’s incredibly dangerous.
- #14 I think it’s time for a bizarre example. It’s been minutes, and we haven’t had a single bizarre example yet. Now, for those of you who haven’t heard of it, I want to introduce you to a brand new product. It’s called Jobs4Pets. It’s the premier job site for pets in the world. That’s right, if you have a bored border collie lying around the house, you monetize that mutt with Jobs4Pets. It’s Monster for dogs. It’s TaskRabbit for actual rabbits. It’s also not really a thing. It’s just a fake product that I use as an example in all of my presentations. If you’re wondering why I use it, other than because I own the domain, the answer is that I give talks to a lot of startups. Jobs4Pets is both immediately understandable (I mean, you guys know what this is right? It’s a marketplace to get a job for your pets.) and also, it’s so incredibly stupid that I’m almost guaranteed that nobody in the audience is actually building it. And Jobs4Pets, as amazing an idea as it is, has some assumptions baked into it. Assumptions like that people who have pets are upset about how much they cost. Assumptions like that the best way to address the cost problem is to get jobs for the pets. And, of course, assumptions like the idea that pets can perform jobs that other people would pay for. If you look at the different assumptions, you will realize that they’re really quite different from each other. These are assumptions about the Problem, the Solution, and the Implementation.
- #15 An assumption like “People who have cats are upset by the cost” is a problem assumption. You’re assuming that there’s a group of people “pet owners” who are unhappy about something or have an unmet need - “the cost of owning pets.” If you’re wrong about this problem, it doesn’t matter how great your product is, nobody will use it, because it won’t solve a problem for anybody.
- #16 An assumption like “The right way to solve the pet cost problem is to employ your pets” is a solution assumption. You’re assuming that getting a job for a pet is a reasonable, even an attractive (and certainly not at all insane), way to solve the problem you’ve identified and that a marketplace is the best way for people to offer pet jobs and find jobs for their pets. Now remember, a problem can have many potential solutions. Some solutions are better than others. The analogy I always use here is, if I were to ask people if they want to lose 20 pounds quickly, many people would say yes. If I said that the way I could make that happen is to cut off a leg, they would probably say no. Good problem identification. Bad solution selection! Very very bad. You see, If you’re wrong about the solution assumption, people with the problem might express interest or even give you a try, but they will soon find other ways to solve their problem and probably won’t stick around.
- #17 An assumption like “Pets can work” is an implementation assumption. It’s basically addressing the feasibility of your solution. For example, if your implementation relies on something like warp speed travel or the employability of pets, that’s a pretty risky assumption. You’re going to want to validate the assumption before you base your whole product on it. If you’re wrong about the feasibility of the implementation, it doesn’t matter if it’s the best solution to the hardest problem in the world. If, for whatever reason, you can’t build it, it simply won’t work. You’ll need to find a solution you can actually implement. And do remember, I’m not just talking about building a website or a mobile app, or even a physical product. I’m also talking about the feasibility of building a community or a marketplace or an audience, depending on what your product requires. Implementation is about more than just technology. I know I can build a website that connects pets with jobs. I’m not sure I can teach a border collie to type.
- #18 Let’s Take a look at some actual examples that don’t involve pets. Now, to be clear, I have no idea if Dropbox actually went through this process, but they certainly did come up with a great problem, solution, and implementation, so they managed to validate their assumptions whether they did it intentionally or not. So, what’s the Problem Risk for a company like Dropbox? Not now. Think back to the dark ages before Dropbox? What was the initial problem that they had to validate? This isn’t a trick question, by the way. It’s a pretty obvious one.
- #19 That’s right. They had to make sure that a large number of people actually had a very specific problem - that they had multiple devices where they wanted to access the same files. Before you roll your eyes and say, “Well of COURSE that was a problem,” I want you to try to imagine a time or a place where that wouldn’t be a problem. I know most of you are probably too young to remember it, but once upon a time, people really only had one device, if that. Or they’d have their work computer and their home computer, and they wouldn’t share documents. Many people even today work largely off their smart phones, so something like Dropbox isn’t solving a problem for them. Or people who work entirely in the cloud with something like a Chromebook probably don’t have the problem.
- #20 So, the problem assumptions for dropbox that need to be validated are these. People have multiple devices. People want to access the same files across those devices. This is hard to do. All of those need to be true to validate that there is a problem.
- #21 Now, let’s find the solution assumptions. again, thinking back to the beginning. What was the Solution Risk for Dropbox? Once you validate that people do, indeed, have multiple devices and that they want to access the same files across those devices and that they have trouble doing that, what assumptions do you need to validate about how you’re going to solve that problem?
- #22 So, the assumptions they had to validate were that people were willing to download something onto their computers, phones, tablets, etc and to then trust that their files would be safe in an account in the cloud. Again, this all seems terribly obvious now, but it wasn’t originally. The fact is, downloading products onto computers was actually significantly less common when Dropbox launched than it had been several years earlier. And while most of us were comfortable storing various pieces of data, like photos or email, on different services like Flickr or Gmail, it wasn’t at all a forgone conclusion that people would be willing to move important files off their computers into a service run by a startup. Also, when you think about it, the real differentiator for Dropbox was the simple drag and drop interface that downloading a client gave people. Instead of making people go to a website and upload everything explicitly, it simply integrated with the operating system and let people feel like they were using their own computer. This requires a surprising amount of trust.
- #23 So, the solution assumptions that needed to be made were that people were willing to download multiple applications in different places and trust dropbox with their stuff. Also, that people would understand the way that the dropbox folders just worked automatically as long as you put stuff in them. Again, none of these could be taken for granted at the time, and several backup systems existed that handled things differently because they made different assumptions.
- #24 One more example for Dropbox, this time looking at implementation assumptions.
- #25 In this case, DropBox had an interesting challenge - namely, could they sync files across multiple devices and operating systems quickly and accurately enough to make customers happy? Unlike many products, where the implementation details might not be much harder than building an iOS app, this was a non-trivial task. As with the other assumptions, if they hadn’t been able to validate this implementation, Dropbox would not have been nearly as successful. While it might have still been a decent product if it just synced between, say, iPhones and Macs, it wouldn’t have been nearly as useful as something that syncs across all sorts of devices seamlessly.
- #26 So, the implementation assumptions that needed to be validated were that the product could not just be built, but built in a way that was quick, accurate, and secure across multiple kinds of device.
- #27 Ok, now it’s your turn. I want you to start to think about your own product. It can be a new product, like Jobs4Pets. Or it can be adding a feature to a product. Or it can even be something like acquiring new features for an existing product. Whatever you’re working on right now, rest assured, you are making some assumptions, and you want to identify them before you build your whole business on them.
- #28 Here’s an easy way to identify assumptions. Just fill in the blank. For Product to succeed, it is necessary that pets can work. A more dire way to put it is “If it’s not true that pets can work, I will go out of business.” Either way, you need to identify your assumption.
- #29 Now, I’m going to give you some time to do this. I want you to take 3 minutes to think about your product. If you’re here with your team, you’re going to share your info with your team mates, but you will do this first writing part alone. I want you to write down 10 big assumptions about your product. If you’re working on a new feature for a product or a new pricing strategy or whatever, you can use those. If you’re literally not working on anything, you should start. But if you can’t think of anything feel free to use Jobs4Pets or any other product you use on a regular basis. You’re going to write down 10 assumptions. Each assumption goes on a sticky note.
- #30 When you’re done, you’re going to categorize your assumptions. I want you to determine if these are assumptions about your Problem, your Solution, or your Implementation. Hopefully you have a few of each. Go ahead and write P, S, or I on each of the sticky notes to indicate what type of assumption it is. Sometimes, if you have trouble figuring out whether something is an assumption about your problem, your solution, or your implementation, it can indicate a problem with the assumption itself. So, please ask for help if you’re having trouble. [BREAK]
- #31 Now you’ve got 10 assumptions. I want you to divide them into a “will kill” pile and a “will probably not kill” pile. Will kill means that, if that assumption is invalid, it will literally kill your product. This is the part where you’re identifying the riskiest assumptions.
- #32 [Of course, we can’t validate every assumption. It would take forever and be silly. We can assume things like people use the internet. Well, we can assume that if our audience is in a place where the internet is widespread. If you’re making a mobile app to be used in national parks, you might want to validate whether people WILL have internet access. Regardless, the first thing you need to do is to determine whether an assumption is worth validating. In other words, is the assumption risky? Is it at all likely to be invalid?] To do this, we need to identify some of the characteristics of a risky assumption. As I mentioned before, the invalidation of one of these assumptions can kill your product. Or your company.
- #33 When we say that something is risky, we are often talking about two separate factors - how likely is it to happen and how bad would it be if it happened. There are things that are extremely unlikely to happen, but if they did happen, they’d ruin your company or kill somebody. Even though they’re not very likely, they’re still pretty risky. There are other things that may cause no more than an angry phone call to your company, but they’re very very likely to happen to a lot of customers. Those can be risky too. Especially if you’re the one who has to answer the phone. Generally I’ll make this grid and put my various assumptions onto it to help me sort out the most important hypotheses to test. Obviously, the thing at the upper right gets tested immediately. In fact, everything in that upper right quadrant should definitely be validated. After that, it’s up to you and your resources, but whatever you decide to test, this can be a nice, simple tool for prioritizing. Why don’t you take a look through your stickies and decide which one goes in the top, right quadrant of this graph. That’s going to be your riskiest assumption.
- #34 And when you’re figuring out your riskiest assumption, here’s something important to remember. The one thing that you want to do? The one thing that you want to spend all of your time working on? You know what it is. It’s building the product. You want to write code and ship things! Of course you do! That’s the fun bit. That’s the part where you get to see your vision come to life! But the problem is, whether or not you can build something is almost never your riskiest assumption. Can I build a website? Yeah. Of course. Can I build a two sided marketplace? Maybe. That’s kind of hard to do. Can I build a two sided marketplace of people who want to get jobs for their pets or hire other people’s pets? Ummmm….I’m not sure.
- #35 Oh, there’s one exception to that rule. If your name is Elon Musk (and it is not), and you’re build freaking rocketships and magical electric cars, this advice does not apply to you. The riskiest assumption for you from the very beginning may actually be whether you can build the thing. Can you launch something into space before going broke? Whether or not you can sell the thing is the smaller risk in the case of something like Space X.
- #36 But even if you are Elon (and again, you aren’t), even YOUR riskiest assumption is going change over the course of your product’s lifespan. For example, once you have a good idea that you can build a magical, indestructible car like the Tesla, your riskiest assumption may become whether you can sell that car for a hundred thousand dollars. Remember, before Tesla, most of the people who desperately wanted electric cars tended to be hippies who were frankly, rather unlikely to have a hundred grand to drop on something that looks like a Lotus and drives like an iPad. So, at some point, whether you can sell the thing is a pretty big part of the risk. Then, when you have somehow created a waiting list of slavering billionaires who are waiting for your space car, your riskiest assumption becomes whether you can make enough of them to keep up with demand. The point here is that sometimes your riskiest assumption will be about the problem you’re solving, sometimes it will be about your solution for it, and sometimes it will be about the actual implementation. I don’t know which it is for you. That’s something you need to answer for yourself. Targets move constantly. Your riskiest assumption right now may not be your riskiest assumption next week. But your riskiest assumption is not whether you can build an iPhone app. It’s really never that.
- #37 Now that you’ve identified your riskiest assumption, let’s turn it into something that you can test. To do that, we’re going to make it into a hypothesis statement.
- #38 A hypothesis statement should contain: the thing that you believe to be true and the way that you will know if you were right. In other words, you need to restate your assumption in a way that makes it falsifiable. Unless you’re open to the idea that your idea might be wrong, you’ll never really be able to validate it.
- #39 If you want to validate a problem assumption, you do it like this. [read screen] Here’s an example: We believe that there is a large set of pet owners who have a need for offsetting the cost of pet ownership. We will know we have validated this when we get 1000 people to sign up for more information about reducing the cost of pet ownership.
- #40 For a solution assumption,do it like this. [read screen] Here’s an example: We believe that people will agree to rent their pets to other people for money. We will know we have validated this when we get 1000 people to make profiles of their pets and offer them for hire.
- #41 And for an implementation assumption. Do it like this. [read screen] Here’s an example: We believe that pets can do valuable work. We will know we have validated this when we have successfully gotten jobs for 100 pets with other people and had 80% of those people say they were satisfied with their experience.
- #42 Go ahead and take a minute and convert your riskiest assumption into a hypothesis statement. Remember, write down 1. What you believe and 2. How to know if you’re right.
- #43 Now that you have a falsifiable hypothesis statement, you should try to falsify it! That’s right, you’re going to try to prove you wrong. Or right. Whichever. The goal is to design some way to decrease the risk that you’re building on top of a shaky assumption and that means testing just how shaky that assumption really is. And being open to the idea that you might have to start over. [Now, I’m not going to go through examples of a bunch of different tests, mostly because I’ve already done that. If you watch my presentation from the Lean Startup conference in 2013 on YouTube, you can see me explaining several different ways of testing assumptions in a fair amount of detail. Those types of tests are: ]
- #44 [Dashboards, Audience Building, Concierge, Wizard of Oz, Fake Doors, and Selling] You can think of each of these as a Minimum Viable Product with the goal just to test a different type of assumption. We’ll also talk about what sorts of assumptions this would be useful for validating.
- #45 I want to just head off an important question that I know you’re going to have before I let you jump into this. I know this is starting to sound a little like science with our assumptions and our hypotheses and our validation tests. Your goal is not actually to scientifically validate or invalidate this hypothesis. You don’t need to prove beyond any doubt that you’re right or not. We’re not seeking FDA approval. What you need to do is significantly lower the risk that you’re wrong. The fact is, none of these methods - or any method really beyond building the product - will be 100% effective in determining whether your assumptions are all correct. However, you can absolutely reduce the chances that you are building something that nobody will want by testing each of your assumptions.
- #46 Once you’ve watched the video from last year’s talk, you can use this handy chart to decide how to test your hypothesis statements. Think about what your assumption was: was it the problem, the solution, or the implementation? Also, think about where your product is currently. Is it still just an idea? You’re going to want to pick one of the validation methods that works without a real product, like audience building or landing pages. Is it a product of some sort already? You might try something like a Fake Door to gauge interest. If you don’t have engineering resources, you’re not going to be able to use things like Wizard of Oz or Fake Doors, obviously, but Concierge and Audience Building are great for people who don’t have tech resources. Basically, find the method that works best for you. Or feel free to come up with your own method or use some sort of combination of these. It’s entirely up to you. The only rule is that it has to help you figure out if your riskiest assumptions is wrong. Because if it is, it really could ruin your chances to build something amazing that a lot of people want to buy.
- #47 Ok, everybody. Thanks so much for listening to me! If you have any questions, please let me know! I answer all the emails I receive, and I’m happy to discuss these sorts of things with you. If you enjoyed the talk, you’ll love my book, UX for Lean Startups! It makes a great last minute holiday gift for all the entrepreneurs on your shopping list! Thanks again!