Welcome! It’s great to see so many people interested in lean experiments. How many people work at companies of 50 or less people? 50-100? 100+ ?
This talk is aimed towards employees who may be just starting out on their own venture, or leading a team within a larger company and looking for new methods of working and ensuring the success of your products.
Have worked at different sized companies, and seen the way products are validated at each one Today most of my examples will be focused on an early stage product I’m working on at FullContact called FullContact for Teams.
The thing all these companies share? They all took a prototype out to customers and tested it on the spot with them. Then they fixed the patterns they saw, went out there again, and iterated their way from there.
We assume their success was immediate, but in reality they had been perfecting these ideas for years w/ early customers before the products really took off. Story of doing things that don’t scale - Airbnb flew to NY to speak with customers in that market. Stripe sat down with potential customers to sign them up on the spot. Slack ran a design sprint to test different ideas for onboarding. All of these companies spent time with their customers at the early stage to validate what they were building and iterate quickly.
Companies that: Fail as soon as they are publicly launched in the market Replicate the same features that a much larger competitor has Are so many things that consumers just can’t understand what it does
We are all in this room for a reason - we want to make great products. All we have to do is get all these factors right. With all these different factors, we shouldn’t leave it up to fate - it makes sense that we should take any chance we get to reduce the risk of the unknown.
No matter what you’re working on, whether it’s a company, product, or feature, this is the fundamental question you have to be able to answer. (lead to next slide)
If you don’t know how to answer that, this is a great opportunity to run some experiments and validate or invalidate while you are still in the early stages. Every initiative has core assumptions. Decide which ones are riskiest with your team. Then validate / invalidate them by running quick experiments. We can take the lean startup methodology, and circumvent that by taking hypotheses, designing experiments, and finding an answer in less than a week. It’s like I just invented time travel…
These are not the experiments you are thinking of. No complicated processes, don’t have to wear a lab coat, and it’s okay if it’s not perfect.
Experiments are any test you do for the sole purpose of coming to a conclusion about an assumption. The hypothesis you test has to be 1) 2). Without either of these, you end up with results that are useless. Extremely helpful for reducing risk around a product, especially when it requires significant engineering investment.
We can all agree that validating our assumptions is good, and we all want our products to be successful. So why aren’t more companies running experiments?
We all have five excuses for anything - for example, I have five excuses for why I’m not going to the gym today. I am here to go through the top 5 misconceptions about experimentation, and provide lessons learned and tips for overcoming each one.
My goal for you today is to leave you with the knowledge that this is easier, takes less time, and less people that you may think - all you have to do is get started.
This is the number 1 excuse we hear. You can’t lose momentum, or you can’t not have work for engineers to do, etc.
I couldn’t find the graphic I wanted, so I took one and modified it. Here you can see how a product goes from a MVP to a fully fledged solution. As our products become more complex, the cost of changing something also increases. This is why it’s important to invest the time at the early stage to make sure you’re building the right thing.
The big lesson here is that by investing time now, you can get quicker feedback loops from customers that will affect your decisions around the product. It can also act as a morale booster because of the short nature of experiments and the conclusions at the end.
Example: we built a prototype of all the features we thought were needed. I took it out to customers. I discovered that we had two sets of features that appealed to two different user segments. One was similar to our current user base, and the other was a new one. This taught us that we could cut down the scope in order to launch a v1 that appealed to our current userbase, then start work on a v2. Discovering we could cut down on scope helped reduce our overall development time.
Just kidding. Most managers aren’t like this.
Reason #2 - also a common one. A lot of the pressure here stems from reason #1 that I just went through - it’s hard for us to hop out of this “always keep shipping” mode. We are sometimes afraid to pause and not look productive.
The argument you can use here is that we incur more risk if we blindly push ahead. A few small assumptions made without validation can still get you a product that works, but add up many of them and your product can start snowballing out of control.
Here are a couple of tips for getting buy-in from management. First off, know what you’re dealing with - maybe you’ve done your homework and all your assumptions aren’t around fundamental aspects of your product. When you have your list of assumptions, assign a risk factor to them. From this list, pick the top three you want to validate and come up with some simple experiments for them. Bonus points if you think up experiments that take advantage of lax resources.
Think through what your worst-case scenario is. Let’s say a core assumption was wrong, and you have to change it. What does that cost now, and what does that cost a year from now?
Third - this is especially relevant when you’re working at a hardware company. There are still ways to experiment, but you’re not going to get the 3-5 day experiments that software companies can get.
The example here is Savioke. They were building robots that could perform simple tasks in hotels, for example bringing a toothbrush to occupants who forgot theirs. One of the ways they were able to test this was to control the robot with a game controller, and replace the touchpad with an iPad that showed Powerpoint slides.
Powerpoint is just one of the tools at your fingertips to hack together an experience that can fool your user. For software products, we swear by Invision to fake an experience on web or mobile. The fact that we have so many different shapes of screens we can project on (desktop size, tablet size, phone size) makes it even more extensible. For hardware products, I’ve seen companies test the UX of a product by 3D printing it and having users handle it to see how ergonomic it was.
This one is relevant to many folks in Colorado due to the many high-tech industries in the area. Our companies tend to lean more B2B, so this can be admittedly tough to get past.
Your existing customers are going to be some of the best testers of your work. We oftentimes underestimate how much people love giving feedback - especially when they are invested in your success. When you need fresh eyes though, I’d recommend LinkedIn for doing targeted searches for the profession you want. This isn’t the most obscure industry, but a lot of our experiments were tied to potential target markets. I chased down consultants, HR recruiters, marketing agencies, you name it. I found all of these people through LinkedIn, and reached out to them asking for a quick 20 minute interview. If the interview went well, I asked if they had other folks they’d recommend I speak with.
Our last excuse is a fear of hurting the relationship - of making a mistake so terrible, your customer hates you. Luckily, you have the powers of marketing and framing.
Framing is your friend here - we’ve found that if we go to our customers and get them in the right mindset, they are very forgiving and willing to give feedback.
Now that you’ve got everything in your arsenal to make sure you go back to your office ready to build that next big product, let’s do a recap of major themes. The goal with experimenting early and often is to validate your hunches. Oftentimes our teams get into big arguments made up of many differing opinions. The best way to pick the winning idea is by doing a short experiment.
This can be a 20-minute brainstorming exercise for you and your team together. Brainstorm a list of risks, then assign severity levels to each of them. Once you’ve identified the highest risks, you can design experiments to validate or invalidate them.
This exercise is good to do with your team because it sets a baseline and gets everyone onto the same page, and makes them all advocates for experimenting because you’ll all have talked about ideas for quick experiments you can do.
The lessons I shared in this presentation are about how I experimented my way into an MVP, but you can run experiments at any lifecycle stage of product. The most important time to do it though, is when your product is at a prototype stage and you are deciding which parts to “productize” and make production-ready.
Experimented in many areas: Pricing was the most interesting - we decided to implement a trial program to speed up the rate at which customers had to give us answers, and our Sales team tested out different price points with different customers to find out what stuck. Customer development - testing to see if we had product/market fit by doing interviews with many different industries.
Here you can see the prototype I was showing to customers - don’t be afraid to show them something that’s not perfect.
Experimenting Your Way to MVP
Your Way to
PM @ FullContact
Product Manager at FullContact
Previously PM at SendGrid, Generalist at BandPage
Bay Area native
Our job is this
Address the right target market, with the right problem, with the right solution and
marketing, at the right time… Easy right?
So how do we make sure we’re
working on the right things?
Shorten the Lean Startup cycle
Catch core problems
before investing too
Validate your riskiest
Image from Google Ventures
experiments? We run experiments to test
1. Can be validated / invalidated
2. Address areas of risk in the
No brainer, right? So why aren’t more
companies doing it?
Top 5 Reasons
1. I can’t justify taking time away
from building the product.
2. I don’t have approval from
management to do this.
3. We have a deeply complex
product. There’s no way to
simulate things on it.
4. Our product is too niche. I can’t
find users to talk to.
5. I don’t want to damage our
relationship with potential
I can’t justify taking time away from
building the product.
Time $ $$ $$$ $$$$
Cost of Change Increases with Product Complexity
Example: Customer Development
Goal: Product/market fit
Assumption: [X] industry is in our target market
Methods: Customer development interviews with 5 people from each industry
Most powerful questions asked:
“What are the top three challenges you face in your industry?”
“How do you currently solve [x] problem?”Results
● Findings helped me reorder our product roadmap
● Saved us development time by cutting down v1 scope
I don’t have approval from management to
Have a plan & explain it in cost/benefits
1. Make a list of big assumptions and costs associated with them if you get it
a. Have a few experiments prepared to validate/invalidate
b. Lean towards experiments that are quick and easy to run but get you big insights
2. Are there easy tests you can run to mitigate big expensive risks?
3. Frame as “early risk mitigation”
● If you can validate/invalidate your most important
assumption, it will 1) inform your product direction and 2)
pave the way for more experiments.
We have a deeply complex product.
There’s no way to simulate things on it.
Niche industries, Hardware, AI, VR, etc.
Powerpoint is your friend
Take shortcuts however you can
Invision: string together high-fidelity mockups to simulate interactivity
Powerpoint: when you need to mock something up very quickly
3D Printing: when you are testing the physical UX of a product
● Ability to go through rounds of changes before
engineering starts work
Our product is too niche. I can’t find users
to talk to.
Use the power of LinkedIn stalking
LinkedIn has robust advanced search features (job title, company, industry)
Craigslist ads for local in-person interviews
Leverage existing customers
● Once I got one person to talk to me, they would often
introduce me to their peers in the industry. Always ask
for intros to more people.
I don’t want to damage our relationship
with potential customers.
Frame the interviews
Pick customers who will be sympathetic
Use framing techniques to prime the customer for giving feedback
“I appreciate your time…”
“This is still early stages but we wanted to get your opinion at the start…”
“Since you are a long-time customer, I wanted to get your feedback on a change I’m considering…”
Make them feel special
● Many of our private beta testers are still avid users of the
app and respond with feedback when asked. They also have
free access to the app.
Goal is to test your riskiest
assumptions early in the game
Experiments run for 3-5 days
Multiple experiments can run at
5 people = pattern
Don’t worry, it doesn’t have to be a
Team exercise: biggest risks in your
Risk Severity (1-5, 5 highest) Hypothesis 1 Experiment 1
People won’t like my
5, obviously People hate the comics
but are too nice to say
What’d We Do at FullContact?
Team of 5: two Sales, one PM, one PMM, one Customer Success, one Engineer
Trello board for tracking experiments
Weekly check-ins to share learnings
Our prototype when we started.
Now: FullContact for Teams
Beta - onwards to v1!
De-scoped the feature set into two releases, by customer segment
Validated the need to invest in building a core set of features
Able to test out different pricing on potential customers
Discovered the problem we are solving is on a top 3 list of problems for multiple
Always Be Experimenting!
👋 Thanks for listening