The document discusses effective testing methods in product development, focusing on three categories: open-ended, goal-oriented, and guided tests. It emphasizes the importance of appropriate demographic testing and behavioral tracking over self-predictive questioning to gather useful insights. Additionally, it highlights the necessity of treating participants with respect during tests and measuring outcomes accurately to validate hypotheses.

1. ~ rewarding good ~
How to Create Tests
That Fail
by Pete Oliver-Krueger
@petekrueger | @MillionMunkeys

2. ~ rewarding good ~
“Any Test is Better Than No Test”

3. ~ rewarding good ~
“Any Test is Better Than No Test”
• 3 Main Categories of Tests
1. Open-Ended Tests: “Show me how you _____”
Customer Discovery
Problem Discovery
2. Goal-Oriented Tests: “Use this to do _____”
 Solution Discovery
3. Guided Testing: “Do these steps.”
 Functional Testing

4. ~ rewarding good ~
“Would You Use This?”

5. ~ rewarding good ~
“Would You Use This?”
• People notoriously can’t predict their own future.
• “If you knew the answers to your own problems,
they wouldn’t be problems.”
• Don’t ask questions, track behaviour.
• The only questions you should ask are non-
leading, clarifying questions.
• “When you say _____, what do you mean?”
• “Can you think out loud? What do you want to do?”

6. ~ rewarding good ~
“I’ll Get Mom/Spouse/Joe to Test It”

7. ~ rewarding good ~
“I’ll Get Mom/Spouse/Joe to Test It”
• Testing the wrong demographic is worse than not
testing at all ==> False Metrics ==> False Confidence
 Are they the decision makers?
 Is it worth something (of value) to them?
• Don’t know your demographic? ==> First Test!
 Who has the problem you’re solving?
 Who is willing to pay for it?

8. ~ rewarding good ~
“Stay On Topic”

9. ~ rewarding good ~
“Stay On Topic”
• Your gold branches off your initial tunnel.
• Active neurons excite the other neurons that
they connect to the most often.
• If someone goes on a tangent, explore the
connection.

10. ~ rewarding good ~
“No. Wait. Do it This Way.”

11. ~ rewarding good ~
“No. Wait. Do it This Way.”
• The single most-important statement you can say to ensure
the success of your usability tests: "This is a test of US.
This is not a test of YOU. YOU are grading US on how well
WE understand and meet YOUR needs.”
• People love to give advice; People love to judge; People
don't like to be judged.
• If you judge anything they say, they will clam up and filter
everything they say from that point forward.
• This is literally the most important thing to learn to run a
great test: Treat them like royalty, and hang on their every
word with fascination.

12. ~ rewarding good ~
“I Already Ran a Test”

13. ~ rewarding good ~
“I Already Ran a Test”
3-5 people (per demographic)
• 1 Person < 30%
• 3 People = 65%
• 5 People = 85%
Uses:
• Volume testing is for testing scalability.
• Quality testing is for product/market fit.
Nielsen Norman Group © 2000

14. ~ rewarding good ~
“I’m Testing My Product”

15. ~ rewarding good ~
“I’m Testing My Product”
• You are testing… psychology always!
• You are testing… your customer’s problem.
• You are testing… that they’ll embrace your solution.
• You are testing… that they hurt enough to change.
• You are testing… that they’ll give you money.
You don’t even need a product. Haven’t built it yet?
Test using someone else’s product.

16. ~ rewarding good ~
Your Tests Are Your Product
• No one creates products to maintain the
status quo.
• Everything you build is to test the effect of a
solution on a customer.
• Are you creating tests that fail?
• Are you measuring your results?
• How accurate is your measuring?
• Do your results match your intentions?

17. ~ rewarding good ~
Further Information
Want to practice designing metrics?
facebook.com/groups/LeanPolitics
We’re applying Lean Startup & Metrics to
political issues, to help cut through the hype.
For advice, questions, or help in designing good
tests, email me: pete@munkeys.com