Validation and hypothesis based product management by Abdallah Al-Khalidi

S§
Validation and Hypothesis based
Product Management

Focus
• Root cause for problems.
• Ditches ideas that irrelevant.
Why should you care?
Feasible
• Solves the real problem.
• No wasted efforts.
Objectivity
• Eliminate personal bias for
what causes what.
Replicable
• Results can be used as facts.
• Results are durable.
User-Centric
• Providing real value to users.

The Approach: Step by Step Process (Scientific Method)
1- Observations
2- Hypothesis
3- Experiment
4- Refine
5- Validate
Questions to answer. Observing a trend or a problem.
What is a hypothesis? How to build one?
& Good vs Bad hypothesis.
How to test hypothesis? What data to look at?
Refine hypothesis
How to measure the effect? Success metrics!
FYI: Scientific Method was founded by Ibn Al Haytham around 250 years ago.

Observations
Observing a trend or a problem.

Sources for trends or problems
Customer Service
SalesOnline Discussions
Product Vision

Sources for trends or problems

Prioritization Methods
Relative Weighting RICE Analysis
Based on business metric Based on usage frequency
Perfect for new features Perfect for optimizing existing features
Considers development effort Considers development effort
Reach, Impact, Conﬁdence, Effort

Relative Weighting
Prioritization Criteria Evaluation Factors
Increase Sales; maximize ROI
Increase customer trust
Establish a competitive
advantage
Improve productivity
Cost of development
Value Score

Prioritized List
Issue Type and ID ROI ICS ECA IP
Total
Value
Value
Percent
Estimate
Cost
Percent
Priority
Add FB sign up
Feature 10
Cash on Delivery
Bug 3
Feature 7
1 6 1 3
3 1 6 1
6 6 1 1
1 8 1 1
6 6 6 3
11
11
14
11
21
10
20
20
40
55
Add Gmail sign up 3 6 6 6 21 15
Pin Address on map 6 6 3 6 21 35
Bug 5 3 3 3 3 12 25
Feature 8 1 3 6 1 11 40
8%
8%
11%
8%
16%
16%
16%
9%
8%
4%
8%
8%
15%
21%
6%
13%
10%
15%
2
1
1.37
0.54
0.76
2.66
1.23
0.9
0.54

1- Observations
2- Hypothesis
3- Experiment
4- Reﬁne
5- Validate
The Approach: Step by Step Process
Reﬁne hypothesis

Hypothesis
What is a hypothesis? How to build it?
& Good vs Bad one.

What is a hypothesis?
"If _____[I do this] _____, then _____[this]_____ will happen."
If I added gmail login, then number of registered users will increase.
If I include Cash on Delivery payment method during checkout, then number of purchases will increase.
If I enable customers to pin their location on a map, then number of purchases will increase.

Hypotheses Tips
Before you make a hypothesis, you have to clearly identify the question
you are interested in studying.
The question comes first
A hypothesis is a
statement, not a question
Your hypothesis is not the scientific question in your project. The
hypothesis is an educated, testable prediction about what will happen.
Make it clear A good hypothesis is written in clear and simple language.
Keep the variables in mind
A good hypothesis defines the variables in easy-to-measure terms, like
who the participants are, what changes during the testing.
Make sure your
hypothesis is “testable”
Don't bite off more than
you can chew!
To prove or disprove your hypothesis, you need to be able to do an
experiment and take measurements to see how two things are related.
Make sure your hypothesis is a specific statement relating to a single
experiment.

Good vs bad hypothesis
Good Hypothesis Bad Hypothesis
Testable
Simple
Written as a statement
Establishes the
participants & variables
Predicts effect
Not testable
Not simply explained
Written as a questions
Doesn’t identify
participants & variables
Cannot use to predict
effect

Good vs bad hypothesis
Good Hypothesis Bad Hypothesis
If I added gmail login, then number of
registered users will increase.
If I include Cash on Delivery payment
method during checkout, then number of
purchases will increase.
If I enable customers to pin their location
on a map, then number of purchases will
increase.
Would adding more login options increase
number of registered users?
Cash on delivery payment is requested by
users
We have 10,000 calls from users to track
their orders

The point is to prove or disprove a hypothesis
Disproving a hypothesis matters as much as proving it. Both leads to form better
conclusions about the problem at hand!

Experiment

Creating a mathematical model
What questions do you want to
answer?
What data do you want to look
for?
Where the data you want is
available?
How to use the data available?
How to verify the data?
Will adding Gmail login improve sign up?
How many customers have signed up
using Gmail email?
Customers data base
Find % of customers signed up with Gmail.
Gmail customers / all customers = % of gmail customers.
Consider removing dummy created accounts

answer?
for?
available?
using Gmail email?
Customers data base
Gmail customers / all customers = % of
gmail customers.
Consider removing dummy created
accounts
150K
150K / 1.3M = 12%

Reﬁne
Reﬁne mathematical model, removing bias!

using Gmail email?
Gmail customers / all customers = % of
gmail customers.
accounts
150K
150K / 1.3M = 12%
answer?
for?
available?
Customers data base

We live in a dynamic world, so always consider things
in a timely manner!

answer?
for?
available?
using Gmail email in the past 6 months?
Customers data base
Find % of customers signed up with Gmail
in the past 6 months.
Gmail sign ups / all sign ups in the past 6
months = % of gmail customers.
accounts
120K
120K / 600K = 25%

1- Observations
2- Hypothesis
3- Experiment
4- Reﬁne
5- Validate
Reﬁne hypothesis based on test.

Validate

What is a success metric?
Metrics are a lens into your product’s
health and performance.

1- Observations
2- Hypothesis
3- Experiment
4- Reﬁne
5- Validate
Reﬁne hypothesis.

Thank you
Abdallah Al-Khalidi
Linkedin: abzkhaldi | Twitter: @abzkhaldi
Email: abdallah.khalidi@gmail.com

Validation and hypothesis based product management by Abdallah Al-Khalidi

More Related Content

What's hot

Similar to Validation and hypothesis based product management by Abdallah Al-Khalidi

Recently uploaded

Validation and hypothesis based product management by Abdallah Al-Khalidi