SlideShare a Scribd company logo
S§
Validation and Hypothesis based
Product Management
Focus
• Root cause for problems.
• Ditches ideas that irrelevant.
Why should you care?
Feasible
• Solves the real problem.
• No wasted efforts.
Objectivity
• Eliminate personal bias for
what causes what.
Replicable
• Results can be used as facts.
• Results are durable.
User-Centric
• Providing real value to users.
The Approach: Step by Step Process (Scientific Method)
1- Observations
2- Hypothesis
3- Experiment
4- Refine
5- Validate
Questions to answer. Observing a trend or a problem.
What is a hypothesis? How to build one?
& Good vs Bad hypothesis.
How to test hypothesis? What data to look at?
Refine hypothesis
How to measure the effect? Success metrics!
FYI: Scientific Method was founded by Ibn Al Haytham around 250 years ago.
Observations
Observing a trend or a problem.
Sources for trends or problems
Customer Service
SalesOnline Discussions
Product Vision
Sources for trends or problems
Impact Based Prioritization
Prioritization Methods
Relative Weighting RICE Analysis
Based on business metric Based on usage frequency
Perfect for new features Perfect for optimizing existing features
Considers development effort Considers development effort
Reach, Impact, Confidence, Effort
Relative Weighting
Prioritization Criteria Evaluation Factors
Increase Sales; maximize ROI
Increase customer trust
Establish a competitive
advantage
Improve productivity
Cost of development
Value Score
Prioritized List
Issue Type and ID ROI ICS ECA IP
Total
Value
Value
Percent
Estimate
Cost
Percent
Priority
Add FB sign up
Feature 10
Cash on Delivery
Bug 3
Feature 7
1 6 1 3
3 1 6 1
6 6 1 1
1 8 1 1
6 6 6 3
11
11
14
11
21
10
20
20
40
55
Add Gmail sign up 3 6 6 6 21 15
Pin Address on map 6 6 3 6 21 35
Bug 5 3 3 3 3 12 25
Feature 8 1 3 6 1 11 40
8%
8%
11%
8%
16%
16%
16%
9%
8%
4%
8%
8%
15%
21%
6%
13%
10%
15%
2
1
1.37
0.54
0.76
2.66
1.23
0.9
0.54
RICE Analysis
Prioritized List
1- Observations
2- Hypothesis
3- Experiment
4- Refine
5- Validate
The Approach: Step by Step Process
Questions to answer. Observing a trend or a problem.
What is a hypothesis? How to build one?
& Good vs Bad hypothesis.
How to test hypothesis? What data to look at?
Refine hypothesis
How to measure the effect? Success metrics!
Hypothesis
What is a hypothesis? How to build it?
& Good vs Bad one.
What is a hypothesis?
"If _____[I do this] _____, then _____[this]_____ will happen."
If I added gmail login, then number of registered users will increase.
If I include Cash on Delivery payment method during checkout, then number of purchases will increase.
If I enable customers to pin their location on a map, then number of purchases will increase.
Hypotheses Tips
Before you make a hypothesis, you have to clearly identify the question
you are interested in studying.
The question comes first
A hypothesis is a
statement, not a question
Your hypothesis is not the scientific question in your project. The
hypothesis is an educated, testable prediction about what will happen.
Make it clear A good hypothesis is written in clear and simple language.
Keep the variables in mind
A good hypothesis defines the variables in easy-to-measure terms, like
who the participants are, what changes during the testing.
Make sure your
hypothesis is “testable”
Don't bite off more than
you can chew!
To prove or disprove your hypothesis, you need to be able to do an
experiment and take measurements to see how two things are related.
Make sure your hypothesis is a specific statement relating to a single
experiment.
Good vs bad hypothesis
Good Hypothesis Bad Hypothesis
Testable
Simple
Written as a statement
Establishes the
participants & variables
Predicts effect
Not testable
Not simply explained
Written as a questions
Doesn’t identify
participants & variables
Cannot use to predict
effect
Good vs bad hypothesis
Good Hypothesis Bad Hypothesis
If I added gmail login, then number of
registered users will increase.
If I include Cash on Delivery payment
method during checkout, then number of
purchases will increase.
If I enable customers to pin their location
on a map, then number of purchases will
increase.
Would adding more login options increase
number of registered users?
Cash on delivery payment is requested by
users
We have 10,000 calls from users to track
their orders
The point is to prove or disprove a hypothesis
Disproving a hypothesis matters as much as proving it. Both leads to form better
conclusions about the problem at hand!
Sources for trends or problems
1- Observations
2- Hypothesis
3- Experiment
4- Refine
5- Validate
The Approach: Step by Step Process
Questions to answer. Observing a trend or a problem.
What is a hypothesis? How to build one?
& Good vs Bad hypothesis.
How to test hypothesis? What data to look at?
Refine hypothesis
How to measure the effect? Success metrics!
Experiment
How to test hypothesis? What data to look at?
Creating a mathematical model
What questions do you want to
answer?
What data do you want to look
for?
Where the data you want is
available?
How to use the data available?
How to verify the data?
Will adding Gmail login improve sign up?
How many customers have signed up
using Gmail email?
Customers data base
Find % of customers signed up with Gmail.
Gmail customers / all customers = % of gmail customers.
Consider removing dummy created accounts
Creating a mathematical model
What questions do you want to
answer?
What data do you want to look
for?
Where the data you want is
available?
How to use the data available?
How to verify the data?
Will adding Gmail login improve sign up?
How many customers have signed up
using Gmail email?
Customers data base
Find % of customers signed up with Gmail.
Gmail customers / all customers = % of
gmail customers.
Consider removing dummy created
accounts
150K
150K / 1.3M = 12%
1- Observations
2- Hypothesis
3- Experiment
4- Refine
5- Validate
The Approach: Step by Step Process
Questions to answer. Observing a trend or a problem.
What is a hypothesis? How to build one?
& Good vs Bad hypothesis.
How to test hypothesis? What data to look at?
Refine hypothesis
How to measure the effect? Success metrics!
Refine
Refine mathematical model, removing bias!
Will adding Gmail login improve sign up?
How many customers have signed up
using Gmail email?
Find % of customers signed up with Gmail.
Gmail customers / all customers = % of
gmail customers.
Consider removing dummy created
accounts
150K
150K / 1.3M = 12%
What questions do you want to
answer?
What data do you want to look
for?
Where the data you want is
available?
How to use the data available?
How to verify the data?
Creating a mathematical model
Customers data base
We live in a dynamic world, so always consider things
in a timely manner!
Creating a mathematical model
What questions do you want to
answer?
What data do you want to look
for?
Where the data you want is
available?
How to use the data available?
How to verify the data?
Will adding Gmail login improve sign up?
How many customers have signed up
using Gmail email in the past 6 months?
Customers data base
Find % of customers signed up with Gmail
in the past 6 months.
Gmail sign ups / all sign ups in the past 6
months = % of gmail customers.
Consider removing dummy created
accounts
120K
120K / 600K = 25%
1- Observations
2- Hypothesis
3- Experiment
4- Refine
5- Validate
The Approach: Step by Step Process
Questions to answer. Observing a trend or a problem.
What is a hypothesis? How to build one?
& Good vs Bad hypothesis.
How to test hypothesis? What data to look at?
Refine hypothesis based on test.
How to measure the effect? Success metrics!
Validate
How to measure the effect? Success metrics!
What is a success metric?
Metrics are a lens into your product’s
health and performance.
Success Metric
Measure the uplift
1- Observations
2- Hypothesis
3- Experiment
4- Refine
5- Validate
The Approach: Step by Step Process
Questions to answer. Observing a trend or a problem.
What is a hypothesis? How to build one?
& Good vs Bad hypothesis.
How to test hypothesis? What data to look at?
Refine hypothesis.
How to measure the effect? Success metrics!
Thank you
Abdallah Al-Khalidi
Linkedin: abzkhaldi | Twitter: @abzkhaldi
Email: abdallah.khalidi@gmail.com

More Related Content

Similar to Validation and hypothesis based product management by Abdallah Al-Khalidi

Measurement and metrics
Measurement and metricsMeasurement and metrics
Measurement and metrics
Alex Pandel
 
Master the Essentials of Conversion Optimization
Master the Essentials of Conversion OptimizationMaster the Essentials of Conversion Optimization
Master the Essentials of Conversion Optimization
joshuapaulharper
 
Analytics for Startups
Analytics for StartupsAnalytics for Startups
Analytics for Startups
Kissmetrics on SlideShare
 
Promoting Lean Startup at Zhihu in 2012
Promoting Lean Startup at Zhihu in 2012Promoting Lean Startup at Zhihu in 2012
Promoting Lean Startup at Zhihu in 2012
Tao Huang
 
6 Guidelines for A/B Testing
6 Guidelines for A/B Testing6 Guidelines for A/B Testing
6 Guidelines for A/B Testing
Emily Robinson
 
Unlocking the 4 Gateways of Growth
Unlocking the 4 Gateways of GrowthUnlocking the 4 Gateways of Growth
Unlocking the 4 Gateways of Growth
Kissmetrics on SlideShare
 
ESADE Business Analytics Master: Metrics that matter
ESADE Business Analytics Master: Metrics that matterESADE Business Analytics Master: Metrics that matter
ESADE Business Analytics Master: Metrics that matter
Juliana Mendez
 
Data-Driven UI/UX Design with A/B Testing
Data-Driven UI/UX Design with A/B TestingData-Driven UI/UX Design with A/B Testing
Data-Driven UI/UX Design with A/B Testing
Jack Nguyen (Hung Tien)
 
Steve Lawrence - Agile Metrics
Steve Lawrence - Agile MetricsSteve Lawrence - Agile Metrics
Steve Lawrence - Agile Metrics
AgileNZ Conference
 
Optimizing your marketing automation program for success
Optimizing your marketing automation program for successOptimizing your marketing automation program for success
Optimizing your marketing automation program for success
GetResponse
 
Market Validation (Scale) (4).pdf
Market Validation (Scale) (4).pdfMarket Validation (Scale) (4).pdf
Market Validation (Scale) (4).pdf
Aneel Mitra
 
How to Enter the Data Analytics Industry?
How to Enter the Data Analytics Industry?How to Enter the Data Analytics Industry?
How to Enter the Data Analytics Industry?
Ganes Kesari
 
Funnels Workshop Web Summit 2014 @geckoboard @GA
Funnels Workshop Web Summit 2014 @geckoboard @GAFunnels Workshop Web Summit 2014 @geckoboard @GA
Funnels Workshop Web Summit 2014 @geckoboard @GA
Sofia Quintero
 
Realise the Benefits of Holistic Testing
Realise the Benefits of Holistic TestingRealise the Benefits of Holistic Testing
Realise the Benefits of Holistic Testing
Holistic Email Marketing
 
Data mining - Machine Learning
Data mining - Machine LearningData mining - Machine Learning
Data mining - Machine Learning
RupaDutta3
 
Digital marketing metrics and measurement
Digital marketing metrics and measurementDigital marketing metrics and measurement
Digital marketing metrics and measurement
semrush_webinars
 
Lean LaunchPad: Analytics Workshop
Lean LaunchPad: Analytics WorkshopLean LaunchPad: Analytics Workshop
Lean LaunchPad: Analytics Workshop
Stanford University
 
Entering the Data Analytics industry
Entering the Data Analytics industryEntering the Data Analytics industry
Entering the Data Analytics industry
Gramener
 
CS 2014-15-2
CS 2014-15-2CS 2014-15-2
CS 2014-15-2Jim Head
 

Similar to Validation and hypothesis based product management by Abdallah Al-Khalidi (20)

Measurement and metrics
Measurement and metricsMeasurement and metrics
Measurement and metrics
 
Master the Essentials of Conversion Optimization
Master the Essentials of Conversion OptimizationMaster the Essentials of Conversion Optimization
Master the Essentials of Conversion Optimization
 
Analytics for Startups
Analytics for StartupsAnalytics for Startups
Analytics for Startups
 
Promoting Lean Startup at Zhihu in 2012
Promoting Lean Startup at Zhihu in 2012Promoting Lean Startup at Zhihu in 2012
Promoting Lean Startup at Zhihu in 2012
 
6 Guidelines for A/B Testing
6 Guidelines for A/B Testing6 Guidelines for A/B Testing
6 Guidelines for A/B Testing
 
Unlocking the 4 Gateways of Growth
Unlocking the 4 Gateways of GrowthUnlocking the 4 Gateways of Growth
Unlocking the 4 Gateways of Growth
 
ESADE Business Analytics Master: Metrics that matter
ESADE Business Analytics Master: Metrics that matterESADE Business Analytics Master: Metrics that matter
ESADE Business Analytics Master: Metrics that matter
 
Data-Driven UI/UX Design with A/B Testing
Data-Driven UI/UX Design with A/B TestingData-Driven UI/UX Design with A/B Testing
Data-Driven UI/UX Design with A/B Testing
 
Analytics for startups
Analytics for startupsAnalytics for startups
Analytics for startups
 
Steve Lawrence - Agile Metrics
Steve Lawrence - Agile MetricsSteve Lawrence - Agile Metrics
Steve Lawrence - Agile Metrics
 
Optimizing your marketing automation program for success
Optimizing your marketing automation program for successOptimizing your marketing automation program for success
Optimizing your marketing automation program for success
 
Market Validation (Scale) (4).pdf
Market Validation (Scale) (4).pdfMarket Validation (Scale) (4).pdf
Market Validation (Scale) (4).pdf
 
How to Enter the Data Analytics Industry?
How to Enter the Data Analytics Industry?How to Enter the Data Analytics Industry?
How to Enter the Data Analytics Industry?
 
Funnels Workshop Web Summit 2014 @geckoboard @GA
Funnels Workshop Web Summit 2014 @geckoboard @GAFunnels Workshop Web Summit 2014 @geckoboard @GA
Funnels Workshop Web Summit 2014 @geckoboard @GA
 
Realise the Benefits of Holistic Testing
Realise the Benefits of Holistic TestingRealise the Benefits of Holistic Testing
Realise the Benefits of Holistic Testing
 
Data mining - Machine Learning
Data mining - Machine LearningData mining - Machine Learning
Data mining - Machine Learning
 
Digital marketing metrics and measurement
Digital marketing metrics and measurementDigital marketing metrics and measurement
Digital marketing metrics and measurement
 
Lean LaunchPad: Analytics Workshop
Lean LaunchPad: Analytics WorkshopLean LaunchPad: Analytics Workshop
Lean LaunchPad: Analytics Workshop
 
Entering the Data Analytics industry
Entering the Data Analytics industryEntering the Data Analytics industry
Entering the Data Analytics industry
 
CS 2014-15-2
CS 2014-15-2CS 2014-15-2
CS 2014-15-2
 

Recently uploaded

Epistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI supportEpistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI support
Alan Dix
 
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Thierry Lestable
 
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Tobias Schneck
 
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
Product School
 
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Product School
 
JMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and GrafanaJMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and Grafana
RTTS
 
UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4
DianaGray10
 
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
Product School
 
DevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA ConnectDevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA Connect
Kari Kakkonen
 
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdfSmart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
91mobiles
 
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdfFIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance
 
Search and Society: Reimagining Information Access for Radical Futures
Search and Society: Reimagining Information Access for Radical FuturesSearch and Society: Reimagining Information Access for Radical Futures
Search and Society: Reimagining Information Access for Radical Futures
Bhaskar Mitra
 
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptxIOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
Abida Shariff
 
"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi
Fwdays
 
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdfFIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance
 
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
Product School
 
When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...
Elena Simperl
 
Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........
Alison B. Lowndes
 
PHP Frameworks: I want to break free (IPC Berlin 2024)
PHP Frameworks: I want to break free (IPC Berlin 2024)PHP Frameworks: I want to break free (IPC Berlin 2024)
PHP Frameworks: I want to break free (IPC Berlin 2024)
Ralf Eggert
 
GraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge GraphGraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge Graph
Guy Korland
 

Recently uploaded (20)

Epistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI supportEpistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI support
 
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
 
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
 
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
 
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...
 
JMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and GrafanaJMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and Grafana
 
UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4
 
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
 
DevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA ConnectDevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA Connect
 
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdfSmart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
 
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdfFIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
 
Search and Society: Reimagining Information Access for Radical Futures
Search and Society: Reimagining Information Access for Radical FuturesSearch and Society: Reimagining Information Access for Radical Futures
Search and Society: Reimagining Information Access for Radical Futures
 
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptxIOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
 
"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi
 
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdfFIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
 
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
 
When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...
 
Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........
 
PHP Frameworks: I want to break free (IPC Berlin 2024)
PHP Frameworks: I want to break free (IPC Berlin 2024)PHP Frameworks: I want to break free (IPC Berlin 2024)
PHP Frameworks: I want to break free (IPC Berlin 2024)
 
GraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge GraphGraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge Graph
 

Validation and hypothesis based product management by Abdallah Al-Khalidi

  • 1. S§ Validation and Hypothesis based Product Management
  • 2. Focus • Root cause for problems. • Ditches ideas that irrelevant. Why should you care? Feasible • Solves the real problem. • No wasted efforts. Objectivity • Eliminate personal bias for what causes what. Replicable • Results can be used as facts. • Results are durable. User-Centric • Providing real value to users.
  • 3. The Approach: Step by Step Process (Scientific Method) 1- Observations 2- Hypothesis 3- Experiment 4- Refine 5- Validate Questions to answer. Observing a trend or a problem. What is a hypothesis? How to build one? & Good vs Bad hypothesis. How to test hypothesis? What data to look at? Refine hypothesis How to measure the effect? Success metrics! FYI: Scientific Method was founded by Ibn Al Haytham around 250 years ago.
  • 5. Sources for trends or problems Customer Service SalesOnline Discussions Product Vision
  • 6. Sources for trends or problems
  • 8. Prioritization Methods Relative Weighting RICE Analysis Based on business metric Based on usage frequency Perfect for new features Perfect for optimizing existing features Considers development effort Considers development effort Reach, Impact, Confidence, Effort
  • 9. Relative Weighting Prioritization Criteria Evaluation Factors Increase Sales; maximize ROI Increase customer trust Establish a competitive advantage Improve productivity Cost of development Value Score
  • 10. Prioritized List Issue Type and ID ROI ICS ECA IP Total Value Value Percent Estimate Cost Percent Priority Add FB sign up Feature 10 Cash on Delivery Bug 3 Feature 7 1 6 1 3 3 1 6 1 6 6 1 1 1 8 1 1 6 6 6 3 11 11 14 11 21 10 20 20 40 55 Add Gmail sign up 3 6 6 6 21 15 Pin Address on map 6 6 3 6 21 35 Bug 5 3 3 3 3 12 25 Feature 8 1 3 6 1 11 40 8% 8% 11% 8% 16% 16% 16% 9% 8% 4% 8% 8% 15% 21% 6% 13% 10% 15% 2 1 1.37 0.54 0.76 2.66 1.23 0.9 0.54
  • 13. 1- Observations 2- Hypothesis 3- Experiment 4- Refine 5- Validate The Approach: Step by Step Process Questions to answer. Observing a trend or a problem. What is a hypothesis? How to build one? & Good vs Bad hypothesis. How to test hypothesis? What data to look at? Refine hypothesis How to measure the effect? Success metrics!
  • 14. Hypothesis What is a hypothesis? How to build it? & Good vs Bad one.
  • 15. What is a hypothesis? "If _____[I do this] _____, then _____[this]_____ will happen." If I added gmail login, then number of registered users will increase. If I include Cash on Delivery payment method during checkout, then number of purchases will increase. If I enable customers to pin their location on a map, then number of purchases will increase.
  • 16. Hypotheses Tips Before you make a hypothesis, you have to clearly identify the question you are interested in studying. The question comes first A hypothesis is a statement, not a question Your hypothesis is not the scientific question in your project. The hypothesis is an educated, testable prediction about what will happen. Make it clear A good hypothesis is written in clear and simple language. Keep the variables in mind A good hypothesis defines the variables in easy-to-measure terms, like who the participants are, what changes during the testing. Make sure your hypothesis is “testable” Don't bite off more than you can chew! To prove or disprove your hypothesis, you need to be able to do an experiment and take measurements to see how two things are related. Make sure your hypothesis is a specific statement relating to a single experiment.
  • 17. Good vs bad hypothesis Good Hypothesis Bad Hypothesis Testable Simple Written as a statement Establishes the participants & variables Predicts effect Not testable Not simply explained Written as a questions Doesn’t identify participants & variables Cannot use to predict effect
  • 18. Good vs bad hypothesis Good Hypothesis Bad Hypothesis If I added gmail login, then number of registered users will increase. If I include Cash on Delivery payment method during checkout, then number of purchases will increase. If I enable customers to pin their location on a map, then number of purchases will increase. Would adding more login options increase number of registered users? Cash on delivery payment is requested by users We have 10,000 calls from users to track their orders
  • 19. The point is to prove or disprove a hypothesis Disproving a hypothesis matters as much as proving it. Both leads to form better conclusions about the problem at hand!
  • 20. Sources for trends or problems
  • 21. 1- Observations 2- Hypothesis 3- Experiment 4- Refine 5- Validate The Approach: Step by Step Process Questions to answer. Observing a trend or a problem. What is a hypothesis? How to build one? & Good vs Bad hypothesis. How to test hypothesis? What data to look at? Refine hypothesis How to measure the effect? Success metrics!
  • 22. Experiment How to test hypothesis? What data to look at?
  • 23. Creating a mathematical model What questions do you want to answer? What data do you want to look for? Where the data you want is available? How to use the data available? How to verify the data? Will adding Gmail login improve sign up? How many customers have signed up using Gmail email? Customers data base Find % of customers signed up with Gmail. Gmail customers / all customers = % of gmail customers. Consider removing dummy created accounts
  • 24. Creating a mathematical model What questions do you want to answer? What data do you want to look for? Where the data you want is available? How to use the data available? How to verify the data? Will adding Gmail login improve sign up? How many customers have signed up using Gmail email? Customers data base Find % of customers signed up with Gmail. Gmail customers / all customers = % of gmail customers. Consider removing dummy created accounts 150K 150K / 1.3M = 12%
  • 25. 1- Observations 2- Hypothesis 3- Experiment 4- Refine 5- Validate The Approach: Step by Step Process Questions to answer. Observing a trend or a problem. What is a hypothesis? How to build one? & Good vs Bad hypothesis. How to test hypothesis? What data to look at? Refine hypothesis How to measure the effect? Success metrics!
  • 27. Will adding Gmail login improve sign up? How many customers have signed up using Gmail email? Find % of customers signed up with Gmail. Gmail customers / all customers = % of gmail customers. Consider removing dummy created accounts 150K 150K / 1.3M = 12% What questions do you want to answer? What data do you want to look for? Where the data you want is available? How to use the data available? How to verify the data? Creating a mathematical model Customers data base
  • 28. We live in a dynamic world, so always consider things in a timely manner!
  • 29. Creating a mathematical model What questions do you want to answer? What data do you want to look for? Where the data you want is available? How to use the data available? How to verify the data? Will adding Gmail login improve sign up? How many customers have signed up using Gmail email in the past 6 months? Customers data base Find % of customers signed up with Gmail in the past 6 months. Gmail sign ups / all sign ups in the past 6 months = % of gmail customers. Consider removing dummy created accounts 120K 120K / 600K = 25%
  • 30. 1- Observations 2- Hypothesis 3- Experiment 4- Refine 5- Validate The Approach: Step by Step Process Questions to answer. Observing a trend or a problem. What is a hypothesis? How to build one? & Good vs Bad hypothesis. How to test hypothesis? What data to look at? Refine hypothesis based on test. How to measure the effect? Success metrics!
  • 31. Validate How to measure the effect? Success metrics!
  • 32. What is a success metric? Metrics are a lens into your product’s health and performance.
  • 35. 1- Observations 2- Hypothesis 3- Experiment 4- Refine 5- Validate The Approach: Step by Step Process Questions to answer. Observing a trend or a problem. What is a hypothesis? How to build one? & Good vs Bad hypothesis. How to test hypothesis? What data to look at? Refine hypothesis. How to measure the effect? Success metrics!
  • 36. Thank you Abdallah Al-Khalidi Linkedin: abzkhaldi | Twitter: @abzkhaldi Email: abdallah.khalidi@gmail.com