SlideShare a Scribd company logo
Hypothesize like you
mean it
The Essentials of Metrics, Hypotheses, and Iteration
Greetings :)
Chris Massey
Product lead @ Mind the Product
chris@mindtheproduct.com
@camassey
Lisa Long
Founder @ Before You Code
lisa@beforeyoucode.com
@beforeyoucode
Learning outcomes
Metrics &
Hypotheses &
Prioritisation &
Iteration
● What makes a good metric
● How to design an effective
hypothesis
● The essentials of
Prioritisation
● Experimental Iteration
Let’s go!
What is a Metric? Meaningful metrics
(data+context) drive targeted,
effective actions that create
tangible value
Good metric /
Bad metric
Does your metric suggest
success without being directly
tied to the outcome you care
about?
Is your metric a placeholder for
something that is hard to
measure?
What is a
hypothesis?
A statement of the
specific (& testable)
impact you believe a
proposed change will
have.
The purpose of a
hypothesis
To structure thinking
To reveal assumptions
To direct resources
To avoid waste
To help you prioritise
Null Hypotheses Assume your worldview wrong
!
High vs Low-level
Hypotheses
Understand your chain of value
Make sure you can isolate your
variables / metrics
Pick one
Designers & Dragons
Goals
HIGH-LEVEL MISSION,
VISION OR OBJECTIVES
DESIGN PARAMETERS
Metrics
Framing Context
Features
SPECIFIC, MEASURABLE,
CONTEXTUALISED ACTION-
ORIENTED
ENVIRONMENT, AUDIENCE,
TIMEFRAMES ...
HIGH-LEVEL FEATURE
CONCEPTS, SPECIFIC
IMPLEMENTATIONS ...
Goals
● Tourist attraction
● More awareness of
Cambridge history /
significance
DESIGN PARAMETERS
Metrics
Framing Context
Features
● Tour company inclusion
● Pre-test / Post-test of
local history
● Traffic-flow assessment
● Social media buzz
ENVIRONMENT, AUDIENCE,
TIMEFRAMES ...
HIGH-LEVEL FEATURE
CONCEPTS, SPECIFIC
IMPLEMENTATIONS ...
Instant Art Generator ™
1. Kinetic
2. Projection
3. Light-based
4. Physical interaction
5. Multiple pieces
6. Historical
7. Phone interaction
8. Large scale
9. Persistent changes
10. Consistent start state
11. All-weather
12. University collaboration
Go design
Activity: Design a civic art installation
15 mins
Goals
Brainstorm a potential design for an informative civic art installation
Activity
● Use google to get inspiration for art installations that use your
constraints
● Consider what historically significant or helpful facts about
Cambridge could be represented in a public installation
● Come up with as many ways as possible to represent those facts
● Converge on a set of design “features” for your installation
Deliverable
An informative civic art installation, described in “features”, potentially
supported by rough sketches
What’s your riskiest
assumption?
To the hypothesis canvas!
We believe that ... For ...
Will lead to ... Because...
SOME FEATURE OR CHANGE AN ENVIRONMENT OR CONTEXT
SOME SPECIFIC,
MEASURABLE CHANGE
YOUR RATIONALE
HYPOTHESIS CANVAS
Go
Hypothesise
Goals
Form a testable hypothesis around one aspect of your art concept
Activity
● Pick the metric you want to effect
● Pick your riskiest assumption / feature
● Select your experimental context
● Think through your experimental setup
● Articulate your experimental rationale
Deliverable
A well-formed, testable hypothesis, allowing you remove uncertainty /
validate an assumption around one “feature” of your design product
Activity: Form a hypothesis
10 mins
How’d it go?
Designers & Diagnosis
& Dragons
Some Role-play Heads = Failure
Tails = Ambiguous
What impacts
experimental
outcomes?
● Bias
● Confounding factors
● Chance!
● True effect
Common mistakes
● Not everything deserves a hypothesis
● When to abandon (sunk cost)
● Check sample size & source
○ (67% conversion on 3 actions out of 10 million users)
● Are you asking the right question?
● Is every test succeeding / failing?
○ (check for bias or poor systematic design)
● Bad data foundations
● Technical challenges
What
happened?
Activity: Diagnose errors in your experiment
10 mins
Goals
Brainstorm a wide range of possible ways your experiment might have
yielded an ambiguous or unsuccessful result
Activity
● Revisit & reassess your underlying assumptions
● List any ways your assumptions could be wrong
○ Don’t underestimate human stupidity
● Step through your experiment setup
● List the possible errors in that setup
Deliverable
A list of possible incorrect assumptions and experimental errors
NOW what’s your riskiest
assumption?
Let’s talk about
prioritisation
● Go back to your metrics
● What are your assumptions?
● What are you uncertain about?
● What are the risks?
● What are the costs to test?
Uncertainty
Risk
TRACK TEST
IGNORE MITIGATE
Low High
High
Risk
Cost
TEST ASSESS
ASSESS IGNORE
Low High
High
What’s the next Hypothesis?
We believe that ... For ...
Will lead to ... Because...
SOME FEATURE OR CHANGE AN ENVIRONMENT OR CONTEXT
SOME SPECIFIC,
MEASURABLE CHANGE
YOUR RATIONALE
HYPOTHESIS CANVAS
Goals
Create a better-informed, testable hypothesis around one aspect of your art
concept
Activity
● Consider what you’ve learned thus far about your context
● Pick the metric you want to effect
● Pick your riskiest assumption / feature
● Select your experimental context
● Think through your experimental setup
● Articulate your experimental rationale
Deliverable
A well-formed, better-informed, testable hypothesis, allowing you remove
uncertainty / validate an assumption around one “feature” of your design product
Activity: Form a new hypothesis
5 mins
What
questions do
you have?
What did we learn?
Metrics &
Hypotheses &
Prioritisation &
Iteration
● What makes a good metric
● How to design an effective
hypothesis
● The essentials of
Prioritisation
● Experimental Iteration
Further Reading
http://experimentationhub.com/hypothesis-kit.html
+
Thanks! Chris Massey & Lisa Long
@camassey | @beforeyoucode
chris@mindtheproduct.com
lisa@beforeyoucode.com
Bonus round
How did it go this time?
Some more role-play Rock = Success
Paper = Failure
Scissors = Ambiguous
Presentation to Design Council
… in one week
… to secure more £££
What do you do?
How to present
experimental
findings
Context, Context, Context
Your sources
Your sample size
Your assumptions
Your experiment design
Your findings

More Related Content

What's hot

A basic course on analytical thinking
A basic course on analytical thinking A basic course on analytical thinking
A basic course on analytical thinking
Vincent G Valentyn PhD
 
Cause and effect diagram
Cause and effect diagramCause and effect diagram
Cause and effect diagram
COEPD HR
 
March 03 decision making
March 03   decision makingMarch 03   decision making
March 03 decision making
Eben Schaeffer
 
Blind mountain climbing: design process
Blind mountain climbing: design processBlind mountain climbing: design process
Blind mountain climbing: design process
Nathan Kane
 
Analytical skills training course – make valid decisions with maximum confidence
Analytical skills training course – make valid decisions with maximum confidenceAnalytical skills training course – make valid decisions with maximum confidence
Analytical skills training course – make valid decisions with maximum confidence
Skilldom
 
5 why analysis
5 why analysis5 why analysis
5 why analysis
Saurabh Kumar
 
Cause and effect diagram
Cause and effect diagramCause and effect diagram
Cause and effect diagram
Lizzette Danan
 
Heuristics for Becoming a Learning Organisation
Heuristics for Becoming a Learning OrganisationHeuristics for Becoming a Learning Organisation
Heuristics for Becoming a Learning Organisation
Karl Scotland
 
Analytical Thinking Training
Analytical Thinking TrainingAnalytical Thinking Training
Analytical Thinking Training
M Furqan Aslam
 
Problem solving and analytical skills
Problem solving and analytical skillsProblem solving and analytical skills
Problem solving and analytical skills
tayyabaways
 
The Seven Problem Solving Steps
The Seven Problem Solving StepsThe Seven Problem Solving Steps
The Seven Problem Solving Steps
Deborah_W
 
Designing work
Designing workDesigning work
Designing work
farzanashoma
 
Ergonaute presentation on Structured Ideation Toolbox at Silicon Halton
Ergonaute presentation on Structured Ideation Toolbox at Silicon HaltonErgonaute presentation on Structured Ideation Toolbox at Silicon Halton
Ergonaute presentation on Structured Ideation Toolbox at Silicon Halton
Silicon Halton
 
Design process powerpoint
Design process powerpointDesign process powerpoint
Design process powerpoint
rhiannon_folan
 
Analytical thinking training
Analytical thinking trainingAnalytical thinking training
Analytical thinking training
ras1215
 
Stanford Peace Innovation Lab: ITBA Argentina workshop day 2
Stanford Peace Innovation Lab: ITBA Argentina workshop day 2Stanford Peace Innovation Lab: ITBA Argentina workshop day 2
Stanford Peace Innovation Lab: ITBA Argentina workshop day 2
Margarita Quihuis
 
5 why’s technique and cause and effect analysis
5 why’s technique and cause and effect analysis5 why’s technique and cause and effect analysis
5 why’s technique and cause and effect analysis
Bhagya Silva
 
Hsu2 engdesignprocessbv
Hsu2 engdesignprocessbvHsu2 engdesignprocessbv
Hsu2 engdesignprocessbv
Kawaldeep Singh
 
Fishbone diagam guide
Fishbone diagam guideFishbone diagam guide
Fishbone diagam guide
丹 丹
 
Искусство задавать правильные вопросы, Наталья Рыбник
Искусство задавать правильные вопросы, Наталья РыбникИскусство задавать правильные вопросы, Наталья Рыбник
Искусство задавать правильные вопросы, Наталья Рыбник
Sigma Software
 

What's hot (20)

A basic course on analytical thinking
A basic course on analytical thinking A basic course on analytical thinking
A basic course on analytical thinking
 
Cause and effect diagram
Cause and effect diagramCause and effect diagram
Cause and effect diagram
 
March 03 decision making
March 03   decision makingMarch 03   decision making
March 03 decision making
 
Blind mountain climbing: design process
Blind mountain climbing: design processBlind mountain climbing: design process
Blind mountain climbing: design process
 
Analytical skills training course – make valid decisions with maximum confidence
Analytical skills training course – make valid decisions with maximum confidenceAnalytical skills training course – make valid decisions with maximum confidence
Analytical skills training course – make valid decisions with maximum confidence
 
5 why analysis
5 why analysis5 why analysis
5 why analysis
 
Cause and effect diagram
Cause and effect diagramCause and effect diagram
Cause and effect diagram
 
Heuristics for Becoming a Learning Organisation
Heuristics for Becoming a Learning OrganisationHeuristics for Becoming a Learning Organisation
Heuristics for Becoming a Learning Organisation
 
Analytical Thinking Training
Analytical Thinking TrainingAnalytical Thinking Training
Analytical Thinking Training
 
Problem solving and analytical skills
Problem solving and analytical skillsProblem solving and analytical skills
Problem solving and analytical skills
 
The Seven Problem Solving Steps
The Seven Problem Solving StepsThe Seven Problem Solving Steps
The Seven Problem Solving Steps
 
Designing work
Designing workDesigning work
Designing work
 
Ergonaute presentation on Structured Ideation Toolbox at Silicon Halton
Ergonaute presentation on Structured Ideation Toolbox at Silicon HaltonErgonaute presentation on Structured Ideation Toolbox at Silicon Halton
Ergonaute presentation on Structured Ideation Toolbox at Silicon Halton
 
Design process powerpoint
Design process powerpointDesign process powerpoint
Design process powerpoint
 
Analytical thinking training
Analytical thinking trainingAnalytical thinking training
Analytical thinking training
 
Stanford Peace Innovation Lab: ITBA Argentina workshop day 2
Stanford Peace Innovation Lab: ITBA Argentina workshop day 2Stanford Peace Innovation Lab: ITBA Argentina workshop day 2
Stanford Peace Innovation Lab: ITBA Argentina workshop day 2
 
5 why’s technique and cause and effect analysis
5 why’s technique and cause and effect analysis5 why’s technique and cause and effect analysis
5 why’s technique and cause and effect analysis
 
Hsu2 engdesignprocessbv
Hsu2 engdesignprocessbvHsu2 engdesignprocessbv
Hsu2 engdesignprocessbv
 
Fishbone diagam guide
Fishbone diagam guideFishbone diagam guide
Fishbone diagam guide
 
Искусство задавать правильные вопросы, Наталья Рыбник
Искусство задавать правильные вопросы, Наталья РыбникИскусство задавать правильные вопросы, Наталья Рыбник
Искусство задавать правильные вопросы, Наталья Рыбник
 

Similar to Hypothesise like you Mean it!

Lean UX workshop - Part Two
Lean UX workshop - Part TwoLean UX workshop - Part Two
Lean UX workshop - Part Two
Christopher Barklem
 
Eng design 4
Eng design 4Eng design 4
Eng design 4
Masimbaashe Musekiwa
 
The Lean Startup
The Lean StartupThe Lean Startup
The Lean Startup
Naoise
 
The Leader's Guide Workshop - Pivotal Labs Tokyo
The Leader's Guide Workshop - Pivotal Labs TokyoThe Leader's Guide Workshop - Pivotal Labs Tokyo
The Leader's Guide Workshop - Pivotal Labs Tokyo
Jeana Alayaay
 
Behind The Scenes Data Science Coolblue 2018-03-22
Behind The Scenes Data Science Coolblue 2018-03-22Behind The Scenes Data Science Coolblue 2018-03-22
Behind The Scenes Data Science Coolblue 2018-03-22
Matthias Schuurmans
 
Lean UX principles
Lean UX principlesLean UX principles
Lean UX principles
Tristan Libersat
 
Application of analytics
Application of analyticsApplication of analytics
Application of analytics
Ravi Kumar Peela
 
How to Create Fun User Experience by Shutterstock Dir of Product
How to Create Fun User Experience by Shutterstock Dir of ProductHow to Create Fun User Experience by Shutterstock Dir of Product
How to Create Fun User Experience by Shutterstock Dir of Product
Product School
 
The Art of A/B Testing in Product by YogaGlo VP of Product
The Art of A/B Testing in Product by YogaGlo VP of ProductThe Art of A/B Testing in Product by YogaGlo VP of Product
The Art of A/B Testing in Product by YogaGlo VP of Product
Product School
 
Come flying on Divergence Airways with Mike Biggs -"We always land"
Come flying on Divergence Airways with Mike Biggs -"We always land"Come flying on Divergence Airways with Mike Biggs -"We always land"
Come flying on Divergence Airways with Mike Biggs -"We always land"
Mike Biggs GAICD
 
The #NoEstimates Debate
The #NoEstimates DebateThe #NoEstimates Debate
The #NoEstimates Debate
Killick Agile Consulting Services
 
Divergence Airways UPDATED
Divergence Airways UPDATEDDivergence Airways UPDATED
Divergence Airways UPDATED
Mike Biggs GAICD
 
How to Pitch a Lean Startup Idea
How to Pitch a Lean Startup IdeaHow to Pitch a Lean Startup Idea
How to Pitch a Lean Startup Idea
BenBlank
 
Dowhy: An end-to-end library for causal inference
Dowhy: An end-to-end library for causal inferenceDowhy: An end-to-end library for causal inference
Dowhy: An end-to-end library for causal inference
Amit Sharma
 
#ImpactSalesforceSaturday: Drum into understanding of prediction builder with...
#ImpactSalesforceSaturday: Drum into understanding of prediction builder with...#ImpactSalesforceSaturday: Drum into understanding of prediction builder with...
#ImpactSalesforceSaturday: Drum into understanding of prediction builder with...
New Delhi Salesforce Developer Group
 
How might we make reviewing high school work easier through visualizations?
How might we make reviewing high school work easier through visualizations?How might we make reviewing high school work easier through visualizations?
How might we make reviewing high school work easier through visualizations?
Manu Gupta
 
Formulate stronger hypotheses
Formulate stronger hypothesesFormulate stronger hypotheses
Formulate stronger hypotheses
Carmen Brion
 
Experimentation for PMs: A Primer by Amazon Director of Product
Experimentation for PMs: A Primer by Amazon Director of ProductExperimentation for PMs: A Primer by Amazon Director of Product
Experimentation for PMs: A Primer by Amazon Director of Product
Product School
 
[DevDay2019] Lean UX - By Bryant Castro, Bryant Castro at Wizeline
[DevDay2019] Lean UX - By  Bryant Castro,  Bryant Castro at Wizeline[DevDay2019] Lean UX - By  Bryant Castro,  Bryant Castro at Wizeline
[DevDay2019] Lean UX - By Bryant Castro, Bryant Castro at Wizeline
DevDay.org
 
Estimation tricks and traps
Estimation tricks and trapsEstimation tricks and traps
Estimation tricks and traps
Marta Kossowska
 

Similar to Hypothesise like you Mean it! (20)

Lean UX workshop - Part Two
Lean UX workshop - Part TwoLean UX workshop - Part Two
Lean UX workshop - Part Two
 
Eng design 4
Eng design 4Eng design 4
Eng design 4
 
The Lean Startup
The Lean StartupThe Lean Startup
The Lean Startup
 
The Leader's Guide Workshop - Pivotal Labs Tokyo
The Leader's Guide Workshop - Pivotal Labs TokyoThe Leader's Guide Workshop - Pivotal Labs Tokyo
The Leader's Guide Workshop - Pivotal Labs Tokyo
 
Behind The Scenes Data Science Coolblue 2018-03-22
Behind The Scenes Data Science Coolblue 2018-03-22Behind The Scenes Data Science Coolblue 2018-03-22
Behind The Scenes Data Science Coolblue 2018-03-22
 
Lean UX principles
Lean UX principlesLean UX principles
Lean UX principles
 
Application of analytics
Application of analyticsApplication of analytics
Application of analytics
 
How to Create Fun User Experience by Shutterstock Dir of Product
How to Create Fun User Experience by Shutterstock Dir of ProductHow to Create Fun User Experience by Shutterstock Dir of Product
How to Create Fun User Experience by Shutterstock Dir of Product
 
The Art of A/B Testing in Product by YogaGlo VP of Product
The Art of A/B Testing in Product by YogaGlo VP of ProductThe Art of A/B Testing in Product by YogaGlo VP of Product
The Art of A/B Testing in Product by YogaGlo VP of Product
 
Come flying on Divergence Airways with Mike Biggs -"We always land"
Come flying on Divergence Airways with Mike Biggs -"We always land"Come flying on Divergence Airways with Mike Biggs -"We always land"
Come flying on Divergence Airways with Mike Biggs -"We always land"
 
The #NoEstimates Debate
The #NoEstimates DebateThe #NoEstimates Debate
The #NoEstimates Debate
 
Divergence Airways UPDATED
Divergence Airways UPDATEDDivergence Airways UPDATED
Divergence Airways UPDATED
 
How to Pitch a Lean Startup Idea
How to Pitch a Lean Startup IdeaHow to Pitch a Lean Startup Idea
How to Pitch a Lean Startup Idea
 
Dowhy: An end-to-end library for causal inference
Dowhy: An end-to-end library for causal inferenceDowhy: An end-to-end library for causal inference
Dowhy: An end-to-end library for causal inference
 
#ImpactSalesforceSaturday: Drum into understanding of prediction builder with...
#ImpactSalesforceSaturday: Drum into understanding of prediction builder with...#ImpactSalesforceSaturday: Drum into understanding of prediction builder with...
#ImpactSalesforceSaturday: Drum into understanding of prediction builder with...
 
How might we make reviewing high school work easier through visualizations?
How might we make reviewing high school work easier through visualizations?How might we make reviewing high school work easier through visualizations?
How might we make reviewing high school work easier through visualizations?
 
Formulate stronger hypotheses
Formulate stronger hypothesesFormulate stronger hypotheses
Formulate stronger hypotheses
 
Experimentation for PMs: A Primer by Amazon Director of Product
Experimentation for PMs: A Primer by Amazon Director of ProductExperimentation for PMs: A Primer by Amazon Director of Product
Experimentation for PMs: A Primer by Amazon Director of Product
 
[DevDay2019] Lean UX - By Bryant Castro, Bryant Castro at Wizeline
[DevDay2019] Lean UX - By  Bryant Castro,  Bryant Castro at Wizeline[DevDay2019] Lean UX - By  Bryant Castro,  Bryant Castro at Wizeline
[DevDay2019] Lean UX - By Bryant Castro, Bryant Castro at Wizeline
 
Estimation tricks and traps
Estimation tricks and trapsEstimation tricks and traps
Estimation tricks and traps
 

Recently uploaded

Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptxChoosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
Brandon Minnick, MBA
 
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
shyamraj55
 
5th LF Energy Power Grid Model Meet-up Slides
5th LF Energy Power Grid Model Meet-up Slides5th LF Energy Power Grid Model Meet-up Slides
5th LF Energy Power Grid Model Meet-up Slides
DanBrown980551
 
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdfMonitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Tosin Akinosho
 
UI5 Controls simplified - UI5con2024 presentation
UI5 Controls simplified - UI5con2024 presentationUI5 Controls simplified - UI5con2024 presentation
UI5 Controls simplified - UI5con2024 presentation
Wouter Lemaire
 
WeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation TechniquesWeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation Techniques
Postman
 
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAUHCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
panagenda
 
Taking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdfTaking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdf
ssuserfac0301
 
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
名前 です男
 
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and MilvusGenerating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
Zilliz
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
DianaGray10
 
June Patch Tuesday
June Patch TuesdayJune Patch Tuesday
June Patch Tuesday
Ivanti
 
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development ProvidersYour One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
akankshawande
 
Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)
Jakub Marek
 
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial IntelligenceAI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
IndexBug
 
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptxHow to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
danishmna97
 
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Jeffrey Haguewood
 
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practicesNational Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
Quotidiano Piemontese
 
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracyGraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
Tomaz Bratanic
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
Chart Kalyan
 

Recently uploaded (20)

Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptxChoosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
 
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
 
5th LF Energy Power Grid Model Meet-up Slides
5th LF Energy Power Grid Model Meet-up Slides5th LF Energy Power Grid Model Meet-up Slides
5th LF Energy Power Grid Model Meet-up Slides
 
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdfMonitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
 
UI5 Controls simplified - UI5con2024 presentation
UI5 Controls simplified - UI5con2024 presentationUI5 Controls simplified - UI5con2024 presentation
UI5 Controls simplified - UI5con2024 presentation
 
WeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation TechniquesWeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation Techniques
 
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAUHCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
 
Taking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdfTaking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdf
 
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
 
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and MilvusGenerating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
 
June Patch Tuesday
June Patch TuesdayJune Patch Tuesday
June Patch Tuesday
 
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development ProvidersYour One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
 
Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)
 
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial IntelligenceAI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
 
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptxHow to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
 
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
 
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practicesNational Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
 
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracyGraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
 

Hypothesise like you Mean it!

  • 1. Hypothesize like you mean it The Essentials of Metrics, Hypotheses, and Iteration
  • 3. Chris Massey Product lead @ Mind the Product chris@mindtheproduct.com @camassey
  • 4. Lisa Long Founder @ Before You Code lisa@beforeyoucode.com @beforeyoucode
  • 6. Metrics & Hypotheses & Prioritisation & Iteration ● What makes a good metric ● How to design an effective hypothesis ● The essentials of Prioritisation ● Experimental Iteration
  • 8. What is a Metric? Meaningful metrics (data+context) drive targeted, effective actions that create tangible value
  • 9. Good metric / Bad metric Does your metric suggest success without being directly tied to the outcome you care about? Is your metric a placeholder for something that is hard to measure?
  • 10. What is a hypothesis? A statement of the specific (& testable) impact you believe a proposed change will have.
  • 11. The purpose of a hypothesis To structure thinking To reveal assumptions To direct resources To avoid waste To help you prioritise
  • 12. Null Hypotheses Assume your worldview wrong !
  • 13. High vs Low-level Hypotheses Understand your chain of value Make sure you can isolate your variables / metrics Pick one
  • 15. Goals HIGH-LEVEL MISSION, VISION OR OBJECTIVES DESIGN PARAMETERS Metrics Framing Context Features SPECIFIC, MEASURABLE, CONTEXTUALISED ACTION- ORIENTED ENVIRONMENT, AUDIENCE, TIMEFRAMES ... HIGH-LEVEL FEATURE CONCEPTS, SPECIFIC IMPLEMENTATIONS ...
  • 16. Goals ● Tourist attraction ● More awareness of Cambridge history / significance DESIGN PARAMETERS Metrics Framing Context Features ● Tour company inclusion ● Pre-test / Post-test of local history ● Traffic-flow assessment ● Social media buzz ENVIRONMENT, AUDIENCE, TIMEFRAMES ... HIGH-LEVEL FEATURE CONCEPTS, SPECIFIC IMPLEMENTATIONS ...
  • 17. Instant Art Generator ™ 1. Kinetic 2. Projection 3. Light-based 4. Physical interaction 5. Multiple pieces 6. Historical 7. Phone interaction 8. Large scale 9. Persistent changes 10. Consistent start state 11. All-weather 12. University collaboration
  • 19. Activity: Design a civic art installation 15 mins Goals Brainstorm a potential design for an informative civic art installation Activity ● Use google to get inspiration for art installations that use your constraints ● Consider what historically significant or helpful facts about Cambridge could be represented in a public installation ● Come up with as many ways as possible to represent those facts ● Converge on a set of design “features” for your installation Deliverable An informative civic art installation, described in “features”, potentially supported by rough sketches
  • 21. To the hypothesis canvas!
  • 22. We believe that ... For ... Will lead to ... Because... SOME FEATURE OR CHANGE AN ENVIRONMENT OR CONTEXT SOME SPECIFIC, MEASURABLE CHANGE YOUR RATIONALE HYPOTHESIS CANVAS
  • 24. Goals Form a testable hypothesis around one aspect of your art concept Activity ● Pick the metric you want to effect ● Pick your riskiest assumption / feature ● Select your experimental context ● Think through your experimental setup ● Articulate your experimental rationale Deliverable A well-formed, testable hypothesis, allowing you remove uncertainty / validate an assumption around one “feature” of your design product Activity: Form a hypothesis 10 mins
  • 27. Some Role-play Heads = Failure Tails = Ambiguous
  • 28. What impacts experimental outcomes? ● Bias ● Confounding factors ● Chance! ● True effect
  • 29. Common mistakes ● Not everything deserves a hypothesis ● When to abandon (sunk cost) ● Check sample size & source ○ (67% conversion on 3 actions out of 10 million users) ● Are you asking the right question? ● Is every test succeeding / failing? ○ (check for bias or poor systematic design) ● Bad data foundations ● Technical challenges
  • 31. Activity: Diagnose errors in your experiment 10 mins Goals Brainstorm a wide range of possible ways your experiment might have yielded an ambiguous or unsuccessful result Activity ● Revisit & reassess your underlying assumptions ● List any ways your assumptions could be wrong ○ Don’t underestimate human stupidity ● Step through your experiment setup ● List the possible errors in that setup Deliverable A list of possible incorrect assumptions and experimental errors
  • 32. NOW what’s your riskiest assumption?
  • 33. Let’s talk about prioritisation ● Go back to your metrics ● What are your assumptions? ● What are you uncertain about? ● What are the risks? ● What are the costs to test?
  • 36. What’s the next Hypothesis?
  • 37. We believe that ... For ... Will lead to ... Because... SOME FEATURE OR CHANGE AN ENVIRONMENT OR CONTEXT SOME SPECIFIC, MEASURABLE CHANGE YOUR RATIONALE HYPOTHESIS CANVAS
  • 38. Goals Create a better-informed, testable hypothesis around one aspect of your art concept Activity ● Consider what you’ve learned thus far about your context ● Pick the metric you want to effect ● Pick your riskiest assumption / feature ● Select your experimental context ● Think through your experimental setup ● Articulate your experimental rationale Deliverable A well-formed, better-informed, testable hypothesis, allowing you remove uncertainty / validate an assumption around one “feature” of your design product Activity: Form a new hypothesis 5 mins
  • 40. What did we learn?
  • 41. Metrics & Hypotheses & Prioritisation & Iteration ● What makes a good metric ● How to design an effective hypothesis ● The essentials of Prioritisation ● Experimental Iteration
  • 43. Thanks! Chris Massey & Lisa Long @camassey | @beforeyoucode chris@mindtheproduct.com lisa@beforeyoucode.com
  • 45. How did it go this time?
  • 46. Some more role-play Rock = Success Paper = Failure Scissors = Ambiguous
  • 47. Presentation to Design Council … in one week … to secure more £££
  • 48. What do you do?
  • 49. How to present experimental findings Context, Context, Context Your sources Your sample size Your assumptions Your experiment design Your findings

Editor's Notes

  1. 50:50 - activities to talking 10-15 minutes of lecture (10m without interactivity), then 15-20m activity