Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
GOOD

HYPOTHESIS

TESTING
IS SURPRISING

Timothy Fitz (.com)
Timothy Fitz
Coined Continuous Deployment
Technical Lead at IMVU
CTO at Canvas
Software Consultant
TimothyFitz.com
Systems...
WHICH SHADE OF BLUE?
Buy Buy Buy
Buy Buy Buy
Buy Buy Buy
Buy Buy Buy
MULTI-ARMED BANDIT
GOOD HYPOTHESIS
• Documented
• Testable
• Actionable
• Interesting
• Surprising
DOCUMENTED
NCSU Libraries Mobile Redesign for 2012
Objective
The redesign of the NCSU Libraries Mobile website will improv...
DOCUMENTED
Hypothesis
Students want to use their phones to track their
outstanding library books.
The details really matter.
Tiny code changes can change
results dramatically.
TESTABLE
• Are you collecting the data?
• Can you collect and analyze it quickly?
• Is it statistically significant?
HIGHTEMPOTESTING
3 experiments per week!
Hypothesis:
Adding Group Chat (a 12-month effort) will result in
a 25% increase in 3-month retention.
Hypothesis: Users want to chat with strangers.
Hypothesis: Users want to chat with strangers
about specific topics.



Hypo...
Hypothesis: Users want to chat with strangers.
Hypothesis: Users want to chat with strangers
about specific topics.



Hypo...
Hypothesis: Users want to chat with strangers.
Hypothesis: Users want to chat with strangers
about specific topics.



Hypo...
Hypothesis: Users want to chat with strangers.
Hypothesis: Users want to chat with strangers
about specific topics.



Hypo...
ACTIONABLE
The results of testing your
hypothesis cause action.
ACTIONABLE
Hypothesis: Adding a new payment method will
increase overall revenue by 5%
• If confirmed, we will keep the pay...
INTERESTING
The action you take is
important and

dependent on the outcome.
Hypothesis: Adding a new payment method will
increase overall revenue by 5%
• If confirmed, we will keep the payment method...
GOOD PRODUCT MANAGERS
Can rationalize ANY action
from ANY outcome.

:( :( :(
Hypothesis: Adding a new payment method will
increase overall revenue by 2%
• If confirmed, we will keep the payment method...
SURPRISING
Bayesian definition of surprise:





“The total number of wows experienced when
simultaneously considering all ...
SURPRISING
surprise
=
change in beliefs
SURPRISING
The results of your hypothesis tests
should frequently and significantly
change your beliefs about your product!
Don’t let a single hypothesis
test invalidate

your whole vision.
GOOD HYPOTHESIS
• Documented
• Testable
• Actionable
• Interesting
• Surprising
Upcoming SlideShare
Loading in …5
×

of

Good hypothesis testing is surprising! Slide 1 Good hypothesis testing is surprising! Slide 2 Good hypothesis testing is surprising! Slide 3 Good hypothesis testing is surprising! Slide 4 Good hypothesis testing is surprising! Slide 5 Good hypothesis testing is surprising! Slide 6 Good hypothesis testing is surprising! Slide 7 Good hypothesis testing is surprising! Slide 8 Good hypothesis testing is surprising! Slide 9 Good hypothesis testing is surprising! Slide 10 Good hypothesis testing is surprising! Slide 11 Good hypothesis testing is surprising! Slide 12 Good hypothesis testing is surprising! Slide 13 Good hypothesis testing is surprising! Slide 14 Good hypothesis testing is surprising! Slide 15 Good hypothesis testing is surprising! Slide 16 Good hypothesis testing is surprising! Slide 17 Good hypothesis testing is surprising! Slide 18 Good hypothesis testing is surprising! Slide 19 Good hypothesis testing is surprising! Slide 20 Good hypothesis testing is surprising! Slide 21 Good hypothesis testing is surprising! Slide 22 Good hypothesis testing is surprising! Slide 23 Good hypothesis testing is surprising! Slide 24 Good hypothesis testing is surprising! Slide 25 Good hypothesis testing is surprising! Slide 26
Upcoming SlideShare
Identifying Variables
Next
Download to read offline and view in fullscreen.

3 Likes

Share

Download to read offline

Good hypothesis testing is surprising!

Download to read offline

Lean Software talk at Agile India 2016

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Good hypothesis testing is surprising!

  1. 1. GOOD
 HYPOTHESIS
 TESTING IS SURPRISING
 Timothy Fitz (.com)
  2. 2. Timothy Fitz Coined Continuous Deployment Technical Lead at IMVU CTO at Canvas Software Consultant TimothyFitz.com SystemsLive.com @TimothyFitz
  3. 3. WHICH SHADE OF BLUE? Buy Buy Buy Buy Buy Buy Buy Buy Buy Buy Buy Buy
  4. 4. MULTI-ARMED BANDIT
  5. 5. GOOD HYPOTHESIS • Documented • Testable • Actionable • Interesting • Surprising
  6. 6. DOCUMENTED NCSU Libraries Mobile Redesign for 2012 Objective The redesign of the NCSU Libraries Mobile website will improve the extensibility of the service and introduce several user experience enhancements to the existing set of mobile library services. Requirements ● Redesign the homepage interface in a way that enables deployment of new apps with minimal effort ● Migrate mobile dev framework from forked MIT Mobile Web framework to jQuery Mobile framework ● Implement editorial recommendations on app names, taglines, and graphic identity; update the color scheme to be consistent with the NCSU Libraries website ● Perform a comprehensive review of the usage stats; use this information to remove underutilized services ● Maintain or improve usage data collection; improve usage data reporting tools ● Retire legacy MoboiLIB service Out of Scope ● Development of brand new apps ● Deep redesign of existing apps ● Custom iPad mobile website ● Native app deployment (iOS/Android) Team Project Manager: Tito Sierra UI/UX Lead: Adam Rogers Tech Lead: Cory Lown Schedule Launch: 2012.12.08
  7. 7. DOCUMENTED Hypothesis Students want to use their phones to track their outstanding library books.
  8. 8. The details really matter. Tiny code changes can change results dramatically.
  9. 9. TESTABLE • Are you collecting the data? • Can you collect and analyze it quickly? • Is it statistically significant?
  10. 10. HIGHTEMPOTESTING 3 experiments per week!
  11. 11. Hypothesis: Adding Group Chat (a 12-month effort) will result in a 25% increase in 3-month retention.
  12. 12. Hypothesis: Users want to chat with strangers. Hypothesis: Users want to chat with strangers about specific topics.
 
 Hypothesis: Users want a directory of group chats. Hypothesis: Users who group chat will
 have higher retention.
  13. 13. Hypothesis: Users want to chat with strangers. Hypothesis: Users want to chat with strangers about specific topics.
 
 Hypothesis: Users want a directory of group chats. Hypothesis: Users who group chat will
 have higher retention. ✓
  14. 14. Hypothesis: Users want to chat with strangers. Hypothesis: Users want to chat with strangers about specific topics.
 
 Hypothesis: Users want a directory of group chats. Hypothesis: Users who group chat will
 have higher retention. ✓
  15. 15. Hypothesis: Users want to chat with strangers. Hypothesis: Users want to chat with strangers about specific topics.
 
 Hypothesis: Users want a directory of group chats. Hypothesis: Users who group chat will
 have higher retention.
  16. 16. ACTIONABLE The results of testing your hypothesis cause action.
  17. 17. ACTIONABLE Hypothesis: Adding a new payment method will increase overall revenue by 5% • If confirmed, we will keep the payment method
  18. 18. INTERESTING The action you take is important and
 dependent on the outcome.
  19. 19. Hypothesis: Adding a new payment method will increase overall revenue by 5% • If confirmed, we will keep the payment method • If denied, we will keep the payment method INTERESTING+DOCUMENTED
  20. 20. GOOD PRODUCT MANAGERS Can rationalize ANY action from ANY outcome.
 :( :( :(
  21. 21. Hypothesis: Adding a new payment method will increase overall revenue by 2% • If confirmed, we will keep the payment method • If denied, we will DELETE the payment method INTERESTING+DOCUMENTED
  22. 22. SURPRISING Bayesian definition of surprise:
 
 
 “The total number of wows experienced when simultaneously considering all models is obtained through the integration over the model class.” http://ilab.usc.edu/surprise/
  23. 23. SURPRISING surprise = change in beliefs
  24. 24. SURPRISING The results of your hypothesis tests should frequently and significantly change your beliefs about your product!
  25. 25. Don’t let a single hypothesis test invalidate
 your whole vision.
  26. 26. GOOD HYPOTHESIS • Documented • Testable • Actionable • Interesting • Surprising
  • powerirs

    Jan. 28, 2021
  • sagittatius

    Jun. 26, 2019
  • cyetain

    Mar. 20, 2016

Lean Software talk at Agile India 2016

Views

Total views

986

On Slideshare

0

From embeds

0

Number of embeds

70

Actions

Downloads

9

Shares

0

Comments

0

Likes

3

×