Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Good hypothesis testing is surprising!

702 views

Published on

Lean Software talk at Agile India 2016

Published in: Software
  • Be the first to comment

Good hypothesis testing is surprising!

  1. 1. GOOD
 HYPOTHESIS
 TESTING IS SURPRISING
 Timothy Fitz (.com)
  2. 2. Timothy Fitz Coined Continuous Deployment Technical Lead at IMVU CTO at Canvas Software Consultant TimothyFitz.com SystemsLive.com @TimothyFitz
  3. 3. WHICH SHADE OF BLUE? Buy Buy Buy Buy Buy Buy Buy Buy Buy Buy Buy Buy
  4. 4. MULTI-ARMED BANDIT
  5. 5. GOOD HYPOTHESIS • Documented • Testable • Actionable • Interesting • Surprising
  6. 6. DOCUMENTED NCSU Libraries Mobile Redesign for 2012 Objective The redesign of the NCSU Libraries Mobile website will improve the extensibility of the service and introduce several user experience enhancements to the existing set of mobile library services. Requirements ● Redesign the homepage interface in a way that enables deployment of new apps with minimal effort ● Migrate mobile dev framework from forked MIT Mobile Web framework to jQuery Mobile framework ● Implement editorial recommendations on app names, taglines, and graphic identity; update the color scheme to be consistent with the NCSU Libraries website ● Perform a comprehensive review of the usage stats; use this information to remove underutilized services ● Maintain or improve usage data collection; improve usage data reporting tools ● Retire legacy MoboiLIB service Out of Scope ● Development of brand new apps ● Deep redesign of existing apps ● Custom iPad mobile website ● Native app deployment (iOS/Android) Team Project Manager: Tito Sierra UI/UX Lead: Adam Rogers Tech Lead: Cory Lown Schedule Launch: 2012.12.08
  7. 7. DOCUMENTED Hypothesis Students want to use their phones to track their outstanding library books.
  8. 8. The details really matter. Tiny code changes can change results dramatically.
  9. 9. TESTABLE • Are you collecting the data? • Can you collect and analyze it quickly? • Is it statistically significant?
  10. 10. HIGHTEMPOTESTING 3 experiments per week!
  11. 11. Hypothesis: Adding Group Chat (a 12-month effort) will result in a 25% increase in 3-month retention.
  12. 12. Hypothesis: Users want to chat with strangers. Hypothesis: Users want to chat with strangers about specific topics.
 
 Hypothesis: Users want a directory of group chats. Hypothesis: Users who group chat will
 have higher retention.
  13. 13. Hypothesis: Users want to chat with strangers. Hypothesis: Users want to chat with strangers about specific topics.
 
 Hypothesis: Users want a directory of group chats. Hypothesis: Users who group chat will
 have higher retention. ✓
  14. 14. Hypothesis: Users want to chat with strangers. Hypothesis: Users want to chat with strangers about specific topics.
 
 Hypothesis: Users want a directory of group chats. Hypothesis: Users who group chat will
 have higher retention. ✓
  15. 15. Hypothesis: Users want to chat with strangers. Hypothesis: Users want to chat with strangers about specific topics.
 
 Hypothesis: Users want a directory of group chats. Hypothesis: Users who group chat will
 have higher retention.
  16. 16. ACTIONABLE The results of testing your hypothesis cause action.
  17. 17. ACTIONABLE Hypothesis: Adding a new payment method will increase overall revenue by 5% • If confirmed, we will keep the payment method
  18. 18. INTERESTING The action you take is important and
 dependent on the outcome.
  19. 19. Hypothesis: Adding a new payment method will increase overall revenue by 5% • If confirmed, we will keep the payment method • If denied, we will keep the payment method INTERESTING+DOCUMENTED
  20. 20. GOOD PRODUCT MANAGERS Can rationalize ANY action from ANY outcome.
 :( :( :(
  21. 21. Hypothesis: Adding a new payment method will increase overall revenue by 2% • If confirmed, we will keep the payment method • If denied, we will DELETE the payment method INTERESTING+DOCUMENTED
  22. 22. SURPRISING Bayesian definition of surprise:
 
 
 “The total number of wows experienced when simultaneously considering all models is obtained through the integration over the model class.” http://ilab.usc.edu/surprise/
  23. 23. SURPRISING surprise = change in beliefs
  24. 24. SURPRISING The results of your hypothesis tests should frequently and significantly change your beliefs about your product!
  25. 25. Don’t let a single hypothesis test invalidate
 your whole vision.
  26. 26. GOOD HYPOTHESIS • Documented • Testable • Actionable • Interesting • Surprising

×