Learning Fast With A/B Testing and Continuous Deployment

2,452 views

Published on

IMVU's use of A/B testing combined with Continuous Deployment to quickly identify and respond to opportunities.

Published in: Business, Technology, Education

Learning Fast With A/B Testing and Continuous Deployment

  1. 1. REAL TIME LEARNING USING A/B TESTING ANDCONTINUOUS DEPLOYMENT Brett G. Durrett,VP Engineering & Operations, IMVU - @bdurrett LOGIN Conference, 2011 1
  2. 2. DEVELOP RELEASE PRAY FOR CUSTOMERS 2
  3. 3. DEVELOP RELEASE ITERATE LEARN 3
  4. 4. “Casual Players Evolving Fast”Richard GarriottKeynote“Only 4% of games that go into production are profitable”Scott DodsonGamification Will Eat Itself: How Gamification Can Evolve, And Why It Must.“Most people are not willing to bet their business oninstant perfection”Rajat KongoviStop wasting your data! Using data to drive performance.. 4
  5. 5. 5
  6. 6. UH O HNO PROBLEM 6
  7. 7. * The Pepsi challenge was not A/B testing 7
  8. 8. http://www.abtests.com/test/278002/landing-for-dhl-express 8
  9. 9. VALIDATES PRODUCT DECISIONS IDENTIFIES OPPORTUNITIES GIVES FREEDOM TO EXPERIMENTWILL NOT DESIGN YOUR PRODUCT! 9
  10. 10. 1-2-3 EASY TESTING 1. EASY TO CODE2. EASY TO ADMINISTER 3. EASY TO READ RESULTS
  11. 11. $experiment = array( ‘name’ -> ‘DemoOnline2011’, initial_rollout => 0, branches => array( Enhanced => 0.5, // can specify multiple weighted branches ) );// Helper function for readabilityfunction user_should_see_test_feature( $uid = NULL ) { return get_branch_for_customer( $uid, $experiment ) ) == Enhanced; // assigns customer if first time called}// The experiment branchif( user_should_see_test_feature( $uid ) ) { // behavior for experiment} else { // control behavior} 11
  12. 12. CLOSE EXPERIMENTS CONTROLPARTICIPANTS 12
  13. 13. UNDESIRED RESULTDESIRED RESULTNO STATISTICAL SIGNIFICANCE 13
  14. 14. 14
  15. 15. CONTINUOUSDEPLOYMENT 15
  16. 16. NO STAGING CLUSTER NO QA REVIEWCOMMIT CODE TO LIVE IN PRODUCTION 16
  17. 17. 17
  18. 18. WAY EASIER TO FINDREGRESSIONS IN SMALLBATCHES OF COMMITS VS. 18
  19. 19. WAY EASIER TO FINDREGRESSIONS IN SMALL LESS THAN 30 LINES OFBATCHES OF COMMITS CODE TO INVESTIGATE VS. 19
  20. 20. FAST RESPONSE TIMES 20
  21. 21. 21
  22. 22. CONTINUOUSINTEGRATION
  23. 23. 23
  24. 24. CLUSTER IMMUNE SYSTEM 24
  25. 25. Local tests pass, Rollbackengineer commits (Blocks) code No Lots and lots of Metrics Yes Code deployed to tests run good? all servers All tests Yes Code deployed to Metrics No pass? % of servers still good? No Yes Revert commit Win! (Blocks)
  26. 26. COD E HE RE!
  27. 27. DEVELOP RELEASE RELEASE ITERATE ITERATE LEARN LEARN 27
  28. 28. EMPOWERED! 28
  29. 29. LEARN FASTER + RELEASE FASTER + CULTURE OF EMPOWERMENT + MORE CUSTOMER-FACING VALUEINCRESED CHANCE OF SUCCESS 29
  30. 30. Thank  You!    Any  Ques0ons?   Bre4  G.  Durre4   IMVU recognized as:bdurre4@imvu.com   Inc. 500Twi4er:  @bdurre4   http://bit.ly/dv52wK Red Herring 100:   Tweet  this  talk!   http://bit.ly/bbz5Ex Best Place to Work:   http://bit.ly/aAVdp8 #login2011   (and were hiring) http://www.imvu.com/jobs 30

×