Learning Fast With A/B Testing and Continuous Deployment
•
5 likes•1,760 views
IMVU's use of A/B testing combined with Continuous Deployment to quickly identify and respond to opportunities.
Recommended
Recommended
More Related Content
Similar to Learning Fast With A/B Testing and Continuous Deployment
Similar to Learning Fast With A/B Testing and Continuous Deployment (20)
Recently uploaded
Recently uploaded (20)
Learning Fast With A/B Testing and Continuous Deployment
- 1. REAL TIME LEARNING USING A/B TESTING AND CONTINUOUS DEPLOYMENT Brett G. Durrett,VP Engineering & Operations, IMVU - @bdurrett LOGIN Conference, 2011 1
- 2. DEVELOP RELEASE PRAY FOR CUSTOMERS 2
- 3. DEVELOP RELEASE ITERATE LEARN 3
- 4. “Casual Players Evolving Fast” Richard Garriott Keynote “Only 4% of games that go into production are profitable” Scott Dodson Gamification Will Eat Itself: How Gamification Can Evolve, And Why It Must. “Most people are not willing to bet their business on instant perfection” Rajat Kongovi Stop wasting your data! Using data to drive performance.. 4
- 5. 5
- 6. UH O H NO PROBLEM 6
- 7. * The Pepsi challenge was not A/B testing 7
- 9. VALIDATES PRODUCT DECISIONS IDENTIFIES OPPORTUNITIES GIVES FREEDOM TO EXPERIMENT WILL NOT DESIGN YOUR PRODUCT! 9
- 10. 1-2-3 EASY TESTING 1. EASY TO CODE 2. EASY TO ADMINISTER 3. EASY TO READ RESULTS
- 11. $experiment = array( ‘name’ -> ‘DemoOnline2011’, 'initial_rollout' => '0', 'branches' => array( 'Enhanced' => 0.5, // can specify multiple weighted branches ) ); // Helper function for readability function user_should_see_test_feature( $uid = NULL ) { return get_branch_for_customer( $uid, $experiment ) ) == 'Enhanced'; // assigns customer if first time called } // The experiment branch if( user_should_see_test_feature( $uid ) ) { // behavior for experiment } else { // control behavior } 11
- 12. CLOSE EXPERIMENTS CONTROL PARTICIPANTS 12
- 13. UNDESIRED RESULT DESIRED RESULT NO STATISTICAL SIGNIFICANCE 13
- 14. 14
- 15. CONTINUOUS DEPLOYMENT 15
- 16. NO STAGING CLUSTER NO QA REVIEW COMMIT CODE TO LIVE IN PRODUCTION 16
- 17. 17
- 18. WAY EASIER TO FIND REGRESSIONS IN SMALL BATCHES OF COMMITS VS. 18
- 19. WAY EASIER TO FIND REGRESSIONS IN SMALL LESS THAN 30 LINES OF BATCHES OF COMMITS CODE TO INVESTIGATE VS. 19
- 20. FAST RESPONSE TIMES 20
- 21. 21
- 23. 23
- 24. CLUSTER IMMUNE SYSTEM 24
- 25. Local tests pass, Rollback engineer commits (Blocks) code No Lots and lots of Metrics Yes Code deployed to tests run good? all servers All tests Yes Code deployed to Metrics No pass? % of servers still good? No Yes Revert commit Win! (Blocks)
- 26. COD E HE RE!
- 27. DEVELOP RELEASE RELEASE ITERATE ITERATE LEARN LEARN 27
- 28. EMPOWERED! 28
- 29. LEARN FASTER + RELEASE FASTER + CULTURE OF EMPOWERMENT + MORE CUSTOMER-FACING VALUE INCRESED CHANCE OF SUCCESS 29
- 30. Thank You! Any Ques0ons? Bre4 G. Durre4 IMVU recognized as: bdurre4@imvu.com Inc. 500 Twi4er: @bdurre4 http://bit.ly/dv52wK Red Herring 100: Tweet this talk! http://bit.ly/bbz5Ex Best Place to Work: http://bit.ly/aAVdp8 #login2011 (and we're hiring) http://www.imvu.com/jobs 30