Continuous Deployment at Lean LA

CONTINUOUS DEPLOYMENTBrett G. Durrett,VP Engineering & Operations, IMVU - @bdurrett 1

Destination siteChat and Games with AvatarsRevenue over $40 million annuallyProfitableEngineering staff > 50 people 2

NO STAGING CLUSTER NO QA REVIEWCOMMIT CODE TO LIVE IN PRODUCTION 3

DEVELOP RELEASE PRAY FOR CUSTOMERS 5

DEVELOP RELEASE ITERATE LEARN 6

WAY EASIER TO FINDREGRESSIONS IN SMALL LESS THAN 30 LINES OFBATCHES OF COMMITS CODE TO INVESTIGATE VS. 9

FAST RESPONSE TIMES 10

DON’T TAKE MY WORD FOR IT…Reducing batch size…reduces cycle timereduces variability in flowaccelerates feedbackreduces riskreduces overheadLarge batches…reduce efficiencyinherently lower motivation and energycause exponential cost and schedule growthlead to even larger batchesThe entire batch is limited by its worst element Chapter 5 – Reducing Batch Size

Local tests pass, Rollbackengineer commits (Blocks) code No Lots and lots of Metrics Yes Code deployed to tests run good? all servers All tests Yes Code deployed to Metrics No pass? % of servers still good? No Yes Revert commit Win! (Blocks)

COD E HE RE!

CONTINUOUSINTEGRATION

CLUSTER IMMUNE SYSTEM 24

IS IT IMPORTANT TO YOUR BUSINESS? MONITOR ITSYSTEM PERFORMANCE BUSINESS PERFORMANCE

COMMIT TO MAKING FORWARD PROGRESSNEW PRODUCT ESTABLISHED PRODUCTSTART WITH SANDBOX START WITH PRODUCTIONJUST PUSH! AUTOMATE DEPLOYSIDEAL TIME FOR FAILURES!!! BUILD CONFIDENCE 32

EXPECT SOMEHURDLES

BUILD ISOLATION! FASTER TEST RUNS BUY HARDWARE SORT TESTS BY SPEED DEPENDENCY INJECTION AND… THE HYPOTHESIS BUILDER!

The Trickier Bits •  Catching issues that fail slowly –  SELECT * FROM growing_table WHERE 1 •  Some cri9cal areas cause hard lock-‐ups –  MySQL –  Memcached •  Lack of test coverage of older code –  Not an issue if you start with test coverage •  Outsourcing 37

BUILD SYSTEMS ARE CRITICAL BUSINESS FUNCTIONS RUN THEM THAT WAY!

COD E HE RE!

$experiment = array( ‘name’ -> ‘LeanLA’, initial_rollout => 0, branches => array( Enhanced => 0.5, // can specify multiple weighted branches ) );// Helper function for readabilityfunction user_should_see_test_feature( $uid = NULL ) { return get_branch_for_customer( $uid, $experiment ) ) == Enhanced; // assigns customer if first time called}// The experiment branchif( user_should_see_test_feature( $uid ) ) { // behavior for experiment} else { // control behavior} 41

CLOSE EXPERIMENTS CONTROLPARTICIPANTS 42

UNDESIRED RESULTDESIRED RESULTNO STATISTICAL SIGNIFICANCE 43

THE “PERFECT” SPRINT Work RemainingSprint Start Planned Sprint End

REALITY STRIKES When releases happen every 15 minutes, this can be arbitrary Outstanding Issues Incomplete Features Tech Review Refactoring Work Remaining Drop or pushSprint Start Planned Sprint End

END THE SPRINT WHEN WORK IS DONE Work Remaining OverageSprint Start Sprint End Goal Sprint End

DEVELOP RELEASE RELEASE ITERATE ITERATE LEARN LEARN 47

EMPOWERED! 48

Thank You! Any Ques9ons? BreH G. DurreH IMVU recognized as:bdurreH@imvu.com Inc. 500TwiHer: @bdurreH http://bit.ly/dv52wK Red Herring 100: Tweet this talk! http://bit.ly/bbz5Ex Best Place to Work: http://bit.ly/aAVdp8 @bdurreH (and were hiring) http://www.imvu.com/jobs 49

Thank You! •  The amazing engineers at IMVU that make this possible •  Patrick Vlaskovitz, Joe Zulli and Peter Mauro for invi9ng me to speak at LeanLA •  Avesta Rasouli and CoLoX for hos9ng and providing a great workspace •  John Overmeyer, Single Lane Traﬃc Photo: hHp://www.ﬂickr.com/photos/onebigchickenman/4869442019/ 50

http://bostinno.com	388
http://www.maxmednik.com	323
http://www.techstarthub.com	102
http://bostinnovation.com	93
http://www.weebly.com	86
https://twitter.com	17
http://www.techstartupjobs.com	17
http://twitter.com	10
http://tweetedtimes.com	4
http://paper.li	2
http://t.co	1
http://www.twylah.com	1
http://www.imvu.com	1
http://kmccarth.com	1
http://www.google.com	1

Continuous Deployment at Lean LA

by Brett Durrett on Jul 02, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

15 Embeds 1,047

Statistics

Continuous Deployment at Lean LA — Presentation Transcript