Your SlideShare is downloading. ×
0
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Continuous Deployment at Lean LA
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Continuous Deployment at Lean LA

8,938

Published on

June 30, 2011 Continuous Deployment presentation at Lean LA. Complete walkthrough of IMVU's systems from sandbox to production cluster.

June 30, 2011 Continuous Deployment presentation at Lean LA. Complete walkthrough of IMVU's systems from sandbox to production cluster.

2 Comments
17 Likes
Statistics
Notes
No Downloads
Views
Total Views
8,938
On Slideshare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
0
Comments
2
Likes
17
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. CONTINUOUS DEPLOYMENTBrett G. Durrett,VP Engineering & Operations, IMVU - @bdurrett 1
  • 2. Destination siteChat and Games with AvatarsRevenue over $40 million annuallyProfitableEngineering staff > 50 people 2
  • 3. NO STAGING CLUSTER NO QA REVIEWCOMMIT CODE TO LIVE IN PRODUCTION 3
  • 4. 4
  • 5. DEVELOP RELEASE PRAY FOR CUSTOMERS 5
  • 6. DEVELOP RELEASE ITERATE LEARN 6
  • 7. 7
  • 8. 8
  • 9. WAY EASIER TO FINDREGRESSIONS IN SMALL LESS THAN 30 LINES OFBATCHES OF COMMITS CODE TO INVESTIGATE VS. 9
  • 10. FAST RESPONSE TIMES 10
  • 11. 11
  • 12. DON’T TAKE MY WORD FOR IT…Reducing batch size…reduces cycle timereduces variability in flowaccelerates feedbackreduces riskreduces overheadLarge batches…reduce efficiencyinherently lower motivation and energycause exponential cost and schedule growthlead to even larger batchesThe entire batch is limited by its worst element Chapter 5 – Reducing Batch Size
  • 13. Local tests pass, Rollbackengineer commits (Blocks) code No Lots and lots of Metrics Yes Code deployed to tests run good? all servers All tests Yes Code deployed to Metrics No pass? % of servers still good? No Yes Revert commit Win! (Blocks)
  • 14. Local tests pass, Rollbackengineer commits (Blocks) code No Lots and lots of Metrics Yes Code deployed to tests run good? all servers All tests Yes Code deployed to Metrics No pass? % of servers still good? No Yes Revert commit Win! (Blocks)
  • 15. COD E HE RE!
  • 16. Local tests pass, Rollbackengineer commits (Blocks) code No Lots and lots of Metrics Yes Code deployed to tests run good? all servers All tests Yes Code deployed to Metrics No pass? % of servers still good? No Yes Revert commit Win! (Blocks)
  • 17. CONTINUOUSINTEGRATION
  • 18. 20
  • 19. Local tests pass, Rollbackengineer commits (Blocks) code No Lots and lots of Metrics Yes Code deployed to tests run good? all servers All tests Yes Code deployed to Metrics No pass? % of servers still good? No Yes Revert commit Win! (Blocks)
  • 20. Local tests pass, Rollbackengineer commits (Blocks) code No Lots and lots of Metrics Yes Code deployed to tests run good? all servers All tests Yes Code deployed to Metrics No pass? % of servers still good? No Yes Revert commit Win! (Blocks)
  • 21. CLUSTER IMMUNE SYSTEM 24
  • 22. IS IT IMPORTANT TO YOUR BUSINESS? MONITOR ITSYSTEM PERFORMANCE BUSINESS PERFORMANCE
  • 23. Local tests pass, Rollbackengineer commits (Blocks) code No Lots and lots of Metrics Yes Code deployed to tests run good? all servers All tests Yes Code deployed to Metrics No pass? % of servers still good? No Yes Revert commit Win! (Blocks)
  • 24. Local tests pass, Rollbackengineer commits (Blocks) code No Lots and lots of Metrics Yes Code deployed to tests run good? all servers All tests Yes Code deployed to Metrics No pass? % of servers still good? No Yes Revert commit Win! (Blocks)
  • 25. Local tests pass, Rollbackengineer commits (Blocks) code No Lots and lots of Metrics Yes Code deployed to tests run good? all servers All tests Yes Code deployed to Metrics No pass? % of servers still good? No Yes Revert commit Win! (Blocks)
  • 26. Local tests pass, Rollbackengineer commits (Blocks) code No Lots and lots of Metrics Yes Code deployed to tests run good? all servers All tests Yes Code deployed to Metrics No pass? % of servers still good? No Yes Revert commit Win! (Blocks)
  • 27. 31
  • 28. COMMIT TO MAKING FORWARD PROGRESSNEW PRODUCT ESTABLISHED PRODUCTSTART WITH SANDBOX START WITH PRODUCTIONJUST PUSH! AUTOMATE DEPLOYSIDEAL TIME FOR FAILURES!!! BUILD CONFIDENCE 32
  • 29. EXPECT SOMEHURDLES
  • 30. BUILD ISOLATION! FASTER TEST RUNS BUY HARDWARE SORT TESTS BY SPEED DEPENDENCY INJECTION AND… THE HYPOTHESIS BUILDER!
  • 31. The  Trickier  Bits  •  Catching  issues  that  fail  slowly   –  SELECT * FROM growing_table WHERE 1  •  Some  cri9cal  areas  cause  hard  lock-­‐ups   –  MySQL   –  Memcached  •  Lack  of  test  coverage  of  older  code   –  Not  an  issue  if  you  start  with  test  coverage  •  Outsourcing   37
  • 32. BUILD SYSTEMS ARE CRITICAL BUSINESS FUNCTIONS RUN THEM THAT WAY!
  • 33. COD E HE RE!
  • 34. $experiment = array( ‘name’ -> ‘LeanLA’, initial_rollout => 0, branches => array( Enhanced => 0.5, // can specify multiple weighted branches ) );// Helper function for readabilityfunction user_should_see_test_feature( $uid = NULL ) { return get_branch_for_customer( $uid, $experiment ) ) == Enhanced; // assigns customer if first time called}// The experiment branchif( user_should_see_test_feature( $uid ) ) { // behavior for experiment} else { // control behavior} 41
  • 35. CLOSE EXPERIMENTS CONTROLPARTICIPANTS 42
  • 36. UNDESIRED RESULTDESIRED RESULTNO STATISTICAL SIGNIFICANCE 43
  • 37. THE “PERFECT” SPRINT Work RemainingSprint Start Planned Sprint End
  • 38. REALITY STRIKES When releases happen every 15 minutes, this can be arbitrary Outstanding Issues Incomplete Features Tech Review Refactoring Work Remaining Drop or pushSprint Start Planned Sprint End
  • 39. END THE SPRINT WHEN WORK IS DONE Work Remaining OverageSprint Start Sprint End Goal Sprint End
  • 40. DEVELOP RELEASE RELEASE ITERATE ITERATE LEARN LEARN 47
  • 41. EMPOWERED! 48
  • 42. Thank  You!    Any  Ques9ons?   BreH  G.  DurreH   IMVU recognized as:bdurreH@imvu.com   Inc. 500TwiHer:  @bdurreH   http://bit.ly/dv52wK Red Herring 100:   Tweet  this  talk!   http://bit.ly/bbz5Ex Best Place to Work:   http://bit.ly/aAVdp8 @bdurreH   (and were hiring) http://www.imvu.com/jobs 49
  • 43. Thank  You!  •  The  amazing  engineers  at  IMVU  that  make  this  possible  •  Patrick  Vlaskovitz,  Joe  Zulli  and  Peter  Mauro  for  invi9ng  me  to  speak  at  LeanLA  •  Avesta  Rasouli  and  CoLoX  for  hos9ng  and  providing  a  great  workspace  •  John  Overmeyer,  Single  Lane  Traffic  Photo:   hHp://www.flickr.com/photos/onebigchickenman/4869442019/   50

×