Continuous Deployment at Lean LA

9,731 views
9,595 views

Published on

June 30, 2011 Continuous Deployment presentation at Lean LA. Complete walkthrough of IMVU's systems from sandbox to production cluster.

2 Comments
19 Likes
Statistics
Notes
No Downloads
Views
Total views
9,731
On SlideShare
0
From Embeds
0
Number of Embeds
1,556
Actions
Shares
0
Downloads
0
Comments
2
Likes
19
Embeds 0
No embeds

No notes for slide

Continuous Deployment at Lean LA

  1. 1. CONTINUOUS DEPLOYMENTBrett G. Durrett,VP Engineering & Operations, IMVU - @bdurrett 1
  2. 2. Destination siteChat and Games with AvatarsRevenue over $40 million annuallyProfitableEngineering staff > 50 people 2
  3. 3. NO STAGING CLUSTER NO QA REVIEWCOMMIT CODE TO LIVE IN PRODUCTION 3
  4. 4. 4
  5. 5. DEVELOP RELEASE PRAY FOR CUSTOMERS 5
  6. 6. DEVELOP RELEASE ITERATE LEARN 6
  7. 7. 7
  8. 8. 8
  9. 9. WAY EASIER TO FINDREGRESSIONS IN SMALL LESS THAN 30 LINES OFBATCHES OF COMMITS CODE TO INVESTIGATE VS. 9
  10. 10. FAST RESPONSE TIMES 10
  11. 11. 11
  12. 12. DON’T TAKE MY WORD FOR IT…Reducing batch size…reduces cycle timereduces variability in flowaccelerates feedbackreduces riskreduces overheadLarge batches…reduce efficiencyinherently lower motivation and energycause exponential cost and schedule growthlead to even larger batchesThe entire batch is limited by its worst element Chapter 5 – Reducing Batch Size
  13. 13. Local tests pass, Rollbackengineer commits (Blocks) code No Lots and lots of Metrics Yes Code deployed to tests run good? all servers All tests Yes Code deployed to Metrics No pass? % of servers still good? No Yes Revert commit Win! (Blocks)
  14. 14. Local tests pass, Rollbackengineer commits (Blocks) code No Lots and lots of Metrics Yes Code deployed to tests run good? all servers All tests Yes Code deployed to Metrics No pass? % of servers still good? No Yes Revert commit Win! (Blocks)
  15. 15. COD E HE RE!
  16. 16. Local tests pass, Rollbackengineer commits (Blocks) code No Lots and lots of Metrics Yes Code deployed to tests run good? all servers All tests Yes Code deployed to Metrics No pass? % of servers still good? No Yes Revert commit Win! (Blocks)
  17. 17. CONTINUOUSINTEGRATION
  18. 18. 20
  19. 19. Local tests pass, Rollbackengineer commits (Blocks) code No Lots and lots of Metrics Yes Code deployed to tests run good? all servers All tests Yes Code deployed to Metrics No pass? % of servers still good? No Yes Revert commit Win! (Blocks)
  20. 20. Local tests pass, Rollbackengineer commits (Blocks) code No Lots and lots of Metrics Yes Code deployed to tests run good? all servers All tests Yes Code deployed to Metrics No pass? % of servers still good? No Yes Revert commit Win! (Blocks)
  21. 21. CLUSTER IMMUNE SYSTEM 24
  22. 22. IS IT IMPORTANT TO YOUR BUSINESS? MONITOR ITSYSTEM PERFORMANCE BUSINESS PERFORMANCE
  23. 23. Local tests pass, Rollbackengineer commits (Blocks) code No Lots and lots of Metrics Yes Code deployed to tests run good? all servers All tests Yes Code deployed to Metrics No pass? % of servers still good? No Yes Revert commit Win! (Blocks)
  24. 24. Local tests pass, Rollbackengineer commits (Blocks) code No Lots and lots of Metrics Yes Code deployed to tests run good? all servers All tests Yes Code deployed to Metrics No pass? % of servers still good? No Yes Revert commit Win! (Blocks)
  25. 25. Local tests pass, Rollbackengineer commits (Blocks) code No Lots and lots of Metrics Yes Code deployed to tests run good? all servers All tests Yes Code deployed to Metrics No pass? % of servers still good? No Yes Revert commit Win! (Blocks)
  26. 26. Local tests pass, Rollbackengineer commits (Blocks) code No Lots and lots of Metrics Yes Code deployed to tests run good? all servers All tests Yes Code deployed to Metrics No pass? % of servers still good? No Yes Revert commit Win! (Blocks)
  27. 27. 31
  28. 28. COMMIT TO MAKING FORWARD PROGRESSNEW PRODUCT ESTABLISHED PRODUCTSTART WITH SANDBOX START WITH PRODUCTIONJUST PUSH! AUTOMATE DEPLOYSIDEAL TIME FOR FAILURES!!! BUILD CONFIDENCE 32
  29. 29. EXPECT SOMEHURDLES
  30. 30. BUILD ISOLATION! FASTER TEST RUNS BUY HARDWARE SORT TESTS BY SPEED DEPENDENCY INJECTION AND… THE HYPOTHESIS BUILDER!
  31. 31. The  Trickier  Bits  •  Catching  issues  that  fail  slowly   –  SELECT * FROM growing_table WHERE 1  •  Some  cri9cal  areas  cause  hard  lock-­‐ups   –  MySQL   –  Memcached  •  Lack  of  test  coverage  of  older  code   –  Not  an  issue  if  you  start  with  test  coverage  •  Outsourcing   37
  32. 32. BUILD SYSTEMS ARE CRITICAL BUSINESS FUNCTIONS RUN THEM THAT WAY!
  33. 33. COD E HE RE!
  34. 34. $experiment = array( ‘name’ -> ‘LeanLA’, initial_rollout => 0, branches => array( Enhanced => 0.5, // can specify multiple weighted branches ) );// Helper function for readabilityfunction user_should_see_test_feature( $uid = NULL ) { return get_branch_for_customer( $uid, $experiment ) ) == Enhanced; // assigns customer if first time called}// The experiment branchif( user_should_see_test_feature( $uid ) ) { // behavior for experiment} else { // control behavior} 41
  35. 35. CLOSE EXPERIMENTS CONTROLPARTICIPANTS 42
  36. 36. UNDESIRED RESULTDESIRED RESULTNO STATISTICAL SIGNIFICANCE 43
  37. 37. THE “PERFECT” SPRINT Work RemainingSprint Start Planned Sprint End
  38. 38. REALITY STRIKES When releases happen every 15 minutes, this can be arbitrary Outstanding Issues Incomplete Features Tech Review Refactoring Work Remaining Drop or pushSprint Start Planned Sprint End
  39. 39. END THE SPRINT WHEN WORK IS DONE Work Remaining OverageSprint Start Sprint End Goal Sprint End
  40. 40. DEVELOP RELEASE RELEASE ITERATE ITERATE LEARN LEARN 47
  41. 41. EMPOWERED! 48
  42. 42. Thank  You!    Any  Ques9ons?   BreH  G.  DurreH   IMVU recognized as:bdurreH@imvu.com   Inc. 500TwiHer:  @bdurreH   http://bit.ly/dv52wK Red Herring 100:   Tweet  this  talk!   http://bit.ly/bbz5Ex Best Place to Work:   http://bit.ly/aAVdp8 @bdurreH   (and were hiring) http://www.imvu.com/jobs 49
  43. 43. Thank  You!  •  The  amazing  engineers  at  IMVU  that  make  this  possible  •  Patrick  Vlaskovitz,  Joe  Zulli  and  Peter  Mauro  for  invi9ng  me  to  speak  at  LeanLA  •  Avesta  Rasouli  and  CoLoX  for  hos9ng  and  providing  a  great  workspace  •  John  Overmeyer,  Single  Lane  Traffic  Photo:   hHp://www.flickr.com/photos/onebigchickenman/4869442019/   50

×