• Save
Continuous Deployment at Lean LA
Upcoming SlideShare
Loading in...5
×
 

Continuous Deployment at Lean LA

on

  • 8,941 views

June 30, 2011 Continuous Deployment presentation at Lean LA. Complete walkthrough of IMVU's systems from sandbox to production cluster.

June 30, 2011 Continuous Deployment presentation at Lean LA. Complete walkthrough of IMVU's systems from sandbox to production cluster.

Statistics

Views

Total Views
8,941
Views on SlideShare
7,530
Embed Views
1,411

Actions

Likes
16
Downloads
0
Comments
2

18 Embeds 1,411

http://bostinno.com 515
http://www.maxmednik.com 451
http://www.techstarthub.com 103
http://bostinnovation.com 93
http://www.weebly.com 88
http://bostinno.streetwise.co 71
http://www.techstartupjobs.com 50
https://twitter.com 17
http://twitter.com 10
http://tweetedtimes.com 4
http://paper.li 2
http://t.co 1
http://www.google.com 1
http://kmccarth.com 1
http://www.twylah.com 1
http://www.imvu.com 1
https://abs.twimg.com 1
http://www.linkedin.com 1
More...

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • If you run a company on continuous development your very brave or very nuts !
    Imvu seems as a user to be BOTH
    Well done fellows
    Are you sure you want to
    Your message goes here
    Processing…
  • Thanks Brett - great talk.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Continuous Deployment at Lean LA Continuous Deployment at Lean LA Presentation Transcript

  • CONTINUOUS DEPLOYMENTBrett G. Durrett,VP Engineering & Operations, IMVU - @bdurrett 1
  • Destination siteChat and Games with AvatarsRevenue over $40 million annuallyProfitableEngineering staff > 50 people 2
  • NO STAGING CLUSTER NO QA REVIEWCOMMIT CODE TO LIVE IN PRODUCTION 3
  • 4
  • DEVELOP RELEASE PRAY FOR CUSTOMERS 5
  • DEVELOP RELEASE ITERATE LEARN 6
  • 7
  • 8
  • WAY EASIER TO FINDREGRESSIONS IN SMALL LESS THAN 30 LINES OFBATCHES OF COMMITS CODE TO INVESTIGATE VS. 9
  • FAST RESPONSE TIMES 10
  • 11
  • DON’T TAKE MY WORD FOR IT…Reducing batch size…reduces cycle timereduces variability in flowaccelerates feedbackreduces riskreduces overheadLarge batches…reduce efficiencyinherently lower motivation and energycause exponential cost and schedule growthlead to even larger batchesThe entire batch is limited by its worst element Chapter 5 – Reducing Batch Size
  • Local tests pass, Rollbackengineer commits (Blocks) code No Lots and lots of Metrics Yes Code deployed to tests run good? all servers All tests Yes Code deployed to Metrics No pass? % of servers still good? No Yes Revert commit Win! (Blocks)
  • Local tests pass, Rollbackengineer commits (Blocks) code No Lots and lots of Metrics Yes Code deployed to tests run good? all servers All tests Yes Code deployed to Metrics No pass? % of servers still good? No Yes Revert commit Win! (Blocks)
  • COD E HE RE!
  • Local tests pass, Rollbackengineer commits (Blocks) code No Lots and lots of Metrics Yes Code deployed to tests run good? all servers All tests Yes Code deployed to Metrics No pass? % of servers still good? No Yes Revert commit Win! (Blocks)
  • CONTINUOUSINTEGRATION
  • 20
  • Local tests pass, Rollbackengineer commits (Blocks) code No Lots and lots of Metrics Yes Code deployed to tests run good? all servers All tests Yes Code deployed to Metrics No pass? % of servers still good? No Yes Revert commit Win! (Blocks)
  • Local tests pass, Rollbackengineer commits (Blocks) code No Lots and lots of Metrics Yes Code deployed to tests run good? all servers All tests Yes Code deployed to Metrics No pass? % of servers still good? No Yes Revert commit Win! (Blocks)
  • CLUSTER IMMUNE SYSTEM 24
  • IS IT IMPORTANT TO YOUR BUSINESS? MONITOR ITSYSTEM PERFORMANCE BUSINESS PERFORMANCE
  • Local tests pass, Rollbackengineer commits (Blocks) code No Lots and lots of Metrics Yes Code deployed to tests run good? all servers All tests Yes Code deployed to Metrics No pass? % of servers still good? No Yes Revert commit Win! (Blocks)
  • Local tests pass, Rollbackengineer commits (Blocks) code No Lots and lots of Metrics Yes Code deployed to tests run good? all servers All tests Yes Code deployed to Metrics No pass? % of servers still good? No Yes Revert commit Win! (Blocks)
  • Local tests pass, Rollbackengineer commits (Blocks) code No Lots and lots of Metrics Yes Code deployed to tests run good? all servers All tests Yes Code deployed to Metrics No pass? % of servers still good? No Yes Revert commit Win! (Blocks)
  • Local tests pass, Rollbackengineer commits (Blocks) code No Lots and lots of Metrics Yes Code deployed to tests run good? all servers All tests Yes Code deployed to Metrics No pass? % of servers still good? No Yes Revert commit Win! (Blocks)
  • 31
  • COMMIT TO MAKING FORWARD PROGRESSNEW PRODUCT ESTABLISHED PRODUCTSTART WITH SANDBOX START WITH PRODUCTIONJUST PUSH! AUTOMATE DEPLOYSIDEAL TIME FOR FAILURES!!! BUILD CONFIDENCE 32
  • EXPECT SOMEHURDLES
  • BUILD ISOLATION! FASTER TEST RUNS BUY HARDWARE SORT TESTS BY SPEED DEPENDENCY INJECTION AND… THE HYPOTHESIS BUILDER!
  • The  Trickier  Bits  •  Catching  issues  that  fail  slowly   –  SELECT * FROM growing_table WHERE 1  •  Some  cri9cal  areas  cause  hard  lock-­‐ups   –  MySQL   –  Memcached  •  Lack  of  test  coverage  of  older  code   –  Not  an  issue  if  you  start  with  test  coverage  •  Outsourcing   37
  • BUILD SYSTEMS ARE CRITICAL BUSINESS FUNCTIONS RUN THEM THAT WAY!
  • COD E HE RE!
  • $experiment = array( ‘name’ -> ‘LeanLA’, initial_rollout => 0, branches => array( Enhanced => 0.5, // can specify multiple weighted branches ) );// Helper function for readabilityfunction user_should_see_test_feature( $uid = NULL ) { return get_branch_for_customer( $uid, $experiment ) ) == Enhanced; // assigns customer if first time called}// The experiment branchif( user_should_see_test_feature( $uid ) ) { // behavior for experiment} else { // control behavior} 41
  • CLOSE EXPERIMENTS CONTROLPARTICIPANTS 42
  • UNDESIRED RESULTDESIRED RESULTNO STATISTICAL SIGNIFICANCE 43
  • THE “PERFECT” SPRINT Work RemainingSprint Start Planned Sprint End
  • REALITY STRIKES When releases happen every 15 minutes, this can be arbitrary Outstanding Issues Incomplete Features Tech Review Refactoring Work Remaining Drop or pushSprint Start Planned Sprint End
  • END THE SPRINT WHEN WORK IS DONE Work Remaining OverageSprint Start Sprint End Goal Sprint End
  • DEVELOP RELEASE RELEASE ITERATE ITERATE LEARN LEARN 47
  • EMPOWERED! 48
  • Thank  You!    Any  Ques9ons?   BreH  G.  DurreH   IMVU recognized as:bdurreH@imvu.com   Inc. 500TwiHer:  @bdurreH   http://bit.ly/dv52wK Red Herring 100:   Tweet  this  talk!   http://bit.ly/bbz5Ex Best Place to Work:   http://bit.ly/aAVdp8 @bdurreH   (and were hiring) http://www.imvu.com/jobs 49
  • Thank  You!  •  The  amazing  engineers  at  IMVU  that  make  this  possible  •  Patrick  Vlaskovitz,  Joe  Zulli  and  Peter  Mauro  for  invi9ng  me  to  speak  at  LeanLA  •  Avesta  Rasouli  and  CoLoX  for  hos9ng  and  providing  a  great  workspace  •  John  Overmeyer,  Single  Lane  Traffic  Photo:   hHp://www.flickr.com/photos/onebigchickenman/4869442019/   50