WAYSTO MINIMISE PERFORMANCE RISKSIN CONTINUOUS DELIVERYAdriaanThomas4 June 2013
INTRODUCTION
OBJECTIVEPut working software into production as quickly as possible, whilst minimising risk ofload-related problems:• Bad...
TRADITIONAL APPROACHLoad testing through simulationhttp://www.flickr.com/photos/danramarch/4423023837
DECIDE WHATTOTEST•Focus on busiest instant•Model most-hit functionality•Extrapolate to expected load•Look at production tr...
DECIDE ON SCOPEComponent testChain testFull environment test•Test coverage•Level of certainty•Number of systems•Amount of ...
SET UPTEST DATA• Usually starts as a copy from production• Or educated guess what people will enter• Render anonymous• Mak...
DECIDE ON STRATEGYOne or more of:•Scalability test•Stress test•Endurance test•Regression test•Resilience testhttp://www.fli...
DECIDE ONTEST DURATION(which is tricky)http://www.flickr.com/photos/wwarby/3297205226
PROVIDE HARDWAREhttp://www.flickr.com/photos/s_w_ellis/2681151694/Copy of production?Only one copy?Virtualisation?Sharing b...
INTEGRATE INTO PIPELINEUnit testFunctionalintegrationtestLoad testVery fast Fast Takes longer
INTEGRATE INTO PIPELINEUnit testFunctionalintegrationtestLoad testVery fast Takes longer
PERMANENT LOADTESTINGDaytime: constant load, teamsinspect impact of changesNighttime: EndurancetestWeekends: refresh test ...
RESPONSETIMEDNS lookup (www.xebia.com)Time to first byte + loading HTMLTime to renderTime to document completeBrowser CPU u...
IMPACT OFTHE BROWSERwww.browserscope.org
CLEAR REQUIREMENTSResponse timeFail: 10 Now: 3.5 Goal: 1Intention: Users get a response quickly so thatthey are happy and ...
WebPageTest: first view + repeat view (median of 3)95th percentile response times from access logsADJUST REQUIREMENTS DUETO...
Playground to test changesNo impact on real usersLess pressureMore workGuesswork and extrapolationCan take a significant am...
THINGS WILL BREAK...... in spite of your best effortshttp://www.flickr.com/photos/jmarty/1239950166/
SO INSTEAD WE SHOULD FOCUS ONFAST RECOVERYhttp://www.flickr.com/photos/19107136@N02/8386567228/
“MTTR is more important thanMTBF*”John Allspaw* for most types of F
00.51.01.52.099thpercentileresponsetime(s)Test durationMTBF LEADSTO FUD
Time→TTD find cause (RCA) write & test fix build deployvalidatecompiledeploy&testMonitoringAlerts•Skills•Organisation•Cultur...
DEMING FEEDBACK LOOPSPlanDoStudyAct
OODA LOOPSObserveOrientDecideAct
AVOIDTEST-ONLY MEASUREMENTS
SIMPLE ARCHITECTURE
THE ONLYTHINGTHAT MATTERS ISWHAT HAPPENS IN PRODUCTIONEverything else is an assumption.
DEPLOYING CHANGEShttp://www.flickr.com/photos/39463459@N08/5083733600
BLUE-GREEN DEPLOYMENTSVersion n+1Version nAmazonRoute 53ElasticLoadBalancerElasticLoadBalancerInstancesInstances
DARK LAUNCHINGWeb page DB
DARK LAUNCHINGWeb page DB Weather SP
DARK LAUNCHINGWeb page DB Weather SP
FEATURETOGGLES
CANARY RELEASING0% 100%
PRODUCTION-IMMUNE SYSTEMS
CONTROLLED LOADTESTINGInstance RDS DBInstanceRDS DB InstanceRead ReplicaInstanceInstanceAmazonRoute 53ElasticLoadBalancer
MONITORINGhttp://www.flickr.com/photos/smieyetracking/5609671098/
MONITORINGTechnical metrics•CPU use•Memory use•TPS•Response times•etcProcess metrics•# bugs•MTTR, MTTD•Time from idea to l...
MEASURE IMPACT OF CHANGES
tail	  -­‐f	  access_log	  |	  alstat.pl	  -­‐i10	  -­‐n10	  -­‐stt	  	  	  	  Hits	  	  Hits%	  	  	  	  TPS	  AvgTmTk	  ...
MEASURE LATENCYAvg. response times front end vs backendNumber of calls
SMALL DEPLOYMENTShttp://www.flickr.com/photos/rbulmahn/4925464931/
GO/NO-GO MEETINGS• What are the biggest fears?• How can we measure this?• What can be done if it does happen?
RETROSPECTIVESHow can we prevent a failure fromhappening again?How can we detect it earlier?Was there only one root cause?...
INTRODUCE OUTAGESChaos monkeyGame day exerciseshttp://www.flickr.com/photos/frostnova/440551442/
CULTURE• Dev and Ops work together on providing information.• Assumptions are dangerous, try to eliminate as many as possi...
CLAMSCultureLeanAutomationMeasurementSharing
SIMPLE, FLEXIBLE ARCHITECTURE• If the site goes down often, probably its architecture is at fault• Avoid fragile systems• ...
CHANGES FORTHE BUSINESS• Accept to push smaller changes.• Continuous delivery vs continuousdeployment.• Share data.
CONCLUSIONWork on your ability to respond to failure.Trying to prevent failure can slow you downand make you focus on the ...
QUESTIONS?athomas@xebia.com@a32anwww.xebia.comblog.xebia.com(we’re hiring)
Ways to minimise performance risks in continuous delivery
Ways to minimise performance risks in continuous delivery
Upcoming SlideShare
Loading in...5
×

Ways to minimise performance risks in continuous delivery

132

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
132
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Ways to minimise performance risks in continuous delivery

  1. 1. WAYSTO MINIMISE PERFORMANCE RISKSIN CONTINUOUS DELIVERYAdriaanThomas4 June 2013
  2. 2. INTRODUCTION
  3. 3. OBJECTIVEPut working software into production as quickly as possible, whilst minimising risk ofload-related problems:• Bad response times• Lack of capacity• Availability too low• Excessive system resource useWithin the context of websites.
  4. 4. TRADITIONAL APPROACHLoad testing through simulationhttp://www.flickr.com/photos/danramarch/4423023837
  5. 5. DECIDE WHATTOTEST•Focus on busiest instant•Model most-hit functionality•Extrapolate to expected load•Look at production traffic•Or attempt educated guess
  6. 6. DECIDE ON SCOPEComponent testChain testFull environment test•Test coverage•Level of certainty•Number of systems•Amount of work
  7. 7. SET UPTEST DATA• Usually starts as a copy from production• Or educated guess what people will enter• Render anonymous• Make tests deterministic• Synchronise between all systemshttp://www.flickr.com/photos/22168167@N00/3889737939/
  8. 8. DECIDE ON STRATEGYOne or more of:•Scalability test•Stress test•Endurance test•Regression test•Resilience testhttp://www.flickr.com/photos/timjoyfamily/5935279962/
  9. 9. DECIDE ONTEST DURATION(which is tricky)http://www.flickr.com/photos/wwarby/3297205226
  10. 10. PROVIDE HARDWAREhttp://www.flickr.com/photos/s_w_ellis/2681151694/Copy of production?Only one copy?Virtualisation?Sharing between teams?
  11. 11. INTEGRATE INTO PIPELINEUnit testFunctionalintegrationtestLoad testVery fast Fast Takes longer
  12. 12. INTEGRATE INTO PIPELINEUnit testFunctionalintegrationtestLoad testVery fast Takes longer
  13. 13. PERMANENT LOADTESTINGDaytime: constant load, teamsinspect impact of changesNighttime: EndurancetestWeekends: refresh test datahttp://www.flickr.com/photos/renaissancechambara/5106171956/
  14. 14. RESPONSETIMEDNS lookup (www.xebia.com)Time to first byte + loading HTMLTime to renderTime to document completeBrowser CPU useBandwidth# connections to a singlehosthttp://www.webpagetest.org/result/130522_FG_10SC/1/details/SSL handshakeParse timesBlocking client code
  15. 15. IMPACT OFTHE BROWSERwww.browserscope.org
  16. 16. CLEAR REQUIREMENTSResponse timeFail: 10 Now: 3.5 Goal: 1Intention: Users get a response quickly so thatthey are happy and spend more money.Stakeholder: Marketing dept.Scale: 95th percentile of “document complete”response times, in seconds, measured over oneminute.Metric: Page load times as reported by ourRUM tool.Inspired byTom Gilb, Competitive Engineering
  17. 17. WebPageTest: first view + repeat view (median of 3)95th percentile response times from access logsADJUST REQUIREMENTS DUETO LACK OFREAL BROWSERS
  18. 18. Playground to test changesNo impact on real usersLess pressureMore workGuesswork and extrapolationCan take a significant amount of timeMore hardware
  19. 19. THINGS WILL BREAK...... in spite of your best effortshttp://www.flickr.com/photos/jmarty/1239950166/
  20. 20. SO INSTEAD WE SHOULD FOCUS ONFAST RECOVERYhttp://www.flickr.com/photos/19107136@N02/8386567228/
  21. 21. “MTTR is more important thanMTBF*”John Allspaw* for most types of F
  22. 22. 00.51.01.52.099thpercentileresponsetime(s)Test durationMTBF LEADSTO FUD
  23. 23. Time→TTD find cause (RCA) write & test fix build deployvalidatecompiledeploy&testMonitoringAlerts•Skills•Organisation•Culture•Maintainability•Simple architecture•Fastworkstations•Goodtooling•Abletoquicklytestlocally•Automation•Fastbuildserver•EfficienttestsMonitoring•Automation•FlexiblearchitectureTTR
  24. 24. DEMING FEEDBACK LOOPSPlanDoStudyAct
  25. 25. OODA LOOPSObserveOrientDecideAct
  26. 26. AVOIDTEST-ONLY MEASUREMENTS
  27. 27. SIMPLE ARCHITECTURE
  28. 28. THE ONLYTHINGTHAT MATTERS ISWHAT HAPPENS IN PRODUCTIONEverything else is an assumption.
  29. 29. DEPLOYING CHANGEShttp://www.flickr.com/photos/39463459@N08/5083733600
  30. 30. BLUE-GREEN DEPLOYMENTSVersion n+1Version nAmazonRoute 53ElasticLoadBalancerElasticLoadBalancerInstancesInstances
  31. 31. DARK LAUNCHINGWeb page DB
  32. 32. DARK LAUNCHINGWeb page DB Weather SP
  33. 33. DARK LAUNCHINGWeb page DB Weather SP
  34. 34. FEATURETOGGLES
  35. 35. CANARY RELEASING0% 100%
  36. 36. PRODUCTION-IMMUNE SYSTEMS
  37. 37. CONTROLLED LOADTESTINGInstance RDS DBInstanceRDS DB InstanceRead ReplicaInstanceInstanceAmazonRoute 53ElasticLoadBalancer
  38. 38. MONITORINGhttp://www.flickr.com/photos/smieyetracking/5609671098/
  39. 39. MONITORINGTechnical metrics•CPU use•Memory use•TPS•Response times•etcProcess metrics•# bugs•MTTR, MTTD•Time from idea to live on site•etcBusiness metrics•Revenue•# unique visitors•etchttp://www.flickr.com/photos/smieyetracking/5609671098/
  40. 40. MEASURE IMPACT OF CHANGES
  41. 41. tail  -­‐f  access_log  |  alstat.pl  -­‐i10  -­‐n10  -­‐stt        Hits    Hits%        TPS  AvgTmTk  TTmTk%    AvgRSize  RSize%  2013-­‐06-­‐04  19:37:40  (08)            14      0.1%        1.4      1.652      5.7%            2691      0.2%  POST      200  /login.do            14      0.1%        1.4      0.918      3.2%            3739      0.3%  GET        200  /home.do            14      0.1%        1.4      0.879      3.1%            3185      0.2%  POST      200  /order.do              7      0.1%        0.7      0.807      1.4%            1974      0.1%  POST      200  /account.do              4      0.0%        0.4      0.735      0.7%            3228      0.1%  GET        200  /products.do              5      0.0%        0.5      0.697      0.9%              969      0.0%  POST      200  /settings.do              9      0.1%        0.9      0.687      1.5%            1827      0.1%  POST      200  /changeorder.do            27      0.2%        2.7      0.649      4.3%            2997      0.4%  POST      200  /newpasswd.do            15      0.1%        1.5      0.580      2.2%            2488      0.2%  GET        200  /offer.do            95      0.9%        9.5      0.520    12.2%            4801      2.3%  GET        200  /search.do
  42. 42. MEASURE LATENCYAvg. response times front end vs backendNumber of calls
  43. 43. SMALL DEPLOYMENTShttp://www.flickr.com/photos/rbulmahn/4925464931/
  44. 44. GO/NO-GO MEETINGS• What are the biggest fears?• How can we measure this?• What can be done if it does happen?
  45. 45. RETROSPECTIVESHow can we prevent a failure fromhappening again?How can we detect it earlier?Was there only one root cause?http://www.flickr.com/photos/katerha/8380451137
  46. 46. INTRODUCE OUTAGESChaos monkeyGame day exerciseshttp://www.flickr.com/photos/frostnova/440551442/
  47. 47. CULTURE• Dev and Ops work together on providing information.• Assumptions are dangerous, try to eliminate as many as possible.• Small changes are easier to fix than large ones.• Deploy during office hours so everyone is available in case problems happen.• All information, including business metrics, should be accessible to everyone.
  48. 48. CLAMSCultureLeanAutomationMeasurementSharing
  49. 49. SIMPLE, FLEXIBLE ARCHITECTURE• If the site goes down often, probably its architecture is at fault• Avoid fragile systems• Resilience is key• Scalable (redundancy is not waste)• Rather many small systems than a few large ones• State is a “hot brick”
  50. 50. CHANGES FORTHE BUSINESS• Accept to push smaller changes.• Continuous delivery vs continuousdeployment.• Share data.
  51. 51. CONCLUSIONWork on your ability to respond to failure.Trying to prevent failure can slow you downand make you focus on the wrong things.Keep assumptions clearly separated from facts. Make your decisions based on evidence.Measure everything, including the impact of changes to the business.Look for your compromise, try permanent load testing first and learn from that.
  52. 52. QUESTIONS?athomas@xebia.com@a32anwww.xebia.comblog.xebia.com(we’re hiring)
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×