CONTINUOUSDELIVERY WHILEMINIMISINGPERFORMANCERISKS
INTRODUCTION
OBJECTIVEPut working software into production as quickly as possible,whilst minimising risk of load-related problems:›   B...
PREVENTING RISK IS A BIG SUBJECT,WHAT FOLLOWS IS TAKEN FROM OUR EXPERIENCERISK PREVENTION IS A BIG SUBJECT     Photo by ch...
CONTINUOUS DELIVERY LITERATURE PROVIDES METHODSTHAT HELP REDUCE RISK›   Blue-green deployments›   Dark launching›   Featur...
BLUE-GREEN DEPLOYMENTS                    Elastic     Instances                     Load                   Balancer   Vers...
DARK LAUNCHING                 Web page   DB
DARK LAUNCHING                 Web page   DB   Weather SP
DARK LAUNCHING                 Web page   DB   Weather SP
FEATURE TOGGLES
CANARY RELEASING                   0%   100%
PRODUCTION IMMUNE SYSTEMS
USE CONTROLLED LOAD TESTING TO HELP CAPACITYPLANNING                       Instance           RDS DB                      ...
WORK WITH FAILURE›   Optimise for MTTD and MTTR, not MTBF›   Game day exercises›   Chaos monkey›   Go / NoGo meetings›   R...
BUT LEGACY SYSTEMS OFTEN LACK THE REQUIREDRESILIENCE
WHILE WE WORK ON OUR RESILIENCE, WE USE LOAD TESTSTO HELP IDENTIFY THE BIGGEST RISKS
PRE-PROD LOAD TESTING IS NOT FREE›   Extra code to maintain›   Usually test runs last several hours›   A production-like e...
USE IT WISELY, WHERE PRODUCTION TESTING IS STILLINAPPROPRIATE›   It provides no guarantee›   Use it to find any showstoppe...
USE IT AS A PLAYGROUND TO TRY RISKY CHANGES        Photo by vastateparksstaff: www.flickr.com/photos/vastateparksstaff/533...
Load tests              FunctionalBuild, unit              integration test, etc.                 testsVery often    Less ...
Load tests              FunctionalBuild, unit                      Load test              integration test, etc.          ...
THE AIM IS NOT PERFECTION, GO FOR “AS REALISTIC ASNEEDED”
SET UP TEST DATA IN THE WEEKEND, TO MINIMIZEDISRUPTION
WHEN IS A PROBLEM REALLY A PROBLEM?
FIND AN OBJECTIVE WAY TO JUDGE YOUR FINDINGS
ESTABLISH REQUIREMENTS TO MAKE CLEAR WHAT ISACCEPTABLE›   Seen from the main stakeholders’ perspective    – Response time:...
Concurrent users                             Fail:              Now:            Target:                             < 100k...
SO USE A REAL BROWSER TO TESTA REAL USER’S EXPERIENCE
Response time   Fail      [Today]    TargetHomepage.FV     > 6 sec    3.9 sec    2 secHomepage.RV     > 5 sec    2.8 sec  ...
TO MAKE COMPARING SENSIBLE, MAKE YOUR TESTSDETERMINISTICStub systems that you have no control over
LOAD TESTING SHOULD BE OPTIONAL, THE ONLY THINGTHAT COUNTS IS PRODUCTION!›   Your definition of done should reflect that› ...
ANYTHING YOU FIND IS AN OPPORTUNITY TO FIX MORETHAN ONE PROBLEM
SO WHAT MONITORING IS TYPICALLY NEEDED?›   Be able to localise where latency is coming from!    – For every system, all in...
CONCLUSIONIn order to put code live without pre-prod load testing, at leastthe following need to be in place:› Culture› St...
QUESTIONS?             athomas@xebia.com             @a32an             www.xebia.com             blog.xebia.com          ...
Continuous delivery while minimizing performance risks (dutch web ops meetup)
Continuous delivery while minimizing performance risks (dutch web ops meetup)
Continuous delivery while minimizing performance risks (dutch web ops meetup)
Upcoming SlideShare
Loading in...5
×

Continuous delivery while minimizing performance risks (dutch web ops meetup)

280

Published on

I modified my presentation from Velocity a bit, as I had more time to talk here. Given at http://www.meetup.com/Dutch-Web-Operations-Meetup/events/79161962/

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
280
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Continuous delivery while minimizing performance risks (dutch web ops meetup)

  1. 1. CONTINUOUSDELIVERY WHILEMINIMISINGPERFORMANCERISKS
  2. 2. INTRODUCTION
  3. 3. OBJECTIVEPut working software into production as quickly as possible,whilst minimising risk of load-related problems:› Bad response times› Too small capacity› Availability too low› Excessive system resource use
  4. 4. PREVENTING RISK IS A BIG SUBJECT,WHAT FOLLOWS IS TAKEN FROM OUR EXPERIENCERISK PREVENTION IS A BIG SUBJECT Photo by chillihead: www.flickr.com/photos/chillihead/1778980935
  5. 5. CONTINUOUS DELIVERY LITERATURE PROVIDES METHODSTHAT HELP REDUCE RISK› Blue-green deployments› Dark launching› Feature toggles› Canary releasing› Production immune systems Jez Humble, http://continuousdelivery.com
  6. 6. BLUE-GREEN DEPLOYMENTS Elastic Instances Load Balancer Version n Amazon Route 53 Elastic Instances Load Balancer Version n+1
  7. 7. DARK LAUNCHING Web page DB
  8. 8. DARK LAUNCHING Web page DB Weather SP
  9. 9. DARK LAUNCHING Web page DB Weather SP
  10. 10. FEATURE TOGGLES
  11. 11. CANARY RELEASING 0% 100%
  12. 12. PRODUCTION IMMUNE SYSTEMS
  13. 13. USE CONTROLLED LOAD TESTING TO HELP CAPACITYPLANNING Instance RDS DB Instance Amazon Elastic Instance Route 53 Load Balancer Instance RDS DB Instance Read Replica
  14. 14. WORK WITH FAILURE› Optimise for MTTD and MTTR, not MTBF› Game day exercises› Chaos monkey› Go / NoGo meetings› Retrospectives
  15. 15. BUT LEGACY SYSTEMS OFTEN LACK THE REQUIREDRESILIENCE
  16. 16. WHILE WE WORK ON OUR RESILIENCE, WE USE LOAD TESTSTO HELP IDENTIFY THE BIGGEST RISKS
  17. 17. PRE-PROD LOAD TESTING IS NOT FREE› Extra code to maintain› Usually test runs last several hours› A production-like environment is expensive› Realistic testing is hard› Not all developers like writing (performance) tests
  18. 18. USE IT WISELY, WHERE PRODUCTION TESTING IS STILLINAPPROPRIATE› It provides no guarantee› Use it to find any showstoppers you can› Essentially, an optional service that teams can use
  19. 19. USE IT AS A PLAYGROUND TO TRY RISKY CHANGES Photo by vastateparksstaff: www.flickr.com/photos/vastateparksstaff/5330257235
  20. 20. Load tests FunctionalBuild, unit integration test, etc. testsVery often Less often At least once a day (at night)
  21. 21. Load tests FunctionalBuild, unit Load test integration test, etc. script check testsVery often Less often At least once a day (at night)
  22. 22. THE AIM IS NOT PERFECTION, GO FOR “AS REALISTIC ASNEEDED”
  23. 23. SET UP TEST DATA IN THE WEEKEND, TO MINIMIZEDISRUPTION
  24. 24. WHEN IS A PROBLEM REALLY A PROBLEM?
  25. 25. FIND AN OBJECTIVE WAY TO JUDGE YOUR FINDINGS
  26. 26. ESTABLISH REQUIREMENTS TO MAKE CLEAR WHAT ISACCEPTABLE› Seen from the main stakeholders’ perspective – Response time: users – System resources: ops – Capacity: business› Specific› Measurable› Achievable› Relevant
  27. 27. Concurrent users Fail: Now: Target: < 100k 150k 200kIntention: The website should at least be Stakeholder: Businessable to manage our typical daily load, butwe would like some margin for growthand marketing campaigns.Scale: Maximum load in a day, while Meter: Session table row count.response times are still according tospec.
  28. 28. SO USE A REAL BROWSER TO TESTA REAL USER’S EXPERIENCE
  29. 29. Response time Fail [Today] TargetHomepage.FV > 6 sec 3.9 sec 2 secHomepage.RV > 5 sec 2.8 sec 1 secCheckout.FV > 8 sec 6.5 sec 2 secDetails.FV > 6 sec 1.9 sec 2 secDetails.RV > 5 sec 1.7 sec 1 secSearch.FV > 6 sec 4.8 sec 2 secSearch.RV > 5 sec 3.7 sec 1 secCart.FV > 6 sec 4.4 sec 2 secCart.RV > 5 sec 3.4 sec 1 secLoginForm.FV > 6 sec 3.5 sec 2 secLoginForm.RV > 5 sec 2.5 sec 1 sec
  30. 30. TO MAKE COMPARING SENSIBLE, MAKE YOUR TESTSDETERMINISTICStub systems that you have no control over
  31. 31. LOAD TESTING SHOULD BE OPTIONAL, THE ONLY THINGTHAT COUNTS IS PRODUCTION!› Your definition of done should reflect that› The aim is to get early feedback from a safe environment
  32. 32. ANYTHING YOU FIND IS AN OPPORTUNITY TO FIX MORETHAN ONE PROBLEM
  33. 33. SO WHAT MONITORING IS TYPICALLY NEEDED?› Be able to localise where latency is coming from! – For every system, all incoming and outgoing calls (count and time spent stats)› Finite resources (pools, CPU, I/O, etc.)› Number of active users› Response size, where possible› Add whatever you needIt should be identical on all environments!
  34. 34. CONCLUSIONIn order to put code live without pre-prod load testing, at leastthe following need to be in place:› Culture› State-of-the-art monitoring› ResilienceWithout these, support your continuous delivery process withoptional load tests and strong specs.Use the load tests to identify some pain points, so you canmodify the code and add monitoring, making it safer to do(incremental) dark releases and canary testing in production.
  35. 35. QUESTIONS? athomas@xebia.com @a32an www.xebia.com blog.xebia.com (we’re hiring)
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×