CONTINUOUSDELIVERY WHILEMINIMISINGPERFORMANCERISKS
INTRODUCTION
OBJECTIVEPut working software into production as quickly as possible,whilst minimising risk of load-related problems:›   B...
PREVENTING RISK IS A BIG SUBJECT,WHAT FOLLOWS IS TAKEN FROM OUR EXPERIENCERISK PREVENTION IS A BIG SUBJECT     Photo by ch...
CONTINUOUS DELIVERY LITERATURE PROVIDES METHODSTHAT HELP REDUCE RISK›   Blue-green deployments›   Dark launching›   Featur...
BUT LEGACY SYSTEMS OFTEN LACK THE REQUIREDRESILIENCE
WHILE WE WORK ON OUR RESILIENCE, WE USE LOAD TESTSTO HELP IDENTIFY THE BIGGEST RISKS
PRE-PROD LOAD TESTING IS NOT FREE›   Extra code to maintain›   Usually test runs last several hours›   A production-like e...
USE IT WISELY, WHERE PRODUCTION TESTING IS STILLINAPPROPRIATE›   It provides no guarantee›   Use it to find any showstoppe...
USE IT AS A PLAYGROUND TO TRY RISKY CHANGES        Photo by vastateparksstaff: www.flickr.com/photos/vastateparksstaff/533...
Load tests              FunctionalBuild, unit              integration test, etc.                 testsVery often    Less ...
Load tests              FunctionalBuild, unit                      Load test              integration test, etc.          ...
THE AIM IS NOT PERFECTION, GO FOR “AS REALISTIC ASNEEDED”
SET UP TEST DATA IN THE WEEKEND, TO MINIMIZEDISRUPTION
WHEN IS A PROBLEM REALLY A PROBLEM?
FIND AN OBJECTIVE WAY TO JUDGE YOUR FINDINGS
ESTABLISH REQUIREMENTS TO MAKE CLEAR WHAT ISACCEPTABLE›   Seen from the main stakeholders’ perspective    – Response time:...
Concurrent users                             Fail:              Now:            Target:                             < 100k...
FOR RESPONSE TIMES TOO, FOCUS ON THEMAIN STAKEHOLDER! FOR RESPONSE TIMES TOO, FOCUS ON THE MAIN STAKEHOLDER!
SO USE A REAL BROWSER TO TESTA REAL USER’S EXPERIENCE
Response time   Fail      [Today]    TargetHomepage.FV     > 6 sec    3.9 sec    2 secHomepage.RV     > 5 sec    2.8 sec  ...
TO MAKE COMPARING SENSIBLE, MAKE YOUR TESTSDETERMINISTICStub systems that you have no control over
LOAD TESTING SHOULD BE OPTIONAL, THE ONLY THINGTHAT COUNTS IS PRODUCTION!›   Your definition of done should reflect that› ...
ANYTHING YOU FIND IS AN OPPORTUNITY TO FIX MORETHAN ONE PROBLEM
SO WHAT MONITORING IS TYPICALLY NEEDED?›   Be able to localise where latency is coming from!    – For every system, all in...
SET CLEAR TARGETS, SO YOU KNOW YOUR SITUATION›   How many errors would be OK in production?›   What kind of errors do we c...
Number of stale server session requests /                          min              50        0                     100   ...
CONCLUSIONSupport your continuous delivery process with optional loadtests and strong specs.Use the load tests to identify...
QUESTIONS?             athomas@xebia.com             @a32an             www.xebia.com             blog.xebia.com          ...
Continuous delivery while minimizing performance risks
Continuous delivery while minimizing performance risks
Upcoming SlideShare
Loading in...5
×

Continuous delivery while minimizing performance risks

317

Published on

These are the slides to my talk at Velocity Europe 2012: http://velocityconf.com/velocityeu2012/public/schedule/detail/26162

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
317
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
7
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Continuous delivery while minimizing performance risks

  1. 1. CONTINUOUSDELIVERY WHILEMINIMISINGPERFORMANCERISKS
  2. 2. INTRODUCTION
  3. 3. OBJECTIVEPut working software into production as quickly as possible,whilst minimising risk of load-related problems:› Bad response times› Too small capacity› Availability too low› Excessive system resource use
  4. 4. PREVENTING RISK IS A BIG SUBJECT,WHAT FOLLOWS IS TAKEN FROM OUR EXPERIENCERISK PREVENTION IS A BIG SUBJECT Photo by chillihead: www.flickr.com/photos/chillihead/1778980935
  5. 5. CONTINUOUS DELIVERY LITERATURE PROVIDES METHODSTHAT HELP REDUCE RISK› Blue-green deployments› Dark launching› Feature toggles› Canary releasing› Production immune systems Jez Humble, http://continuousdelivery.com
  6. 6. BUT LEGACY SYSTEMS OFTEN LACK THE REQUIREDRESILIENCE
  7. 7. WHILE WE WORK ON OUR RESILIENCE, WE USE LOAD TESTSTO HELP IDENTIFY THE BIGGEST RISKS
  8. 8. PRE-PROD LOAD TESTING IS NOT FREE› Extra code to maintain› Usually test runs last several hours› A production-like environment is expensive› Realistic testing is hard› Not all developers like writing (performance) tests
  9. 9. USE IT WISELY, WHERE PRODUCTION TESTING IS STILLINAPPROPRIATE› It provides no guarantee› Use it to find any showstoppers you can› Essentially, an optional service that teams can use
  10. 10. USE IT AS A PLAYGROUND TO TRY RISKY CHANGES Photo by vastateparksstaff: www.flickr.com/photos/vastateparksstaff/5330257235
  11. 11. Load tests FunctionalBuild, unit integration test, etc. testsVery often Less often About once a day (at night)
  12. 12. Load tests FunctionalBuild, unit Load test integration test, etc. script check testsVery often Less often About once a day (at night)
  13. 13. THE AIM IS NOT PERFECTION, GO FOR “AS REALISTIC ASNEEDED”
  14. 14. SET UP TEST DATA IN THE WEEKEND, TO MINIMIZEDISRUPTION
  15. 15. WHEN IS A PROBLEM REALLY A PROBLEM?
  16. 16. FIND AN OBJECTIVE WAY TO JUDGE YOUR FINDINGS
  17. 17. ESTABLISH REQUIREMENTS TO MAKE CLEAR WHAT ISACCEPTABLE› Seen from the main stakeholders’ perspective – Response time: users – System resources: ops – Capacity: business› Specific› Measurable› Achievable› Relevant
  18. 18. Concurrent users Fail: Now: Target: < 100k 150k 200kIntention: The website should at least be Stakeholder: Businessable to manage our typical daily load, butwe would like some margin for growthand marketing campaigns.Scale: Maximum load in a day, while Meter: Session table row count.response times are still according tospec.
  19. 19. FOR RESPONSE TIMES TOO, FOCUS ON THEMAIN STAKEHOLDER! FOR RESPONSE TIMES TOO, FOCUS ON THE MAIN STAKEHOLDER!
  20. 20. SO USE A REAL BROWSER TO TESTA REAL USER’S EXPERIENCE
  21. 21. Response time Fail [Today] TargetHomepage.FV > 6 sec 3.9 sec 2 secHomepage.RV > 5 sec 2.8 sec 1 secCheckout.FV > 8 sec 6.5 sec 2 secDetails.FV > 6 sec 1.9 sec 2 secDetails.RV > 5 sec 1.7 sec 1 secSearch.FV > 6 sec 4.8 sec 2 secSearch.RV > 5 sec 3.7 sec 1 secCart.FV > 6 sec 4.4 sec 2 secCart.RV > 5 sec 3.4 sec 1 secLoginForm.FV > 6 sec 3.5 sec 2 secLoginForm.RV > 5 sec 2.5 sec 1 sec
  22. 22. TO MAKE COMPARING SENSIBLE, MAKE YOUR TESTSDETERMINISTICStub systems that you have no control over
  23. 23. LOAD TESTING SHOULD BE OPTIONAL, THE ONLY THINGTHAT COUNTS IS PRODUCTION!› Your definition of done should reflect that› The aim is to get early feedback from a safe environment
  24. 24. ANYTHING YOU FIND IS AN OPPORTUNITY TO FIX MORETHAN ONE PROBLEM
  25. 25. SO WHAT MONITORING IS TYPICALLY NEEDED?› Be able to localise where latency is coming from! – For every system, all incoming and outgoing calls (count and time spent stats)› Finite resources (pools, CPU, I/O, etc.)› Number of active users› Response size, where possible› Add whatever you needIt should be identical on all environments!
  26. 26. SET CLEAR TARGETS, SO YOU KNOW YOUR SITUATION› How many errors would be OK in production?› What kind of errors do we care about?
  27. 27. Number of stale server session requests / min 50 0 100 150 250 300 20000:0001:0002:0003:0004:0005:0006:0007:0008:0009:0010:0011:0012:0013:00 Other servers taken out of LB14:0015:0016:0017:0018:0019:0020:0021:00 back in LB22:00 Other servers23:0000:00
  28. 28. CONCLUSIONSupport your continuous delivery process with optional loadtests and strong specs.Use the load tests to identify some pain points, so you canmodify the code and add monitoring, making it safer to dodark releases and canary testing in production.Constantly ask yourself: what would it take to only do this inproduction? Is it adding value?
  29. 29. QUESTIONS? athomas@xebia.com @a32an www.xebia.com blog.xebia.com (we’re hiring)
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×