OBJECTIVEPut working software into production as quickly as possible,whilst minimising risk of load-related problems:› Bad response times› Too small capacity› Availability too low› Excessive system resource use
PREVENTING RISK IS A BIG SUBJECT,WHAT FOLLOWS IS TAKEN FROM OUR EXPERIENCERISK PREVENTION IS A BIG SUBJECT Photo by chillihead: www.flickr.com/photos/chillihead/1778980935
CONTINUOUS DELIVERY LITERATURE PROVIDES METHODSTHAT HELP REDUCE RISK› Blue-green deployments› Dark launching› Feature toggles› Canary releasing› Production immune systems Jez Humble, http://continuousdelivery.com
BUT LEGACY SYSTEMS OFTEN LACK THE REQUIREDRESILIENCE
WHILE WE WORK ON OUR RESILIENCE, WE USE LOAD TESTSTO HELP IDENTIFY THE BIGGEST RISKS
PRE-PROD LOAD TESTING IS NOT FREE› Extra code to maintain› Usually test runs last several hours› A production-like environment is expensive› Realistic testing is hard› Not all developers like writing (performance) tests
USE IT WISELY, WHERE PRODUCTION TESTING IS STILLINAPPROPRIATE› It provides no guarantee› Use it to find any showstoppers you can› Essentially, an optional service that teams can use
USE IT AS A PLAYGROUND TO TRY RISKY CHANGES Photo by vastateparksstaff: www.flickr.com/photos/vastateparksstaff/5330257235
Load tests FunctionalBuild, unit integration test, etc. testsVery often Less often About once a day (at night)
Load tests FunctionalBuild, unit Load test integration test, etc. script check testsVery often Less often About once a day (at night)
THE AIM IS NOT PERFECTION, GO FOR “AS REALISTIC ASNEEDED”
SET UP TEST DATA IN THE WEEKEND, TO MINIMIZEDISRUPTION
ESTABLISH REQUIREMENTS TO MAKE CLEAR WHAT ISACCEPTABLE› Seen from the main stakeholders’ perspective – Response time: users – System resources: ops – Capacity: business› Specific› Measurable› Achievable› Relevant
Concurrent users Fail: Now: Target: < 100k 150k 200kIntention: The website should at least be Stakeholder: Businessable to manage our typical daily load, butwe would like some margin for growthand marketing campaigns.Scale: Maximum load in a day, while Meter: Session table row count.response times are still according tospec.
FOR RESPONSE TIMES TOO, FOCUS ON THEMAIN STAKEHOLDER! FOR RESPONSE TIMES TOO, FOCUS ON THE MAIN STAKEHOLDER!
SO USE A REAL BROWSER TO TESTA REAL USER’S EXPERIENCE
TO MAKE COMPARING SENSIBLE, MAKE YOUR TESTSDETERMINISTICStub systems that you have no control over
LOAD TESTING SHOULD BE OPTIONAL, THE ONLY THINGTHAT COUNTS IS PRODUCTION!› Your definition of done should reflect that› The aim is to get early feedback from a safe environment
ANYTHING YOU FIND IS AN OPPORTUNITY TO FIX MORETHAN ONE PROBLEM
SO WHAT MONITORING IS TYPICALLY NEEDED?› Be able to localise where latency is coming from! – For every system, all incoming and outgoing calls (count and time spent stats)› Finite resources (pools, CPU, I/O, etc.)› Number of active users› Response size, where possible› Add whatever you needIt should be identical on all environments!
SET CLEAR TARGETS, SO YOU KNOW YOUR SITUATION› How many errors would be OK in production?› What kind of errors do we care about?
Number of stale server session requests / min 50 0 100 150 250 300 20000:0001:0002:0003:0004:0005:0006:0007:0008:0009:0010:0011:0012:0013:00 Other servers taken out of LB14:0015:0016:0017:0018:0019:0020:0021:00 back in LB22:00 Other servers23:0000:00
CONCLUSIONSupport your continuous delivery process with optional loadtests and strong specs.Use the load tests to identify some pain points, so you canmodify the code and add monitoring, making it safer to dodark releases and canary testing in production.Constantly ask yourself: what would it take to only do this inproduction? Is it adding value?