OBJECTIVEPut working software into production as quickly as possible,whilst minimising risk of load-related problems:› Bad response times› Too small capacity› Availability too low› Excessive system resource use
PREVENTING RISK IS A BIG SUBJECT,WHAT FOLLOWS IS TAKEN FROM OUR EXPERIENCERISK PREVENTION IS A BIG SUBJECT Photo by chillihead: www.flickr.com/photos/chillihead/1778980935
CONTINUOUS DELIVERY LITERATURE PROVIDES METHODSTHAT HELP REDUCE RISK› Blue-green deployments› Dark launching› Feature toggles› Canary releasing› Production immune systems Jez Humble, http://continuousdelivery.com
BLUE-GREEN DEPLOYMENTS Elastic Instances Load Balancer Version n Amazon Route 53 Elastic Instances Load Balancer Version n+1
USE CONTROLLED LOAD TESTING TO HELP CAPACITYPLANNING Instance RDS DB Instance Amazon Elastic Instance Route 53 Load Balancer Instance RDS DB Instance Read Replica
WORK WITH FAILURE› Optimise for MTTD and MTTR, not MTBF› Game day exercises› Chaos monkey› Go / NoGo meetings› Retrospectives
BUT LEGACY SYSTEMS OFTEN LACK THE REQUIREDRESILIENCE
WHILE WE WORK ON OUR RESILIENCE, WE USE LOAD TESTSTO HELP IDENTIFY THE BIGGEST RISKS
PRE-PROD LOAD TESTING IS NOT FREE› Extra code to maintain› Usually test runs last several hours› A production-like environment is expensive› Realistic testing is hard› Not all developers like writing (performance) tests
USE IT WISELY, WHERE PRODUCTION TESTING IS STILLINAPPROPRIATE› It provides no guarantee› Use it to find any showstoppers you can› Essentially, an optional service that teams can use
USE IT AS A PLAYGROUND TO TRY RISKY CHANGES Photo by vastateparksstaff: www.flickr.com/photos/vastateparksstaff/5330257235
Load tests FunctionalBuild, unit integration test, etc. testsVery often Less often At least once a day (at night)
Load tests FunctionalBuild, unit Load test integration test, etc. script check testsVery often Less often At least once a day (at night)
THE AIM IS NOT PERFECTION, GO FOR “AS REALISTIC ASNEEDED”
SET UP TEST DATA IN THE WEEKEND, TO MINIMIZEDISRUPTION
ESTABLISH REQUIREMENTS TO MAKE CLEAR WHAT ISACCEPTABLE› Seen from the main stakeholders’ perspective – Response time: users – System resources: ops – Capacity: business› Specific› Measurable› Achievable› Relevant
Concurrent users Fail: Now: Target: < 100k 150k 200kIntention: The website should at least be Stakeholder: Businessable to manage our typical daily load, butwe would like some margin for growthand marketing campaigns.Scale: Maximum load in a day, while Meter: Session table row count.response times are still according tospec.
SO USE A REAL BROWSER TO TESTA REAL USER’S EXPERIENCE
TO MAKE COMPARING SENSIBLE, MAKE YOUR TESTSDETERMINISTICStub systems that you have no control over
LOAD TESTING SHOULD BE OPTIONAL, THE ONLY THINGTHAT COUNTS IS PRODUCTION!› Your definition of done should reflect that› The aim is to get early feedback from a safe environment
ANYTHING YOU FIND IS AN OPPORTUNITY TO FIX MORETHAN ONE PROBLEM
SO WHAT MONITORING IS TYPICALLY NEEDED?› Be able to localise where latency is coming from! – For every system, all incoming and outgoing calls (count and time spent stats)› Finite resources (pools, CPU, I/O, etc.)› Number of active users› Response size, where possible› Add whatever you needIt should be identical on all environments!
CONCLUSIONIn order to put code live without pre-prod load testing, at leastthe following need to be in place:› Culture› State-of-the-art monitoring› ResilienceWithout these, support your continuous delivery process withoptional load tests and strong specs.Use the load tests to identify some pain points, so you canmodify the code and add monitoring, making it safer to do(incremental) dark releases and canary testing in production.