Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

0

Share

Download to read offline

PAC 2019 virtual Federico Toledo

Download to read offline

Adding Performance Verifications in Continuous Delivery

Related Books

Free with a 30 day trial from Scribd

See all
  • Be the first to like this

PAC 2019 virtual Federico Toledo

  1. 1. Adding Performance Verifications in Continuous Delivery by Federico Toledo (COO at Abstracta) federico@abstracta.us @fltoledo
  2. 2. WHO IS ABSTRACTA? Abstracta is a world leader in software testing focused on harnessing automation to improve performance and reduce time to market. Our mission is to make the world a better place by co-creating winning technologies of the highest quality. 4 Offices Worldwide 80 Software Quality Engineers 10 Years in Business 95% Client Retention Rate +300 Projects Thought Leaders Testing Educators
  3. 3. www.abstracta.us Agenda Introduction to the typical approach to performance testing Why it does not work for continuous delivery Conclusion A different approach with a different goal 21 3 4
  4. 4. Introduction 1
  5. 5. www.abstracta.us vs Response Times Resource Consumption Performance
  6. 6. What is performance testing about?
  7. 7. www.abstracta.us Source:http://www.miniatur-wunderland.de/ Simulation
  8. 8. www.abstracta.us Measurement
  9. 9. www.abstracta.us Finding the Bottlenecks and the Breaking Point
  10. 10. www.abstracta.us Types of Performance Testing
  11. 11. www.abstracta.us Load Simulation Tools
  12. 12. www.abstracta.us } Simulation
  13. 13. www.abstracta.us Simulation
  14. 14. www.abstracta.us HTTP HTTP Server Simulation
  15. 15. www.abstracta.us Demo Simple DEMO Using JMeter Typical approach: • Define bunch of test cases (user flows) and load scenario. • Automate them with the tool (http level). • Run tests, analyze, work with devs and ops to improve. • Continue until you reach the SLAs or good enough performance (response times and resource consumption).
  16. 16. www.abstracta.us What’s special about Continuous Delivery? We want to enable the business people to deliver the product to the customers as soon as they want. Continuous Integration: merge code at least once a day. Continuous Deployment: deliver continuously to users. Continuous Delivery: deliver continuously to business people, frequently to users (production). 1 2 3 We need an automated pipeline with different checks. Let’s add performance validations to this pipeline!
  17. 17. Do you think that the typical approach for performance testing would work? 2
  18. 18. www.abstracta.us Issues with the typical approach for CI/CD The pipeline will run at least once a day (probably more frequently) End-to-End scripts are fragile (need lots of maintenance) If there is an issue or degradation, it’s hard to find the cause 1 2
  19. 19. What can we do differently? Different approach with different goals 3
  20. 20. www.abstracta.us 1 2 A different approach with a different goal Frequent releases to production implies: • Smaller difference between versions. • Reduced risk (less uncertainty). Detect any degradation in the system as soon as possible. Typical goal: Know if the system will support the expected load. Goal for performance tests in Continuous Delivery:
  21. 21. www.abstracta.us Definition of the test cases to automate Detecting degradations as soon as possible Selection of the tool Design of the load scenarios 1 2 3
  22. 22. www.abstracta.us We love JMeter, but...
  23. 23. www.abstracta.us We love JMeter, but...
  24. 24. www.abstracta.us We love JMeter, but...
  25. 25. www.abstracta.us Understanding what changed
  26. 26. www.abstracta.us Takeaways A Choose a CI-friendly tool. www.abstracta.us
  27. 27. www.abstracta.us Let’s talk about the test scripts What is the simplest test we can run in order to detect degradations? www.abstracta.us
  28. 28. www.abstracta.us What is the simplest test we can run in order to detect degradations? Hint: We want to avoid false negatives. Specially because you will run these tests on a daily basis. End-to-End tests (http level) are fragile. We want to test early and frequently. Another hint:Remember:
  29. 29. www.abstracta.us Cohn’s Pyramid on Test Automation Equivalent to: HTTP traffic between user and server. Combining different endpoints. Test endpoints in isolation.
  30. 30. www.abstracta.us Cohn’s Pyramid on Test Automation User metrics User centered design More costly (scripting, infrastructure) Cheaper (easier) Early results More Frequent No idea about the user experience
  31. 31. www.abstracta.us What we want to do: Let’s add unit performance tests to our pipeline This way we discover automatically any degradation of the system as soon as possible. • Independent tests. • Each one running against one endpoint at a time.
  32. 32. www.abstracta.us Detect degradations right after they are introduced Requestspersecond 0 50.0 100.0 150.0 200.0 250.0 300.0 09/12 09/16 09/20 09/24 09/28 10/02 10/06 Means requests per second Performance assert threshold Release branch created Percent KOs 0 25 50 75 100 PercentageKOs
  33. 33. www.abstracta.us Detect degradations right after they are introduced ResponseTimes 0 100 200 300 400 500 600 09/12 09/16 09/20 09/24 09/28 10/02 10/06 Average response times (ms) Performance assert threshold Release branch created Percent KOs 0 25 50 75 100 PercentageKOs
  34. 34. www.abstracta.us Define scenario and assertions
  35. 35. www.abstracta.us Define scenario and assertions
  36. 36. www.abstracta.us Define scenario and assertions
  37. 37. www.abstracta.us Define scenario and assertions
  38. 38. www.abstracta.us Define scenario and assertions
  39. 39. www.abstracta.us Define scenario and assertions
  40. 40. www.abstracta.us Define scenario and assertions Load ● 350 threads Assertions Given ● Exclusive environment ● < 1% error ● P95 Response Times < 130ms + 10% ● Throughput >= 150 TPS – 10%
  41. 41. www.abstracta.us You should also check Keptn and Pitometer
  42. 42. www.abstracta.us Takeaways A Choose a CI-friendly tool. www.abstracta.us B Unit performance tests to detect degradations as soon as possible.
  43. 43. Is that enough? 4
  44. 44. www.abstracta.us Fallacy of the composition Testing the parts is not enough to know how the whole will behave. You need to test the whole to see how the parts interacts. Parts Parts Parts Parts Whole
  45. 45. www.abstracta.us Testing Strategy Every month? Every week? Every day?
  46. 46. www.abstracta.us Testing Strategy Similar to production Scaled down
  47. 47. www.abstracta.uswww.abstracta.us Takeaways B Choose a CI-friendly tool. Unit performance tests to detect degradations as soon as possible. C Complement with integration and load tests. A
  48. 48. Is that enough? 4
  49. 49. www.abstracta.us What about the client side? Web - Google page speed - Website.io - APM - Etc! What about mobile?
  50. 50. www.abstracta.us Apptim (www.apptim.com)
  51. 51. www.abstracta.us Summary of my learnings www.abstracta.us B C Choose a CI-friendly tool. Unit performance tests to detect degradations as soon as possible. Complement with integration and load tests. Don’t forget the client side. A D
  52. 52. Adding Performance Verifications in Continuous Delivery by Federico Toledo (COO at Abstracta) federico@abstracta.us @fltoledo ¡Muchas gracias!

Adding Performance Verifications in Continuous Delivery

Views

Total views

86

On Slideshare

0

From embeds

0

Number of embeds

0

Actions

Downloads

5

Shares

0

Comments

0

Likes

0

×