Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Lars Wolff - Performance Testing for DevOps in the Cloud - Codemotion Amsterdam 2019

89 views

Published on

Performance tests are not only an important instrument for understanding a system and its runtime environment. It is also essential in order to check stability and scalability – non-functional requirements that might be decisive for success. But won't my cloud hosting service scale for me as long as I can afford it? Yes, but… It only operates and scales resources. It won't automatically make your system fast, stable and scalable. This talk shows how such and comparable questions can be clarified with performance tests and how DevOps teams benefit from regular test practise.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Lars Wolff - Performance Testing for DevOps in the Cloud - Codemotion Amsterdam 2019

  1. 1. Performance Testing for DevOps in the Cloud Lars Wolff Amsterdam | April 2-3, 2019
  2. 2. Introduction @larsvegas
 support@stormforger.com My Journey so far • Software engineering • Consulting and Agile coaching • Performance Testing • Ship it! • AWS User Group Cologne & AWS Community Days Germany
  3. 3. Performance Testing SaaS for DevOps Empower organizations to embrace
 continuous performance testing
  4. 4. Performance
  5. 5. Performance • Frontend Performance • Backend Performance
  6. 6. Performance The ability of a system to execute one task
 within a defined dimension = efficiency of a system
 ↪ 1 server manages 250 rps with p99 500ms.
  7. 7. Scalability
  8. 8. ≠ Performance Scalability
  9. 9. Scalability capacity 0 1000 2000 3000 4000 resources 5 10 15 20 25 30 Effectivity by which the capacity can be increased through adding resources
  10. 10. What makes me recognize that I have a performance problem? http://www.slideshare.net/jboner/scalability-availability-stability-patterns/15-How_do_I_know_if
  11. 11. If your system is slow for one single user.
  12. 12. What makes me recognize that I have a scaling problem? http://www.slideshare.net/jboner/scalability-availability-stability-patterns/15-How_do_I_know_if
  13. 13. If your system is fast for single users
 but slow under load.
  14. 14. The Cloud™ already scales for us 👍
  15. 15. Yes.
  16. 16. But. No.
  17. 17. scaling resources
 ≠ scaling applications
  18. 18. Create an understanding of your system, its cloud run time environment and behaviour
  19. 19. Communication Collaboration Automation Monitoring DevOps
  20. 20. PerformanceTesting
  21. 21. “In software engineering,
 performance testing is in general,
 a testing practice performed to
 determine how a system performs
 in terms of responsiveness and stability
 under a particular workload.”
 https://en.wikipedia.org/wiki/Software_performance_testing – Wikipedia
  22. 22. " 🔬 $
  23. 23. Performance Tests • Collection of 
 non-functional test methods • Not to be defined with 
 absolute selectivity • Different goals and perspectives Load Testing Stress Testing Spike Testing Soak Testing Endurance Testing Resilience Testing Configuration Testing Scalability Testing
  24. 24. Performance Tests? • Not only for testing releases • Evaluation of… • technology, • proof of concepts, • troubleshooting and debugging
  25. 25. less traffic
 ≠
 no performance problems 😉
  26. 26. How do you do it?
  27. 27. Performance Testing Maturity • no performance tests, at all 😅 • tests if problems occur during production 🔥🚨🔥 • tests (shortly) before large-scale (sales & marketing) campaigns ) • regular testing ) ) • tests as a regular, integrated DevOps instrument ) ) ) ) )
  28. 28. https://en.wikipedia.org/wiki/DevOps DevOps
  29. 29. Performance testing is hard
  30. 30. Challenges • organizational obstacles, e.g. buy-in
 (management & team) • testing goal & scope • visibility in your own systems • The right performance test tool for the job
 – (usually) no tooling problem • start! 😉 👍😁
  31. 31. Starting …but how?
  32. 32. Steps to Perf Tests™ • bring together the stakeholders + development 🤓 operations 🐜 QA 👷 product 💰 management 😱 marketing
  33. 33. Steps to Perf Tests™ Kick off for 15 Min
 – All Stakeholders Heads Up STATE
 We’ll NOW create a performance testing culture to make sure that we can hold our promise to the business/customer to be fast and available under different amount of usage/users/transactions.
 We so match performance and availability requirements to keep the business save. ASK
 Do you all agree? STATE
 OK, we start with testing a very easy use case to raise a baseline of numbers about performance and availability. This friday, it’ll take a 2 hrs timebox. + development 🤓 operations 🐜 QA 👷 product 💰 management 😱 marketing
  34. 34. Steps to Perf Tests™ Test Case • Create a first, simple Test Case • Q: Test Data? • Q: How much traffic?
  35. 35. Steps to Perf Tests™ Test Case • Ask Analytics
 for the hour of the highest number of users in a week. • Analytics: 
 We have 10.000 visits in this hour. The Users visits flower-list, one flower detail.
 30% check out and payment.
  36. 36. Steps to Perf Tests™ Test Case • Test Environment &
 Load Generator Setup • Since Infrastructure is code:
 Just start your test environment!
  37. 37. War Room
  38. 38. Steps to Perf Tests™ Run Test • 20 Minutes • Invite ALL stakeholders! • It’s about collaboration
  39. 39. Steps to Perf Tests™ Inspect Report • Response Time? • If errors occurred:
 Do you see these in your monitoring, too?
  40. 40. Steps to Perf Tests™ Meet and communicate to stakeholders: • ONE Statement like
 We aim for baseline numbers.
 We tested flower-list and flower-detail.
 We created 4x traffic of peak hour.
 We saw 20% of errors.
 But always responded in under 1000ms (p99). • Ask
 Are you happy? • Ask
 Whats’ next? Checkout? + development 🤓 operations 🐜 QA 👷 product 💰 management 😱 marketing
  41. 41. Steps to Perf Tests™ Test Case • Add Checkout
  42. 42. Iterate
  43. 43. 😎
  44. 44. Steps to Perf Tests™ Non-functional requirements • No errors! • Each request must be faster than 250ms p99th
  45. 45. Steps to Perf Tests™ Automate • Run test runs from any pipeline • Run automated checks of
 non-functional requirements
  46. 46. 😎😎
  47. 47. It’s a feedback loop Stakeholder Requirements Definition
 (test scenario) Execution
 (test case) Result Interpretation & Reporting Business Needs
  48. 48. Start creating a Community of practice Special Interest Group since performance testing is an orthogonal topic
  49. 49. Thanks! 😁 Lars Wolff @larsvegas support@stormforger.com https://stormforger.com Slides and more:
 http://bit.ly/codemotion2019
  50. 50. Performance Testing SaaS for DevOps Empower organizations to embrace
 continuous performance testing http://bit.ly/codemotion2019
  51. 51. Extras:
  52. 52. Solutions
  53. 53. Defining goals • Defining goals beforehand (why are we doing tests?) • Important!
 Always define goals (and requirements) from the specialist side ☝ • A goal can be very simple at first, e.g. “surviving Black Friday well” or “we want to become 20% faster”.
  54. 54. Where to start with testing? • defining System Under Test (SUT) • finding the right scope • perimeter/end-to-end tests • components/service tests
  55. 55. System Under Test • testing “from top” (end-to-end) or “sideward” against single services • for the compartment side, only integrative tests are interesting • testing single services for the purpose of trouble shooting (technical perspective)
  56. 56. Define requirements • make assumptions for non-functional requirements (NFR) • assistance via “baseline” performance tests and performance budgets • simple requirements might be: “deliver sites under 2sec”, “10k checkouts per hour”, … • also here: start simple, learn, improve, repeat! • do sanity checks!
  57. 57. + development 🤓 operations 🐜 QA scenarios • bouncing visitor • returning customer • walk-in customer • … 👷 product 💰 management 😱 marketing workloads • Black Friday peak hour • average peak hour • 3 hour fire sale • … Who? What?
  58. 58. Testing together • Especially before the first tests: involve all the stakeholders! • GO! • Present setup, assumptions and findings • Check or rework scenario sanity • Repeat!
  59. 59. Performance Budget Browser 🔥Firewall🔥 Load Balancer Webserver App Server Service A Service B … FACT-FinderFACT-FinderFACT-Finder MySQLMySQLMySQL ☁☁ MySQLMySQLCache ……
  60. 60. Testing Types and Strategies
  61. 61. Stress Testing • Tests with increased up to extreme load • Understanding system behavior under (over)load • Identify limits and capacities
  62. 62. Stress Testing procedure • Define quality criteria
 (OK / KO) • Gradually increase load
  63. 63. Stress Testing procedure • Monitor system behavior • Identify bottlenecks • Detect capacity per resource
  64. 64. Scalability Testing • How effective are more resources transformed into capacity? • Foundation for capacity planning and cost estimation
  65. 65. Scalability Testing capacity 0 275 550 825 1100 resources 5 10 15 20 25 30
  66. 66. Spike Testing • How does the system react during (extreme) peak loads? • Are we reacting fast enough? • Occasions: Marketing 😈, Reddit, Hacker News, …, TV shows
  67. 67. Soak Testing • How does the system behave under load during a long period of time? • (very) long load test • Are there any longtime effects? Memory leaks?
  68. 68. Configuration Testing • How does the behavior change if the configuration modifies? • Series of test executions • Investigate the impact of the environment on your own system
  69. 69. …in the cloud • Types of instances • Auto scaling settings • Throughput provisioning • Ideal usage of services?
  70. 70. …and in general… • Hypervisor • Operating system • Web & application server • Software configuration • Software dependencies
  71. 71. Availability & Resilience
 Testing
  72. 72. Availability & Resilience Testing • (Zero-Downtime) deployments under load • Changes in infrastructure • Error scenarios • Failover testing • => Chaos engineering

×