Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

of

TestCon Vilnius 2017 - Stating the obvious Slide 1 TestCon Vilnius 2017 - Stating the obvious Slide 2 TestCon Vilnius 2017 - Stating the obvious Slide 3 TestCon Vilnius 2017 - Stating the obvious Slide 4 TestCon Vilnius 2017 - Stating the obvious Slide 5 TestCon Vilnius 2017 - Stating the obvious Slide 6 TestCon Vilnius 2017 - Stating the obvious Slide 7 TestCon Vilnius 2017 - Stating the obvious Slide 8 TestCon Vilnius 2017 - Stating the obvious Slide 9 TestCon Vilnius 2017 - Stating the obvious Slide 10 TestCon Vilnius 2017 - Stating the obvious Slide 11 TestCon Vilnius 2017 - Stating the obvious Slide 12 TestCon Vilnius 2017 - Stating the obvious Slide 13 TestCon Vilnius 2017 - Stating the obvious Slide 14 TestCon Vilnius 2017 - Stating the obvious Slide 15 TestCon Vilnius 2017 - Stating the obvious Slide 16 TestCon Vilnius 2017 - Stating the obvious Slide 17 TestCon Vilnius 2017 - Stating the obvious Slide 18 TestCon Vilnius 2017 - Stating the obvious Slide 19 TestCon Vilnius 2017 - Stating the obvious Slide 20 TestCon Vilnius 2017 - Stating the obvious Slide 21 TestCon Vilnius 2017 - Stating the obvious Slide 22 TestCon Vilnius 2017 - Stating the obvious Slide 23 TestCon Vilnius 2017 - Stating the obvious Slide 24 TestCon Vilnius 2017 - Stating the obvious Slide 25 TestCon Vilnius 2017 - Stating the obvious Slide 26 TestCon Vilnius 2017 - Stating the obvious Slide 27 TestCon Vilnius 2017 - Stating the obvious Slide 28 TestCon Vilnius 2017 - Stating the obvious Slide 29 TestCon Vilnius 2017 - Stating the obvious Slide 30 TestCon Vilnius 2017 - Stating the obvious Slide 31 TestCon Vilnius 2017 - Stating the obvious Slide 32 TestCon Vilnius 2017 - Stating the obvious Slide 33 TestCon Vilnius 2017 - Stating the obvious Slide 34 TestCon Vilnius 2017 - Stating the obvious Slide 35 TestCon Vilnius 2017 - Stating the obvious Slide 36 TestCon Vilnius 2017 - Stating the obvious Slide 37 TestCon Vilnius 2017 - Stating the obvious Slide 38 TestCon Vilnius 2017 - Stating the obvious Slide 39
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

0 Likes

Share

Download to read offline

TestCon Vilnius 2017 - Stating the obvious

Download to read offline

Session "Stating the obvious: adding performance and scalability tests to a Continuous Integration pipeline" at TestCon Vilnius 2017

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to like this

TestCon Vilnius 2017 - Stating the obvious

  1. 1. Stating the obvious Adding performance/load/stress testing to your Continuous Integration pipeline Giulio Vian giulio@nolbej.com @giulio_vian http://blog.casavian.eu/
  2. 2. Agenda Why obvious? CI / CD Integration Perf / load/ stress testing basics Closing 2
  3. 3. This Session 200 level Tight on time: avoid open questions 3
  4. 4. I. Why obvious? 5
  5. 5. Performance is a Feature «there are two kinds of websites: the quick and the dead» https://blog.codinghorror.com/performance-is-a-feature/
  6. 6. Response time 0.1 second is about the limit for having the user feel that the system is reacting instantaneously, meaning that no special feedback is necessary except to display the result. 1.0 second is about the limit for the user's flow of thought to stay uninterrupted, even though the user will notice the delay. Normally, no special feedback is necessary during delays of more than 0.1 but less than 1.0 second, but the user does lose the feeling of operating directly on the data. 10 seconds is about the limit for keeping the user's attention focused on the dialogue. For longer delays, users will want to perform other tasks while waiting for the computer to finish, so they should be given feedback indicating when the computer expects to be done. Feedback during the delay is especially important if the response time is likely to be highly variable, since users will then not know what to expect. Miller, R. B. (1968). Response time in man- computer conversational transactions. Proc. AFIPS Fall Joint Computer Conference Vol. 33, 267- 277.
  7. 7. Hard facts Google Half a second delay caused a 20% drop in traffic. Amazon.com Even very small delays (100 ms) would result in substantial and costly drops in revenue.
  8. 8. Why lubricant in an engine? Do you like the motor to seize?
  9. 9. Always necessary? Gartner Hype Cycle, Source: Wikipedia
  10. 10. My perf test of the year 2M users 40,000 RPS 2Gbps There’s Nothing Like Production 11 © 2016 IMG Universe, LLC. All Rights Reserved
  11. 11. II. CI / CD Integration 12
  12. 12. Tooling Scripts & Runners Load generators Monitoring
  13. 13. Tools: Script & Runners JMeter Visual Studio (.webtest / .loadtest) Selenium ×Unit family Gatling *Custom & More…
  14. 14. Tools: Load VSTS Cloud-based load testing CA BlazeMeter SOASTA (Akamai) CloudTest HPE LoadRunner *Custom & More…
  15. 15. Tools: Monitoring Loading tool Basic Performance Counters* Application logs Cloud & More…
  16. 16. Analyzing
  17. 17. High level architecture You Load Generator SUT Monitoring CI / CD pipeline
  18. 18. Web Load Testing 19
  19. 19. Considerations for CI/CD Trunk or branch of pipeline Resources KPIs
  20. 20. Time factor Short-running Tests Long-running Tests Test Coverage Frequency of execution Value from test
  21. 21. Hidden Gremlins Cloud infrastructure warm up ELB Default configuration nginx worker processes Client resources
  22. 22. III. Perf / load/ stress testing basics 23
  23. 23. (Technical) Goals Benchmarking Force defects to emerge code, configuration, architecture Capacity planning Breaking point Security leakage
  24. 24. Emerging Problems Connection pool exhaustion File locks Database locks (Thread) deadlocks Memory exahustion Domino Effect Lack of scalability
  25. 25. Indicators Latency Throughput Load Error rate Tool (client) errors
  26. 26. Compound graph +Error rates +Tools errors Relative Load Latency Required threshold Max N seconds 100%60% Throughput Usage
  27. 27. How you read this? Relative Load Latency Required threshold Max N seconds 100%60% Throughput Usage
  28. 28. Analyze Response curve Inflection point Bottlenecks
  29. 29. Common scenarios Desktop / mobile / library Server / web Database server Web serverClient N1 N4 A1 A3 A2N3 N2
  30. 30. What to monitor CPU RAM Disk I/O Network I/O Threads / processes
  31. 31. IV. Closing 32
  32. 32. Resources Sisyphus Choosing to invest 33 (Photo: Elya)
  33. 33. Bibliografy & References http://www.slideshare.net/giuliov/presentations https://github.com/giuliov/Stating-the-obvious/ https://docs.microsoft.com/en-us/vsts/build-release/tasks/test/cloud- based-load-test AWS Well-Architected Framework - Performance Efficiency Pillar https://www.amazon.com/dp/B01MSSLHBX Performance Testing Guidance for Web Applications https://msdn.microsoft.com/en-us/library/bb924375.aspx http://www.brendangregg.com/linuxperf.html 34
  34. 34. Bibliografy (2) Writing High-Performance .NET Code — Ben Watson (Ben Watson) https://www.amazon.it/Writing-High- Performance-NET-Code- Watson/dp/0990583430/ Time Is Money: The Business Value of Web Performance — Tammy Everts (O'Reilly Media) https://www.amazon.com/Time-Money- Business-Value-Performance/dp/1491928743 35
  35. 35. Bibliografy (3) Software Performance and Scalability: A Quantitative Approach — Henry H. Liu (Wiley) https://www.amazon.com/Software- Performance-Scalability-Quantitative- Approach/dp/0470462531 Continuous Delivery with Windows and .NET — Matthew Skelton and Chris O'Dell (O'Reilly) http://www.oreilly.com/webops- perf/free/continuous-delivery-with-windows-and- net.csp 36
  36. 36. Bibliografy (4) Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation — J.Humble, D.Farley (Addison-Wesley) https://www.amazon.com/Continuous- Delivery/dp/0321601912/ The DevOps Handbook — G.Kim, P.Debois, J.Willis, J.Humble (IT Revolution Press) https://www.amazon.com/DevOps-Handbook- World-Class-Reliability- Organizations/dp/1942788002/ 37
  37. 37. About me 38 giulio@nolbej.com @giulio_vian http://blog.casavian.eu/
  38. 38. Call to action 39 (Photo: Francesco Canu)
  39. 39. End of transmission 40

Session "Stating the obvious: adding performance and scalability tests to a Continuous Integration pipeline" at TestCon Vilnius 2017

Views

Total views

184

On Slideshare

0

From embeds

0

Number of embeds

0

Actions

Downloads

5

Shares

0

Comments

0

Likes

0

×