How to Get Automatic Analysis for Load Test Results
•Download as PPTX, PDF•
1 like•391 views
The ability to fully automate your results analysis is vital in today’s Continuous Integration and Continuous Deployment era. Agile practices like Microservices exacerbate this need, giving you tens of services to test, ten times a day! Analysis by the human eye is impossible - you need automation. But automating the analysis is extremely difficult due to the ever-increasing complexities of load testing processes and timeline based reports. As such, contemporary load testing tools and services offer excellent ways to present reports but fall short when it comes to the analysis. In this presentation, Andrey Pokhilko (founder of JMeter-plugins.org and Loadosophia) explores how to take automatic result analysis and decision making to a new level. Join us and discuss: • Why is it tough to fully automate analysis and decision making on test results • How humans analyze the test in practice - which KPIs they look at • Which decisions can be made automatically during test execution • Which facts can be automatically concluded post-test • Practical results from several months of method application
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to How to Get Automatic Analysis for Load Test Results
Similar to How to Get Automatic Analysis for Load Test Results (20)
Recently uploaded
Recently uploaded (20)
How to Get Automatic Analysis for Load Test Results
- 1. Automatic Analysis for Load Test Results by Andrey Pokhilko Chief Scientist, BlazeMeter Inc. February 3rd, 2016 Mountain View, CA
- 2. About Me My key interest: tools that help people enjoy their job Current: Chief Scientist for BlazeMeter Past: Performance Engineering in Yandex JMeter Plugins project founder http://jmeter-plugins.org/ Loadosophia author http://loadosophia.org/ Taurus Tool lead dev http://gettaurus.org/
- 3. Look Out of the Window 1. It’s the era of total speed-up 2. Which is solved by total automation 3. Which leads to CI/CD as standard practices You can’t have a human in the loop anymore
- 4. Understand how humans do it, then automate the process
- 5. Analysis Raw Data Information DecisionsFacts True Analysis is Here
- 6. What is our “Load Test Results Analysis” ? 1. Decide if results are reliable 2. Break complex test timeline into components - surprisingly important 3. Decide if it had hit bottleneck, had memory leak etc 4. Understand APM correlations 5. Decide if something has changed since the last test / over recent N tests
- 7. When do we do it? 1. Online while test is still running 2. Post-test 3. Comparing tests to a specific baseline 4. Observing KPI history from a series of tests Unconsciously most of the time Speed Constraint main challenge Resources Used
- 8. Challenge Accepted Isn’t it trivial? Wasn’t it done before? Why can’t we reuse what APM vendors do these days?
- 9. How can we do it?
- 11. Apply the right test signal First and foremost:
- 12. Right test signal Load Profile Real test intention Measured ResponseSystem
- 13. Common Test Signals aka Load Profiles Stress, Max Throughput => Capacity, “how much” Constant, Soak, Sustain => Quality, “how good” Spike, recovery Combinations => reality requires to adapt Stress Constant Mixed Recovery
- 14. Each testing process stage has own techniques
- 15. In-Test Automation Instant threshold Windowed threshold => average / instant Steadiness threshold => stop constant test earlier Decisions to be made: stop test, mark it failed, mark it passed
- 16. In-Test Automation: Implementations JMeter-Plugins Autostop - show it Taurus Pass/Fail criteria - show it BlazeMeter Thresholds
- 17. Post-Test Analysis Stage breakdown 1. Ramp-Up Stage 2. Constant Stage 3. Leftovers POI detection APM correlation - “measure everything” costs you in the analysis stage
- 18. Post-Test Analysis: Implementations BlazeMeter App => Thresholds Loadosophia.org: stages breakdown test classification refined KPIs - it’s all about speed monitoring correlation 85% to 95% correct
- 19. Test Comparison & Series Trend Have reference test (aka “baseline”) Raise flag on too much difference Notify on trend outlier Notify on trend reaching limit It’s all easy to automate with APIs + simple scripts
- 20. We’ve Learned Today 1. Why to automate results analysis 2. What our “result analysis” is 3. Prerequisites to make it automatable 4. Particular techniques & implementations
Editor's Notes
- Automated analysis is very important because of CI/CD era and today's agile practices and things like microservices. It all gives you tens of services to test, tens of times per day. You physically can't analyse it by human eye, you need automation.
- Give example of JMeter: JTL file with samples => graphs => facts seen from graphs => decision if build is ok to go
- ‘cause moments we do it put important restrictions and requirements
- Today, most of the load testing tools and services do not do analysis for load test results, they do presentation. Automating this is not easy, because the process of a load test is complex and produces a timeline that needs to be analyzed.
- To solve that we need first to set the rules of the load test to make the results as predictable as possible. Many people do not think of their test intentions.
- The Model
- Even if you stopped ramping-up test - so what? Next thing is to tell KPIs from stress-test. Some autostops allow you to remember beginning point of problems
- Then, we need to get the automatic analysis method that is able to work with timeline data and teach it to recognize patterns that we used to recognize with our eyes. Which conclusions to make automatically Many people do not think of their test intentions. => how it detects intentions Technical analysis like forex, although it’s not to predict, but to deduce some facts Which KPIs to take 85% to 95% user satisfaction unsatisfied used to improve the algo Monitoring correlation allows you to go back to level of resources to understand what limits you