Automated Performance Testing with K6

Istanbul
November 22, 2019
TEST AUTOMATION &
DIGITAL QA EVENT
#TAS19

Test the performance of the system when something changes
AUTOMATED
PERFORMANCE
TESTING

MESUT GÜNEŞ
Senior Test Automation Engineer at VNGRS
in/gunesmes
gunesmes
story/gunesmes
gunesmes
testrisk.com
gunesmes@gmail.com

Mesut Güneş - www.testrisk.com
■ Performance testing
■ Type of performance testing (load, stress, soak, volume, capacity,
spike)
■ Automation, Tools, Technologies,
■ Benefits of Automated Tests
■ Integration
■ Thresholds, Percentile, rps
■ CI/CD
■ API
■ Orchestration
■ . . . and more
TALK INCLUDE THESE TOPICS

TABLE OF
CONTENTS
WHAT IS PERFORMANCE
TESTING
Concept of
performance
AUTOMATED
PERFORMANCE TESTING
Concept of automation in
terms of performance
01
03
02 04
05
06
METRICS
What to know about the
metrics during the
performance testing
TOOLS AND
TECHNOLOGIES
What kind of tools we have for
this purpose
THE WAYS OF
IMPLEMENTATION
How we can implement the
automated performance tests
BENEFITS
What kind of benefits
we can have

WHAT IS PERFORMANCE
TESTING
01

Defining the performance of the system by testing the functionalities of the system
in terms of the non-functionalities
Performance Testing is
■ A non-functional testing
■ Testing the functionalities of the system
■ Both black-box and white-box testing
■ Defining how the system works under any loads
■ Applied under different load levels
■ Applied under different load increments
■ Applied under different durations
PERFORMANCE TESTING BY DEFINITION

Performance testing should be also executed in a process. Performance engineers
should ensure that they collected requirements, created correct amount of virtual
users, created test scenarios and they can execute them. As a final they should check
the result and the system, if required after fixing, re-execution may be needed.
PERFORMANCE TESTING BY PROCESS
Planning
Creating test Scripts
Creating the Test Env.
Executing the Tests
Monitoring the Systems
Analyzing & Optimizing

Types of performance testing depend on the strategy that applied during the test by
means of the loads, the duration and increment of the load
Types of Performance Testing is
■ Load
■ Stress
■ Soak
■ Spike
■ Volume
■ Capacity
TYPES OF PERFORMANCE TESTING

Load test is to understand the behavior of the system under a specific load. It is performing
under specific amount of the load to check if the system gives the expected responses with
keeping the it stable for a long term
Load Test is
■ Basic form
■ Widely known-form
■ Specific amount of load
■ Checking the expected response time
■ Giving a result that users are facing
Image: https://images.app.goo.gl/KBysy7qsFPjkZ8Ai7
LOAD TEST

Stress test is to test the upper boundary of the system. When the load is increased to the
level of upper boundary, stress test aims to get the response of the system.
Stress Test is
■ Testing the upper boundary of the system
■ Needing more resources
■ Needing more investigation for boundaries
■ Needing more attention
■ Prone to break the system, aka: soak, endurance testing
■ Preparation for promotion like black friday, 11.11
Image: https://images.app.goo.gl/TVy8DzqnVN4NhCHA8
STRESS TEST

Soak test is to test the upper boundary of the system for a prolonged time. Soak is good if you are
not sure the expected amount of the user for some occasions.
Soak Test is
■ Testing the upper boundary of the system
■ Testing the system for a prolonged time
■ Needing more resources
■ Needing more investigation for boundaries
■ Needing more attention
■ Aims to break the system
■ Preparation for campaigns like black friday, 11.11
Eliud Kipchoge, Image: https://images.app.goo.gl/6FCD6U95h8jFWuXX9
SOAK-ENDURANCE TEST

Spike test is to test the behaviour of the system when the amount of load is suddenly increased
and decreased. It aims to find system failures when loads is changing to the unexpected amount.
Spike Test is
■ Testing the sudden increase and decrease of loads
■ Requiring more load generations
◂ depends of the tools (load the system, then spike it)
■ Good for some occasion like simulating
◂ push notifications for e-coms
◂ announcing critical news
Image: https://images.app.goo.gl/yPxGxZZYc2cuFto28
SPIKE TEST

Capacity testing is aiming to find the maximum amount of the user that the system can
handle. By this test we can find the how many user can use the system.
Capacity Test is
■ Testing the max user
■ Finding the behavior when more users join
■ Providing info about the loss of user data
Image: https://images.app.goo.gl/Y8vr2eNcrjne7Etx5
CAPACITY TEST

Volume test is testing the database directly instead of the whole system. With the volume
testing, we can find if there is any bottleneck with the database system.
Spike Test is
■ Testing the database
■ Testing the with large amount data
Image: https://images.app.goo.gl/EtQyUwUGqx5PCSda8
VOLUME TEST

WHAT IS AUTOMATED
PERFORMANCE TESTING
02

Automated testing is running the tests and reporting results by tools with scripts.
Automated performance testing is designing the performance test scenarios to run them
spontaneously by tools and evaluation the result by tools to decide to go or not to go
further.
Automated Performance Testing is
■ Subset of the performance test scenarios
■ Designed to run spontaneously by tools
■ Designed to evaluate the result by tools
■ Designed not to break the system
■ Having decision metrics, ex: max response time, max-average resp time, ...
■ Designed to create report for concurrent and historical run
AUTOMATED PERFORMANCE TESTING

Automated performance testing can be applied to the all types of performance testing.
However each types need different level of maturity and sanity.
Automated Performance Testing is
■ Easily applied to load testing
■ Hard to apply stress test, spike test, but have benefits
■ Very hard to apply Soak-Endurance test
WHAT TYPES TO AUTOMATE

Since the load test is aiming to response of the system under a specific load and it is
success is belong to the pre-defined metrics, load testing is the best suited performance
testing type.
Automated Load Testing is
■ Needing specific load level
■ Needing pre-defined metrics
■ Needing evaluation of the result
■ Needing exactly the same environment - only one thing can be testable
■ Needing tools and script knowledge
■ Easy to integrate CI/CD pipeline - everything is scripted
■ Reporting the performance of the system when something changes
AUTOMATED LOAD TESTING

METRICS
03

Performance testing is meaningful when the metrics are understandable for each counterparts. These
metrics are the key identifier for evaluating the results and deciding if the test is successful or failed.
Some of the Metrics are
■ Response Metrics
◂ Response time
◂ Average response time
◂ Peak response time
◂ Wait time
◂ Error rate
■ Volume Metrics
◂ Concurrent users
◂ Requests per second
◂ Throughput
METRICS

Response time is the main measurement taken during the test. Then we calculate the average, mean,
median, minimum, maximum values of it.
Variants of the Response Metrics are
■ Response time
◂ Time to complete a request to the system
◂ Time to First Byte (TtFB) - response starts
◂ Time to Last Byte (TtLB) - response ends
◂ Wait time is the time between sending request to TtFB
■ Average, Median, Peak response time
◂ Average is mean of the whole responses
◂ Median is the one in middle between min to max
◂ Peak is the maximum value
■ Error Rate
◂ # of Error / # of Request
RESPONSE METRICS

Volume related metrics are depends on the loads which are generated during the tests
by the tools. These metrics show how much loads was sent to the system and how
much response was receipt. Basically these are requests-per-seconds (RPS),
concurrent user, throughput, hatch rate
Variants of the Volume Metrics are
■ RPS is the number of request sending per second
■ Concurrent User is the number of virtual users which are testing the system
■ Hatch Rate is the number of virtual user adding to the system per second
■ Throughput is the data in kilobytes which are sent from the servers
VOLUME METRICS

Performance testing depends two main parts. The first part is creating loads and the
second part is using these loads for test scenarios. When it comes to do automated
performance testing, we need more tools
Tools Needed for Automated Performance Testing are
■ Performance Testing tool
■ Databases
■ Reporting tools
■ Comparison tools
■ Orchestration Tools
TOOLS FOR AUTOMATED PERFORMANCE
TESTING

We have a variety of tools in the industry, they changes from open source to commercial and/or commercial
support; newly started to 20+ years old performance testing tools.
Some of the well-know Performance Tools are
■ Open source
◂ Jmeter - Apache project, v1.0 1998
◂ Gatling - Enterprise support by Frontline
◂ K6 - Enterprise support by LoadImpact
◂ Locust
◂ and many others
■ Commercial
◂ LoadRunner by Microfocus (previously Mercury - HP)
◂ Blazemeter - support multiple opensource tools
◂ LoadUI Pro by Smartbear
PERFORMANCE TESTING TOOLS

We need to manage these tools to run the test efficiently. Every CI/CD run on a
orchestration tool that provide every needed task by using different tools and scripts.
Some of the well-know Automation Tools are
■ Jenkins
■ TravisCI
■ TeamCity
■ CircleCI
■ Code Ship
■ Gitlab CI
AUTOMATION TOOLS

We need to create test environments, some environment for running performance
testing and also some environment to keeping data and reports. These environments
should be created
Some of the well-know Orchestration Tools are
■ Docker Compose
■ Kubernetes
■ Prometheus
■ Mesosphere DC/OS
■ And others
ORCHESTRATION TOOLS

THE WAYS OF
IMPLEMENTATIONS
05

K6 is an open source performance testing tool. It is designed for developers, it gives us
many good features to run tests in safer, faster, easier integrated way. Very easy to
integrate it to CI/CD.
Some of the Good Features of K6 are
■ Precious in performance, among the top 5 in comparison
■ Written in Golang, allows to write test script in Javascript
■ Already dockerized
■ Easy to connect to DB (such as InfluxDB)
■ Easy to create reports (Grafana)
■ Has build-in Thresholds
■ And many others
IMPLEMENTATION WITH K6

K6 has rich libraries that you can use in the script. Request is sending by http from
k6/http. Grouping is letting you to run same kind request in a group.
WRITE SCRIPT WITH K6

Thresholds is designed to define fail/pass criteria. With it you can define max, min,
average, median, throughput and other metrics.
THRESHOLDS - K6

Command line supports many parameters to run your scripts. For a load test, we have
fixed virtual user and fixed test duration so the run script should look like
RUN LOAD TEST WITH K6

For a spike test, we should load the system with nominal load and the load should
increase suddenly. Let’s focus on at the --stage option for this.
RUN SPIKE TEST WITH K6

Simple add your run script to Jenkins and run it.
JENKINS RUNNING K6

The K6 sends the results to the InfluxDB, we can virtualize this data with Grafana
INFLUXDB + GRAFANA FOR K6

BENEFITS
06

When the benefits of automated test composed of the benefits of performance testing it become
more delicious :)
Some benefits of automated performance testing are
■ Giving current performance of the app
■ Creating awareness about the performance of the system
■ Developers tends to miss performance point
■ Preventing last minute mistake
■ Giving history about the performance of the system
■ Making it easy to run performance testing on demand
Image: https://images.app.goo.gl/TVy8DzqnVN4NhCHA8
BENEFITS

CREDITS: This presentation template was created by Slidesgo,
including icons by Flaticon, and infographics & images by
Freepik.
Please keep this slide for attribution.
Does anyone have any
questions?
THANKS
References:
● https://www.anztb.org/userfiles/files/PerformanceTestingQLD0414.pdf
● https://www.theseus.fi/bitstream/handle/10024/110354/Joonas_Moilanen_Mervi_Jeskanen.pdf?sequence=1
● The Art of Software Testing, http://barbie.uta.edu/~mehra/Book1_The%20Art%20of%20Software%20Testing.pdf
● K6 manual, https://docs.k6.io/docs/welcome
● https://github.com/gunesmes/testhive-django-unittest-behave-selenium-newman-locust-jenkins
● http://www.testrisk.com/2015/09/performance-testing-on-ci-locust-is.html
● http://www.testrisk.com/2016/01/performance-testing-on-ci-integration.html

Automated Performance Testing with K6

Recommended

Recommended

More Related Content

Recently uploaded

Recently uploaded (20)

Featured

Featured (20)

Automated Performance Testing with K6