Introduction to Gatling performance testing tool and how we used it for testing Zonky's REST API. Example of running distributed performance tests in AWS Fargate with real-time monitoring with Logstash/ElasticSearch/Kibana stack.

2. Load testing
in Zonky
Light introduction to
Gatling.io
Petr Vlček

3. Traditional solutions
● We want to tackle our performance issues with performance testing tools
● Traditional performance testing tools have some issues though

4. Complex GUI
Oh no, I want my code
back!
● Too many features
● Scripted invisible logic
● Not reusable
● Refactoring?

5. Versioning is painful
What has changed actually?

6. Performance of 1 user = 1 thread approach
50 JVM THREADS

7. Performance of 1 user = 1 thread approach
200 JVM THREADS

8. Gatling
● Describe load tests with code
○ just as unit tests, integration tests, REST tests, etc.
○ familiar to devs
○ versioning is easy (Git, peer review with pull requests, ...)
● Simple Scala DSL
● Tech under the hood
○ Akka - framework using message-actor pattern
○ Netty - non-blocking IO
● More efficient - can simulate more paralel users with less threads

9. Gatling
● Generates beautiful HTML
reports out-of-the-box
● or reports live data to Graphite

10. Defining a simple sunny day
scenario
1. Access Zonky marketplace page as an
anonymous user to see which loans are
available for investment
1 user, 1 HTTP request

11. Coding the simple simulation
● Configure server URL and default
HTTP headers (10-12)
● Define a scenario (14) executing
one step (15) which does a HTTP
GET request to /loans/marketplace
(17) and checks that response was
200 OK (18). After that wait for 5
seconds (20).
● Execute scenario (22) with 1 virtual
user (21) injected immediately after
start

12. Basic concepts - Scenario
● Represents user’s behavior - a workflow that a virtual user will follow
● Sequence of steps (exec)
● Thinking time simulation (pause)
● Grouping steps (group)

13. Basic concepts - Virtual user
● Represents a distinct user, each one has its own data or maybe has
different behavior
● Modelled as an Akka message passed along Akka Actors backing each step
in a scenario workflow

14. Basic concepts - Simulation
● Description of a load test
○ how many users (possibly with different scenarios) will run
○ which scenarios will they execute
○ how new users will be injected into simulation
● Users are injected to a simulation using different injection profiles

15. Injecting virtual users into simulation
● nothingFor - do not add any users for given duration
● atOnceUsers - add given number of users immediately
● rampUsers - add given number of users with linear ramp over given duration
● constantUsersPerSec - add users at a constant rate, defined by number of users per sec
● rampUsersPerSec - inject a random number of users per second over given duration
● splitUsers - repeat an injection step separated by pauses until given number of users is
reached
● heavisideUsers - Injects a given number of users following a smooth approximation over a
given duration
● See http://gatling.io/docs/current/cheat-sheet/

16. Injecting virtual users into simulation - example

17. Basic concepts - Session
● Each virtual user is backed by a session (modelled as an Akka message)
● State holder where data can be injected or captured from a step and can be
accessed in subsequent steps
● Session objects are immutable! set and setAll methods return new instances,
use chaining when setting more attributes

18. Basic concepts - Feeders
● Inject data from external source into a session of a virtual user
○ authentication data, various input data, …
● Sources
○ Scala data structures - Seq, Array
■ implicit iteration methods - queue, random, shuffle, circular
○ CSV files
○ JSON files
○ JDBC
○ Redis
● Conversion - convert raw data from feeder
● Data files location configurable in gatling.conf (gatling.core.directory.data),
default location is ${resource_root}/data
● See http://gatling.io/docs/current/session/feeder/#feeder for more examples

19. Feeders - example

20. Basic concepts - Checks
● Checks can be used to verify that a server response meets given conditions
○ Status is 200
○ HTTP header is present and matches a regex expression
○ response time in millis is lower than …
○ response body equals …
● Information from response can be parsed and saved to a user session
○ Parse response (JSON), use JSONPath to locate an attribute in the response
○ Extract data from response (find, findAll, findRandom, count)
○ Transform data from response (provide transforming lambda function)
○ Validate data from response (is, not, exists, notExists, in(sequence), optional)
■ RawFileBody, ElFileBody
○ Save, optional step, store parsed information as a session attribute for further steps
● See http://gatling.io/docs/current/http/http_check/#http-check

21. Basic concepts - Assertions
● Defines acceptance criteria on Gatling statistics (assertion fail = test fail)
● Scopes - global, forAll, details
● Statistics - responseTime, allRequests, failedRequests, successfulRequests,
requestsPerSec
● Conditions - lessThan, greaterThan, between, is, in
● See http://gatling.io/docs/current/general/assertions/#assertions

22. Passing parameters with system properties

23. Our infrastructure - AWS
Jenkins
Amazon
Web Services
Jenkins
Jenkins
JenkinsLoad generator
(ec2 instance)
Zonky API
Monitoring
tools
1
2
3
4
Uses Gatling AWS Maven plugin,
https://github.com/electronicarts/gatli
ng-aws-maven-plugin
1. Spin up (or use existing)
load generators in AWS,
distribute test package
2. Execute tests and generate
load on Zonky API
3. Collect performance metrics
with monitoring tools (New
Relic, Grafana, etc.)
4. Collect data for Gatling
report from all load
generators and compile
HTML report

24. Our infrastructure - Jenkins
● Gatling plugin
○ Performance
trends
○ Report viewer
● All jobs on
Performance tab

25. Wait, … that’s so 2016 ?!

26. 3 years later ...

27. Scaling out perf tests with AWS Fargate
● AWS Fargate = serverless Docker runtime environment managed by Amazon
● Pros
○ Easy setup, ideal for small startups, drop-and-run your Docker image
○ No need to provision your own EC2 machines and tear them down after the test
○ You pay only for time when load generators are running
● Cons
○ Higher price per hour
○ Limits
■ max. 50 concurrently running tasks, contact support for raising the limit
■ possibly network performance,
https://stormforger.com/blog/2018/06/04/aws-fargate-network-performance/
■ no persistent local storage for tasks

28. Demo - AWS Fargate + Gatling + ELK stack
● Build
○ Gatling tests as a single executable jar file
○ Docker image with jar file baked in, executed as a task in Fargate, or locally ...
● Deployment
○ Logstash as a Fargate service
○ Fargate task definitions for Gatling load generators
● Runtime
○ Spin up to 50 load generators in Fargate, generators destroyed automatically after they finish
○ Collect data through Logstash to Elastic Search and copy simulation.log to S3 on finish
○ Compile final report from collected simulation.log files

29. Source code
● https://github.com/petrvlcek/gatling-demo
○ load generator
○ Docker configuration
○ scripts for running in AWS Fargate
● https://github.com/petrvlcek/gatling-logstash
○ OSS Logstash with pipeline configuration for Gatling

30. What to test and when
● One does not simply write performance tests
for everything
○ it’s expensive and pointless
○ you can’t predict user’s behavior
● OODA loop
○ Observe - know what’s happening in your system (monitoring, logs, analytics)
○ Orientate - identify hot-spots/bottlenecks in your system based on collected data
○ Decide - prioritize
○ Act - fix critical bottlenecks and validate with performance tests

31. Final thoughts
● Performance tests are tests like any other
○ coded by devs
○ no departments or expert teams are needed, trust your devops
● Integrate them to your CI/CD pipeline
○ trends/regression
○ acceptance criteria
● Decide what to test after observing your users and data from your monitoring