Case study and guidelines for performance testing microservices. The talk focuses on clarifying the goals of performance testing, suggestions on tools that will help get started and common pitfalls. Originally presented at Reactive Summit 2018.

1. Reactive
Performance
Testing

2. Hi, I am
Lilit Yenokyan
2
Director of Engineering at Pivotus
Co-founder and leader of
Silicon Valley
Alumni of and
@lyenokyan on Twitter

3. 3
Measure capacity and
pressure of the pipes
v = π r2h

4. When?
Why?
What to keep in mind?
4
Demonstrate on Pivotus use case

5. Background
5

6. Team
HQ
6
Across three continents
and five time zones.

7. How much load can our systems support?
1 million users? 10 million conversations? How do you
come up with those numbers?
Pivotus use
case
Agent portal
Provide bank agents with
a portal to access and
manage conversations
with their customers.
7
Customer apps
Provide bank
customers with
IOS/Android apps to
instruct with their
users
Reactive
microservices
Backend of 8
microservices that
need to scale

8. The plan for
this session
3. Reactive
Specific
8
2. Discuss
Metrics
Used
1. Define
Performance
Testing
4. Lessons
Learned

9. Performance
Testing
9
Measuring the full capacity
of a dam flow
Kerr Dam~ Polson, Montana
Image by Alpdenglowm

10. Why should
you care?
10
Performant system means spending less money!!

11. What are we
trying to
measure?
3. Reactive
Specific
11
2. Discuss
Metrics
Used
1. Define
Performance
Testing
4. Lessons
Leanded

12. Types of
Performance
Testing
12
Load testing
Stress testing
Endurance testing

13. Types of
Performance
Testing
13
Load testing
Stress testing
Endurance testing

14. Defining
success
criteria Define the load
handling goals
Most importantly what
is the scale that we are
targeting.
Define and implement
test scenarios
Document scenarios
and implement tests
using Jmeter
Establish high
watermarks
Find out the breaking
points of the system
How much load can it
sustain before
breaking/erroring out
14
Monitor the results
with each release
See how the
watermarks shift

15. In the
absence of
production
data
Benchmark Low Load Target Load
# Clients 1 3 10
Agents 10 100 5000
Customers 100 10000 250000
A/C ratio 1/10 1/100 1/500
Messages per day 104 106 108
15
Come up with the goals for your system. In the absence of
production data you need to define your own.

16. 2. How we
log the
metrics
3. Reactive
Specific
16
2. Discuss
Metrics Used
1. Define
Performan
ce Testing
4. Lessons
Learned

17. Tools we use
17

18. Where to run
these tests?
18
Production Cluster Clone of Production cluster
We used AWS for the examples discussed.
The approach is cloud provider/or data center of choice
agnostic.

19. 19
Creating test cases
Logging test steps
Organizing test runs
Tracking results
TestRail

20. Jenkins
20
Single master/slave configuration
Retains the results for 90 days of runs
Jobs designed for partner teams

21. 21
Performs load test
Simulates request to target server
Returns stats about performance
Creates
request to
target
server
JMeter
saves all
responses
Server
responds
JMeter
gathers
data
Start End
Create
Report

22. 22
InfluxDB is time series db for
performance metrics storage
Grafana is visualization tool for
viewing reports

23. 23

24. 24

25. 25
System metrics
monitoring
Viewing
environment state
during tests

26. 26
Elaborate reporting
Providing high level
context to partners

27. Tools starter
kit
27

28. 1. Define
Performance
Testing
3. Approach
for testing
Reactive
Services
3. Reactive
Specific
28
2. Discuss
Metrics
Used
1. Define
Performan
ce Testing
4. Lessons
Learned

29. Microservices
vs Monolith
vs UI testing
29
Monolith
Testing the qualities of the
system as a whole
Microservices
Can be considered in in
separation
User interface
Performance testing is
important to draw limits that
browser and apps can sustain

30. Classical
Distributed
systems vs
Reactive
30
Distinctive characteristics of
reactive microservices:
1. Responsive: the
system responds fast
in a consistent way
and encourages the
user’s interaction.
2. Resilient: the system
stays responsive in
case of failure.
3. Elastic: the system
stays responsive
under varying
workload.
4. Message Driven:
communication by
exchanging
asynchronous
messages.
message-driven
resilientelastic
responsive

31. 4. Discuss
lessons
learned
3. Reactive
Specific
31
2. Discuss
Metrics
Used
1. Define
Performance
Testing
4. Lessons
Learned

32. Interesting
finding #1
32
Measuring 1 vs 3 instances
results
Investigations shows two
reasons for degradation:
1. DB data amount
2. Web socket handling with
multiple instances of services.

33. Interesting
finding #2
33
System did not properly recover after
endurance testing.
Normal operation
Disaster occurrence
Disruption and failure
of operation
Recovery process
Reconstruction -
Normal process
Reasons was:
Incorrect configuration for
recovery mode

34. Interesting
finding #3
34
Mass messaging feature via eyes of an agent.
After sending a mass
message an agent
can view the same
message in all
conversations where
he is a primary agent.

35. Interesting
finding #3
35
Agent Mass message
Customer
Customer
Customer
Customer
Customer
Customer
Agent Mass message Customer Customer Customer Customer
Mass messaging feature is intended for sending of
the same message from agent to her all customers.
It sent messages in parallel.
Redesigned the feature to send sequential messages
instead of parallel. Performance testing adjusted the
targets.

36. Interesting
finding #4
36
In server side we have two types of DB:
1. Read DB
2. Write DB
The system integrates with user authentication providers, like OKTA. After user creation
we usually try to login with that account. We found out that there is ~3 second timeout
between these operations.
Client
UI
Write
API
Read
API
Read query
Domain model
Queries
Commands DB
DB
Sync
Server
Command
Query

37. Interesting
finding #5
37
Performance can degrade over time!
Run regression tests for each release

38. Takeaways
38
Define
expected
performance
before you
commit
Start
measuring
performance
early on
Question the
obvious
Measure
performance
with every
release

39. Q&A
39
@lyenokyan