Event-driven microservices is an architecture maximizing the visibility and scalability of business platforms. Yet, this architecture style is full of challenges for successfully deploying them with quality at speed, Testing is them is therefore a requirement, and this talk shares how we test event-driven microservices at La Redoute. This talk was made a lighting talks at a DevOps Lisbon meetup.

1. Antoine Craske
HOW WE DECIDED TO TEST
EVENT-DRIVEN MICROSERVICES

2. 2
We had to accelerate

3. 3
A first strategic project came in
Order
Management
Transport
Management
(external)
Parcel
Management
Warehouse
Management

4. 4
Moving to event-driven microservices
Order
Management
Transport
Management
(external)
Parcel
Management
Warehouse
Management
Transport
Management
Parcel
Management
App
App
App
App
App
App
MS MS MS MS MS MS MS MS
MS MS MS MS
MS MS MS MS
Order
Management
App
App
Warehouse
Management
App
App
App

5. 5
Brought distributed processing
MS
Order
Mgmt
MS
…
Order
Mgmt
MS MS
…
API

6. 6
Decoupling, Events, Decentralized
Asynchronous event processing testing?
Eventual consistency versus business transactions?
Which test data management (TDM) strategies?
Which test environment management (TEM) strategies?

7. 7
We prioritized our test needs
Functional for end-to-end Business & individual component
Integration for availability and resiliency of the solution
Unit for code architecture and modularity
Load-testing to secure specific sales period (e.g. Black Friday)
GUI testing as only used for configuration elements
Exploratory testing to be performed along the project
Security not that a concern as internal services not exposed
Volume testing with forecasted throughput
1
2

8. 8
Within real-life constraints
• Scalable solution for 20-50 microservices
• 4 weeks (e.g. 2 sprints) to have a working solution
• CI/CD testing through existing environments (qa, uat, staging, prod)
• Reduce the risk and test as early as possible
• Tests to run fast with reliability and confidence

9. 9
Functional Testing
• End to end business process
• Individual functions
MS
Order
Mgmt
MS
…
Order
Mgmt
MS MS
…
API
CreateParcel
GetParcel
Main cases
Edge cases
Error cases

10. 10
Functional Testing
• End to end business process
• Individual functions
MS
Order
Mgmt
MS
…
Order
Mgmt
MS MS
…
API
Use Case 1
Use Case ..
Use Case n
Use Case 1
Use Case ..
Use Case n
Use Case 1
Use Case ..
Use Case n
Use Case 1
Use Case ..
Use Case n
Use Case 1
Use Case ..
Use Case n

11. Do we need integration test with functional and unit test?
• For exceptional specific cases, maybe
Do we need contract testing?
• To verify the decoupled microservices can call Kafka?
Does the project needs to test Kafka/K8S platform availability?
• Is it no the job of SRE team?
But how do we ensure error management and retries?
• Are functional tests covering with edge cases?
Integration Testing Availability and resiliency
11

12. Unit Test Code architecture and modularity
• Simple to write and maintain
• Focused and no external dependencies
• Reliable and repeatable
• Self-checked and explanatory
• Run fast (<100ms)
12
Components under test
Mocks for components
Source: mockframeworks.com

13. 13
Load Testing Specific sales period (e.g. Black Friday)
MS
Order
Mgmt
MS
…
Order
Mgmt
MS MS
…
API Nominal
Loaded
…

14. 14
Test Environment & Data
GUI
E2E (Full)
E2E (Process)
Component
Integration
Unit
Load-Test
Security, Volume, …
dev
staging
qa uat staging prod
qa uat staging prod
uat staging prod
N/A
(Mock/Contract)
Refresh from env
Test Generated &
Cleaned
Refresh from env
Refresh from env
dev
GUI
E2E (Full)
E2E (Process)
Component
Integration
Unit
Load-Test
Security, Volume, …
CI/CD Quality Gates
Manual / Scheduled
Manual / Scheduled
CI / Manual in local
Manual Launch

15. 15
Not yet a Christmas tree ☺
GUI
E2E (Full)
E2E (Process)
Component
Integration
Unit
Load-Test
Security, Volume, …
irregular
This solution is working fine and implemented fast by PO/Dev/QA
teams, for up to 30-50 microservices
GUI
E2E (Full)
E2E (Process)
Component
Integration
Unit
Load-Test
Security, Volume, …
dev
staging
qa uat staging prod
qa uat staging prod
uat staging prod
N/A
(Mock/Contract)
Refresh from env
Test Generated &
Cleaned
Refresh from env
Refresh from env
dev

16. 16
Open source Test Stack
GUI
E2E (Full)
E2E (Process)
Component
Integration
Unit
Load-Test
Security, Volume, …
irregular
This solution is working fine and implemented fast by PO/Dev/QA
teams, for up to 30-50 microservices
GUI
E2E (Full)
E2E (Process)
Component
Integration
Unit
Load-Test
Security, Volume, …
dev
staging
qa uat staging prod
qa uat staging prod
uat staging prod
N/A
(Mock/Contract)
Refresh from env
Test Generated &
Cleaned
Refresh from env
Refresh from env
dev

17. 17
Our evolution roadmap
Secure design quality for reliability, performance and testability
Spaghetti microservices, hard to develop, test and operate
Ephemeral environments for reliability, scaling and lower costs
On-demand test environments, refreshed, destroyed
Systematic and automated load and performance test
Enabling combination of load, stress and volume test

18. 18
Takeaways
Can’t we avoid testing in the first place?
Good design, then common librairies, platform, retry logic, then project requirements
Align test requirements on business, priorities and constraints
Priority between security or Load-test for an internal back-end services?
Prioritize your automation for real value and minimal maturity
“Automation applied to an inefficient operation will magnify the inefficiency.”
“Never perform a machine job by a human.”
e.g. Automate technical retry is a right priority

19. 19
THANK YOU