Talk in the DevOpsPro Europe 2024 at Vilnius

1. Growing at the edge:
doubling traffic while
changing the API
gateway
Paco Orozco
DevOps Pro Europe 2024
May 24th, 2024

2. ➔ Since October 2017
➔ Manager of 4 awesome engineers from 3
different nationalities
➔ Product owner of two of the most adopted
services in the company.
➔ Father of two outstanding daughters
➔ Hiker
Engineering Manager @
Paco Orozco

3. 3
Adevinta around the world
Belgium
Spain
France
Italy
Canada
Germany
Netherlands
Brazil
Ireland Austria
Joint ventures

4. 2.5bn
monthly visits worldwide
25+
marketplaces in our digital
portfolio
5200+
employees (approx), across
10 countries
€1.8bn
revenues in FY 2023

5. Privacy Handlers
5
YAMS

6. Resize & Crop
Sharp & Blur
Pixelation
Watermark
Brightness
Contrast
File Format
360º images
Doc2PDF
HLS video
Document
Image
Video
The YAMS - Your Adevinta Media Service
transformation tools
6

7. The What

8. 8
➔ Missing log correlation (no request-ids)
➔ Low test coverage
➔ Fetch & Management API GWs shared code
➔ High maintenance cost
➔ Obsolete: lack of support, reliability…
➔ Missing HTTP method (PATCH)
➔ Low performance (high spikes)
The Why
5 years in production
3 incidents in 1Q
due to high CPU

9. 9
The Assessment

10. 10
June 2021

11. 11
eBay Marketplaces meet YAMS
👶

12. 12
The preparation
● 3 workshops
● 3 Hands-on workshops
● 2 Stress tests
● Documentation revamp
● Bi-weekly meetings

13. 13
Fetch gateway replacement
Nevada State Route 376. (2022, June 10). In Wikipedia.
https://en.wikipedia.org/wiki/Nevada_State_Route_376

14. The Releasing Pipeline
14
● Unit test
● Integration tests
Code review
● Comments
● 2 developer
approval
Merge
● Unit test
● Integration tests
Dev
Pre
Pro
AMI & docker bakes
Acceptance tests
With canary releases

15. 15
Fetch gateway replacement
Nevada State Route 376. (2022, June 10). In Wikipedia.
https://en.wikipedia.org/wiki/Nevada_State_Route_376

16. 16
Logs & high CPU
Missing headers #1
Missing headers for
redirects
Incident: Missing
CORS headers
High CPU usage
Retries not working
Stelvio Pass. (2024, February 24). In Wikipedia.
https://en.wikipedia.org/wiki/Stelvio_Pass
Fetch Gateway rollbacks

17. 17
Not Lura, not KrakenD… it’s YAMS GW!

18. 18
Management Gateway replacement

19. Wrong
timeout
configuration
1 Wrong
configuration
for reaching
SIAM
2 Trailing ‘/’ in
objects
endpoint
3 Wrong auth
when using
query params
4
19
🙅
4 failed deployments
Management Gateway rollbacks

20. 20
After 10 rollbacks :-D
The Outcomes

21. 21
The Outcomes
Confidence
Higher test
coverage and
updated code
Up to date
Better YAMS user
documentation.
Less support
requests
Less silos
Team has shared
knowledge, less
silos
Cheaper
Less CPU usage,
less instances,
less money
1 Performance
10x more requests
with the same
instance type
2 Consistency
Request-ids and
logs are back
3
4 5 6

22. Fetch Gateway - CPU comparison
22
OLD ZUUL GW
NEW FETCH GW

23. KPIs before and after
23
Sept - Dec 2021 Jan - May 2022
2X
4X

24. 24
Evolution

25. Agenda
01 02
04
03 05
Assess
bravely
Improve
testing
Communicate
changes
Logs &
metrics are
vital
Win together,
lose together
The Takeaways

26. Questions?

27. Gracias Merci
Obrigado Dank u wel
Danke Grazie
Thank
you!
Gràcies
Paco Orozco
@pakusland

28. and many more…