In this presentation Adam Dubiel shares some insight on how monitoring works in Allegro microservices based architecture. He introduces also "Hermes" - messaging solution based on Apache Kafka and overall thoughts when working with this kind of software. Enjoy!

1. Fighting with scale
things that “Getting started” doesn’t cover
Adam Dubiel

2. We’re building a New World

3. We used to have these..

4. ..and now we have these

5. Microservices are needy
discovery
messaging
deployment
metrics
monitoring
storage

6. Need for automation
env business
env business

7. Choosing the right tools
can be hard

9. and then came docker..

10. Metrics - graphite

11. building blocks
frontend
dashboard
aggregator
storage

12. building blocks
graphite-web
tessera
carbon
whisper

13. building blocks
graphite-web
tessera
carbon
ceres

14. building blocks
graphite-web
tessera
carbon
cyanite

15. frontend
dashboard
influxdb
building blocks

16. “getting started” setup
graphite-web
tessera
carbon
whisper

17. 1st iteration

18. 1st iteration - overall

19. 1st iteration - memory

20. 1st iteration - cause

21. 2nd iteration
OTHER
TECH
OTHER
TECH

22. 2nd iteration - memory

23. 2nd iteration - overall

24. not only soft, drivers too

25. Metrics - StatsD

26. architecture
Host1
Host2
Graphite
metric.host1.rate
metric.host2.rate
OK

27. architecture
Host1
Host2
Graphite
metric.rate
metric.rate
CLASH!

28. architecture
Host1
Host2
Graphite
metric.rate
metric.rate
OKStatsD

29. problem?

30. how to scale?
StatsDStatsD
StatsD proxy

31. how to scale proxy…?

32. Messaging - Hermes

33. architecture
Hermes
Frontend
Hermes
Consumer

34. Kafka is great!
performant
fault-tolerant
robust

35. but what with SLA?
strict SLA, p99 < 100ms
stable p999 would be great

36. cluster manipulations can be painful

37. and beware of abusive clients

38. 1st iteration - Kafka leaders

39. 1st iteration - Kafka leaders

40. 1st iteration - Kafka leaders
Kafka does not distribute leaders uniformly
simple take-first algorithm
no tools to make it happen
go with #!/usr/bin/python

41. 2nd iteration - buffering
simple solution to complex problem
use Kafka producer buffer
we can operate without Kafka for 1h!

43. and when we thought we have it all covered..
bam!
corrupt message logs

44. corrupt message logs
Kafka High Level Consumer can’t skip corrupt parts
no easy solution
time pressure - 24h retency
#!/usr/bin/python to the rescue!

45. Lessons learned

46. Production traffic is only on.. Production

47. prepare for the Unknown

48. ..but mostly for problems

49. scripts are your friends when time is the enemy

50. centralisation sounds appealing
(but seriously, don’t)

51. focus on
knowhow
agility
ability to create clusters on demand

52. isolate

53. allegrotech.io
@allegrotechblog