The document discusses the challenges and best practices for monitoring microservices in containerized environments. It highlights the limitations of traditional monitoring models and presents Sysdig as a solution that provides automatic metrics collection and analysis without extensive instrumentation. Key areas addressed include metric collection, information aggregation, troubleshooting, and team organization in microservices infrastructure.

1. Jorge Salamero Sanz
@bencerillo @sysdig
How to
Monitor Microservices

2. How to Monitor Microservices?
Apps
Infra
Health
Checks
JVM/JMX
Custom
metrics
Metrics Processing Unicorns, rainbows
And cute dashboards

4. % whoami
Jorge Salamero Sanz
<jorge.salamero@sysdig.com>
• Working on OSS last 12 years
• Working on monitoring last 3 years
• Containers gamer @sysdig
@bencerillo
@sysdig

5. Agenda
• Challenges of container infrastructures
• Traditional monitoring limitations
• Best practices monitoring Microservices
• Sysdig, container native monitoring & troubleshooting

6. Breaking traditional model
Microservices and containers break traditional monitoring and
troubleshooting models

7. Traditional deployment
Full host OS
kernel
systemd
syslogd
App services
MySQL
Nginx
OpenSSL
Java
App A
App B
App C
Ops Devs

8. Application workflow
Database App Cache
backend middleware frontend

9. The Docker revolution
Servers Virtual Machines Containers
Unit: machine
Orchestration: no
Architecture:
monolithic
Unit: machine
Orchestration:
external
Architecture:
monolithic
Unit: (micro)service
Orchestration:
native
Architecture:
distributed

10. Containerized deployment
Full host OS
kernel
+
Docker
MySQL App A
Ops DevOps
Nginx + OpenSSL App B
Java 8.0 build XXX App C

11. … but in reality
Database App Cache/Frontend
Computing node
Computing node Computing node
Computing node Computing node
Computing node

12. … but in reality
Database App Cache/Frontend
Computing node
Computing node Computing node
Computing node Computing node
Computing node

13. Application workflow
Database App Cache
backend middleware frontend

14. New organization structures

15. Container monitoring
New challenges:
1. How do we get the metrics?
2. How do we shape all this amount of metrics?
3. Analysis and troubleshooting
4. Teams on Microservices infrastructure

16. Container monitoring
New challenges:
1. How do we get the metrics?
2. How do we shape all this amount of metrics?
3. Analysis and troubleshooting
4. Teams on Microservices infrastructure

17. 1. Metric collection
• We containers, because:
– are simple
– are small
– are isolated
– less dependencies
• … but they are an opaque blackbox

18. “Workarounds”
Agent in the
Docker container
Agent in the
Kubernetes pod
Export metrics through
an external agent
App Agent App Agent
App
Agent
App
App
App
1. Complex instrumentation (x2 because just the monitoring) plus
service monitoring configuration
2. Limited and pre-established metric collection (Docker API, etc)

19. Kernel instrumentation
Kernel
Docker
Container
1
Container
2
Container
3
App App
rkt LXC
Sysdig
Docker

20. Why this is cool?
• Just one instrumentation per host:
– spawning or destroying a container is instrumentation-less
• Full visibility: all the system calls:
– automatic service discovery
– all metrics collection (no filtering)
– application monitoring without instrumentation (magic of
decoding protocols)

21. Container monitoring
New challenges:
1. How do we get the metrics?
2. How do we shape all this amount of metrics?
3. Analysis and troubleshooting
4. Teams on Microservices infrastructure

22. Remember... but in reality:
Database App Cache/Frontend
Computing node
Computing node Computing node
Computing node Computing node
Computing node

23. 2. Information aggregation
• Infrastructure monitoring should be transparent and
automatic (no instrumentation no configuration)
• You should handle your custom/biz metrics
• All metrics should be tagged automatically
• All metrics should be aggregated and segmented on a
service level basis

24. Application workflow
Database App Cache
backend middleware frontend

25. Orchestration platforms knows already...

26. Demo!

27. Container monitoring
New challenges:
1. How do we get the metrics?
2. How do we shape all this amount of metrics?
3. Analysis and troubleshooting
4. Teams on Microservices infrastructure

28. Real example
https://github.com/kubernetes/kubernetes/issues/1405
1

29. 3. Analysis & troubleshooting
• Imagine:
strace + wireshark + htop + lsof + iostat + vmstat + *
• Not available on containers, don’t understand
namespaces
• Metrics and logs can bite your in the ass, system
calls have all the truth
• Infrastructure gets more complex and volatile

30. Demo!

31. Container monitoring
New challenges:
1. How do we get the metrics?
2. How do we shape all this amount of metrics?
3. Analysis and troubleshooting
4. Teams on Microservices infrastructure

32. Teams on Microservices
Francesc Zacarias, SRE @ Spotify

33. 4. Teams by service
• Tags/Metadata from the orchestration platform, eg
Kubernetes:
– namespaces (dev, prod)
– services, deployments, RCs, pods
– custom tags
• ACLs out of the box (dashboards, alerts, etc) on
multi-tenant/PaaS scenarios

34. Container monitoring
New challenges:
1. How do we get the metrics?
2. How do we shape all this amount of metrics?
3. Analysis and troubleshooting
4. Teams on Microservices infrastructure

35. Sysdig
• 100% open-source
• 1M+ downloads
• Host analysis
• sysdig.org
• SaaS & on-prem
• 200+ customers
• Cluster analysis
• Dashboards, alerts,
events, teams
• sysdig.com

36. ¡Danke
MicroXchg!
@bencerillo
@sysdig
www.sysdig.org
www.sysdig.com