Slide deck related to the presentation at the KubeDay Singapore event. The session covered 3 pillars of Observability and how to use Jaeger for Distribute Tracing, Loki for Log Aggregation and Prometheus and Grafana for Metrics in a distributed application. Azure Kubernetes Service AKS cluster was used for live demo. https://events.linuxfoundation.org/kubeday-singapore/

1. @nileshgule
Improve
Monitoring and Observability
for
Kubernetes
with
OSS tools

2. Nilesh Gule
ARCHITECT | MICROSOFT MVP
| First Docker Captain in
Singapore
“Code with Passion and
Strive for Excellence”
nileshgule
@nileshgul
e
Nilesh Gule
NileshGule
www.handsonarchitect.co
m
https://www.youtube.com/@nilesh-gule

3. @nileshgule

4. @nileshgule
CNCF cloud trail
https://github.com/cncf/trailmap

5. @nileshgule
CNCF Observability landscape
https://landscape.cncf.io

6. @nileshgule
CNCF Observability Radar
https://radar.cncf.io/2020-09-observability

7. @nileshgule
3 Pillars of Observability
Logs Metrics Traces

8. @nileshgule
Centralized
Logging

9. @nileshgule
❑ Application specific
❖ Long term log retention for compliance reasons
❖ Workloads scheduled on different nodes during
application restarts / updates
❖ Autoscaling workloads
❑ Kubernetes upgrades
❖ Auto healing can reschedule workloads
❖ Underlying nodes added / deleted during cluster
scaling
❖ Underlying nodes replaced during cluster
upgrades
Container based workloads
Why centralized logging
❖ Not much control over underlying infra
❖ Relies on cloud prover specific logging and monitoring
solution
PaaS / Serverless services

10. @nileshgule
Financial Services App Loki integration
Log collector Log storage Log search, visualise,
dashboards
backend-service account-service authentication-service forex-service transaction-service

11. @nileshgule
Demo 1 – Log Aggregation with Loki

12. @nileshgule
Metrics

13. @nileshgule
• Application specific
• Monitor resource usage
• Monitor scaling needs
• Monitor anomalies / outliers
• Kubernetes platform level
• Monitor cluster resources (CPU / RAM)
• API health
• Autoscaling
Container based workloads
Why Metrics
• Monitor resource usage
• Scaling
• Bottlenecks
PaaS / Serverless services

14. @nileshgule
Prometheus Architecture

15. @nileshgule
Demo 2 – Metrics using Prometheus &
Grafana

16. @nileshgule
Financial Services App Prometheus integration
Scrape Metrics Metrics storage visualise,
dashboards
backend-service account-service authentication-service forex-service transaction-service
service-monitor

17. @nileshgule
Distributed Tracing

18. @nileshgule
• Distributed Tracing
• Understanding complex systems
• Performance monitoring and optimizations
• Debugging and problem resolution
Why Distributed Tracing

19. @nileshgule
Financial Services App Jaeger integration
Distributed Traces Visualise Traces
backend-service account-service authentication-service forex-service transaction-service
Jaeger Operator

20. @nileshgule
Demo 3 – Distributed Tracing using Jaeger

21. @nileshgule
End to End Observability
backend-service account-service authentication-service forex-service transaction-service

22. @nileshgule
Analogy - Use right tool for right purpose

23. @nileshgule
Summary
Modern day cloud native applications need new ways to address observability &
monitoring
✓ Use best-of-class for given use case
✓ Rely on open standards (e.g. OpenTelemetry)
✓ Build portable observability systems (e.g. hybrid cloud migration)
Log Aggregation
✓ Loki helps in centralized logging
✓ Grafana is used to visualize logs and build dashboards
Metrics
✓ Prometheus provides easy to use metrics for platforms, applications
✓ Grafana provides visualization capabilities to build intuitive dashboards
Distributed Tracing
✓ Jaeger provides distributed tracing capabilities

24. @nileshgule
Some Recommendations
♣ Too many agents
♣ Instrumentation, vendor lock-in
♣ Cloud native logs
♣ Cloud native metrics
♣ Cloud native traces
♣ Single pane of glass, correlation
∞ OpenTelemetry collector
∞ OpenTelemetry, OpenMetrics
∞ Fluent Bit / Fluentd, OpenSearch, Loki
∞ Prometheus, Cortex, Thanos
∞ OpenTelemetry, Jaeger, Grafana
∞ Grafana
Challenges Tools

25. @nileshgule
References
Log Aggregation
❖ Grafana Loki
Monitoring & Alerting
❖ Prometheus
❖ Grafana
❖ Kube Prometheus stack
❖ Houssem Dellai – Prometheus & Grafana
for monitoring Kubernetes
Distributed Tracing
❖ Jaeger Tracing

26. @nileshgule
Source Code & slide deck
Financial Services Demo
https://github.com/infofractionalservices/microservices/tree/do
cker_build_fixes
https://speakerdeck.com/nileshgule/
https://www.slideshare.net/nileshgule/

27. Q&A