Download as PDF, PPTX
Uploaded byNikolay Stoitsev
A practical introduction to observability
The document provides an introduction to observability, covering monitoring, logging, and distributed tracing in application systems. It emphasizes the importance of using appropriate metrics, structured logging, and symptom-based alerts to effectively manage application performance and diagnose issues. Key recommendations include avoiding high cardinality tags, leveraging percentiles for analysis, and ensuring alerts are actionable and meaningful.
More Related Content
Similar to A practical introduction to observability
A practical introduction to observability
- 1. A practical introduction toobservability Nikolay Stoitsev Engineering Manager @ Halo DX
- 4.
- 5.
- 6.
- 7.
- 8. Monitoring system components Application Application Application Monitoring System TimeSeries Database Dashboard
- 9. Monitoring system components Application Application Application Monitoring System TimeSeries Database Dashboard Prometheus, Graphite, m3db
- 10. Monitoring system components Application Application Application Monitoring System TimeSeries Database Dashboard Prometheus UI, Grafana
- 11.
- 12.
- 13.
- 14.
- 15. What to watchout for?
- 16.
- 17. Cardinality ● search.success, app_version=1,type=Patient ● search.success, app_version=1, type=Exam ● search.success, app_version=2, type=Patient ● search.success, app_version=2, type=Exam
- 18. #1. Don’t addhigh cardinality tags
- 19. Metrics are notaccurate ● DB engine optimizes for faster operations ● When performing some operations for a different time resolution ● When archiving metrics for long term storage
- 20. #2. Don’t relyon metrics infrastructure for BI
- 21. Don’t use averagevalues ● Averages hide the outliers ● Doesn’t represent typical behavior
- 22. Use percentiles ● Representsthe worst experience in 90% of the time ● Can measure p90, p95, p99 p90
- 23. Histograms ● Shows thewhole distribution ● Configurable buckets
- 24. #3. Use percentilesor histograms
- 26.
- 27.
- 28. Alert Levels Send alertto oncall
- 29. Alerting tool isusually built into the metrics system
- 30. Alerts should be ●urgent ● important ● actionable ● real
- 31. Should represent either ongoingor imminent problems
- 32. What to watchout for?
- 33. 1. Better toremove an alert when it’s noisy
- 35. #2. Use successrate
- 36. Symptom-based monitoring ● Numberof 5xx HTTP response codes ● Response time ● Email sending is not working ● Users can’t log in
- 37. Cause-based monitoring ● Freedisk space on database server ● Memory utilisation ● Free file descriptors
- 38. Many causes maytrigger a symptom
- 39. User impact ismost important
- 40.
- 41. Cause-based alerts are alsonecessary
- 42. Picking alerts tostart with Front-end Load Balancer Back-end DB Count rate of successful log-in Count request success rate
- 44.
- 45.
- 46.
- 47.
- 48.
- 49. Finding logs Can searchby: ● content of log message message : *notification* ● all logs from a service kubernetes.labels.app/name.keyword : "api-gateway" ● many more thanks to flexible query schema
- 50. What to watchout for?
- 51. #1. Use appropriatelog level - info, warn, error
- 52. Structured logging ● Appenduseful key=value pairs ● Can group (aggregate) by the keys ● Can sort by aggregations
- 53. #2. Use structuredlogging
- 54. Too many logs Application Application Application Log Aggregation RealTime Search Engine Log Scraper Log Scraper Log Scraper Dashboard
- 55. Too many logs Application Application Application Log Aggregation RealTime Search Engine Log Scraper Log Scraper Log Scraper Dashboard Reduce log retention period
- 56. Too many logs Application Application Application Log Aggregation RealTime Search Engine Log Scraper Log Scraper Log Scraper Dashboard Cold Storage Query UI
- 57. #3. Use properretention period or cold storage
- 59.
- 60. End-to-end summary 1. Configureautomated alerts
- 61. End-to-end summary 1. Configureautomated alerts 2. Use metrics and tracing to pinpoint the problem
- 62. End-to-end summary 1. Configureautomated alerts 2. Use metrics and tracing to pinpoint the problem 3. Use structured logging to find the root cause of the problem easily
- 63. End-to-end summary 1. Configureautomated alerts 2. Use metrics and tracing to pinpoint the problem 3. Use structured logging to find the root cause of the problem easily 4. Fix problems and make sure all metrics are always back to normal
- 64. Thank you! Q&A NikolayStoitsev Engineering Manager at Halo DX Photo by Pixabay, Şahin Sezer Dinçer, Andrea Piacquadio, Ian Beckley from Pexels