Good observability is essential for modern software. It gives us confidence that our systems are working properly. And it also allows us to debug issues efficiently. In this talk, we’ll explore everything you need to know to start applying good observability to your projects. And we’ll see the most common pitfalls you need to be aware of. We will start with the tools and basic concepts in monitoring. And we’ll go over the 3 most common mistakes people make with it. Then we’ll see how to have automatic alerts to detect issues. And, we’ll touch on the principles for setting up good alerts. As a final step, we’ll see how to build our logging system and how to apply it in the most efficient way to debug issues easily.

1. A practical introduction
to observability
Nikolay Stoitsev
Engineering Manager @ Halo DX

4. Monitoring

5. Monitoring
Logging

6. Monitoring
Logging
Distributed Tracing

7. Monitoring

8. Monitoring system components
Application
Application
Application
Monitoring
System
Time Series Database
Dashboard

9. Monitoring system components
Application
Application
Application
Monitoring
System
Time Series Database
Dashboard
Prometheus, Graphite, m3db

10. Monitoring system components
Application
Application
Application
Monitoring
System
Time Series Database
Dashboard
Prometheus UI, Grafana

11. Counter

12. Counter increase

13. Timer

14. Labels

15. What to watch out for?

16. Cardinality

17. Cardinality
● search.success, app_version=1, type=Patient
● search.success, app_version=1, type=Exam
● search.success, app_version=2, type=Patient
● search.success, app_version=2, type=Exam

18. #1. Don’t add high
cardinality tags

19. Metrics are not accurate
● DB engine optimizes for faster operations
● When performing some operations for a different time resolution
● When archiving metrics for long term storage

20. #2. Don’t rely on metrics
infrastructure for BI

21. Don’t use average values
● Averages hide the
outliers
● Doesn’t represent
typical behavior

22. Use percentiles
● Represents the
worst experience in
90% of the time
● Can measure p90,
p95, p99
p90

23. Histograms
● Shows the whole
distribution
● Configurable
buckets

24. #3. Use percentiles or
histograms

26. Example alert

27. Alert Levels
Send Slack/Teams Message

28. Alert Levels
Send alert to oncall

29. Alerting tool is usually built
into the metrics system

30. Alerts should be
● urgent
● important
● actionable
● real

31. Should represent either
ongoing or imminent
problems

32. What to watch out for?

33. 1. Better to remove an alert
when it’s noisy

35. #2. Use success rate

36. Symptom-based monitoring
● Number of 5xx HTTP response codes
● Response time
● Email sending is not working
● Users can’t log in

37. Cause-based monitoring
● Free disk space on database server
● Memory utilisation
● Free file descriptors

38. Many causes may trigger a
symptom

39. User impact is most
important

40. #3. Focus on
symptom-based alerts

41. Cause-based alerts are
also necessary

42. Picking alerts to start with
Front-end
Load
Balancer
Back-end DB
Count rate of
successful
log-in
Count
request
success rate

44. Logging

45. Logging system
Application
Application
Application
Log
Aggregation
Database
Dashboard
Log
Collector
Log
Collector
Log
Collector
Logstash, Fluentd

46. Logging system
Application
Application
Application
Log
Aggregation
Database
Dashboard
Log
Collector
Log
Collector
Log
Collector
Elasticsearch, Loki

47. Logging system
Application
Application
Application
Log
Aggregation
Database
Dashboard
Log
Collector
Log
Collector
Log
Collector
Kibana

48. Log messages

49. Finding logs
Can search by:
● content of log message
message : *notification*
● all logs from a service
kubernetes.labels.app/name.keyword : "api-gateway"
● many more thanks to flexible query schema

50. What to watch out for?

51. #1. Use appropriate log
level - info, warn, error

52. Structured logging
● Append useful key=value pairs
● Can group (aggregate) by the keys
● Can sort by aggregations

53. #2. Use structured logging

54. Too many logs
Application
Application
Application
Log
Aggregation
Real Time Search
Engine
Log Scraper
Log Scraper
Log Scraper
Dashboard

55. Too many logs
Application
Application
Application
Log
Aggregation
Real Time Search
Engine
Log Scraper
Log Scraper
Log Scraper
Dashboard
Reduce log
retention period

56. Too many logs
Application
Application
Application
Log
Aggregation
Real Time Search
Engine
Log Scraper
Log Scraper
Log Scraper
Dashboard
Cold Storage
Query UI

57. #3. Use proper retention
period or cold storage

59. Distributed tracing
https://www.youtube.com/watch?v=rM1z7Q1TxR0

60. End-to-end summary
1. Configure automated alerts

61. End-to-end summary
1. Configure automated alerts
2. Use metrics and tracing to pinpoint the problem

62. End-to-end summary
1. Configure automated alerts
2. Use metrics and tracing to pinpoint the problem
3. Use structured logging to find the root cause of the problem easily

63. End-to-end summary
1. Configure automated alerts
2. Use metrics and tracing to pinpoint the problem
3. Use structured logging to find the root cause of the problem easily
4. Fix problems and make sure all metrics are always back to normal

64. Thank you! Q&A
Nikolay Stoitsev
Engineering Manager at Halo DX
Photo by Pixabay, Şahin Sezer Dinçer, Andrea Piacquadio, Ian Beckley from Pexels