Monitor All the Things! provides tips for effectively monitoring distributed systems. It recommends centralizing logs for ease of access and insights. Metrics such as gauges, counters, and timers should be stored and graphs created for visualization. Services should publish health statuses to enable simple checks that raise alerts on failures. Traces can help identify the paths requests take through systems and pinpoint where slowdowns occur. The document stresses the importance of monitoring all facets of systems for visibility into performance and failures.

1. Monitor All the
Things!
Félix-Étienne Trépanier

2. Call me Félix
- Software Engineer - Independant
- Code Java / Scala (mainly)
- Build Distributed Systems
- Organizer of Scala-Montreal
Meetups

3. scope

4. motivations

5. Facets
- Logs
- Metrics
- Alerts
- Traces

6. logs

7. Typically event based and
unstructured messages
written to files.

8. Often the key for figuring
out what happened.

9. - timestamp
- severity
- thread
- source
- clear complete message
Tips

10. - single line *
- log configuration
- log state/event on error
- avoid log flooding
- local log rotation!
Tips

11. Think about what you
would like to know if it
happens.
Tips

12. In distributed systems,
accessing all the log files
can be painful.

13. A few options
syslog

14. Example
input {
file {
path => "/var/log/demo/demo.log"
}
}
filter {
grok {
match => [ "message", "%{DATESTAMP} %{LOGLEVEL:log_level} %{GREEDYDATA}" ]
}
mutate {
add_field => {
"service" => "demo"
}
}
}
output {
elasticsearch {
host => mysearchserver
}
}

15. Example
{
"message":"2015-02-01 21:31:02,076 INFO [dispatcher-29] awesome log",
"@version":"1",
"@timestamp":"2015-02-02T02:31:02.860Z",
"host":"mammouth",
"path":"/var/log/demo/demo.log",
"log_level":"INFO",
"service":"demo"
}

17. Persistence?

18. metrics

19. Measurement points
in the system.

20. - gauges
- counters
- histograms
- meters
- timers

21. Tips

22. Tips
- call per second (for each endpoint)
- call latency (for each endpoint)
- downstream dependencies latency
- limited resource size
- cache
- heap
- thread pool
- connection pool
- error count/rate
- unexpected conditions
- feature metrics

23. History

25. alerts

26. Notification sent
when the state of a
component is
critical.
(usually during the night)

27. Tips
- simple triggers/checks
- limit false positives
- no false negatives
- alert message should
give some context

28. Tips
For services:
- healthchecks
- test query
- measure around alerts
- log around alerts

29. Some options

30. traces

31. A record of the
paths a request
took in the system.

32. Why is this request
slow?

33. Trace Aggregation
- request sampling
- annotations logged locally
- published ‘trace’ server
- trace reconstruction
- trace indexation
- trace service UI

35. summary

36. Craft your logs.
Centralize them.

37. Store metrics.
Use the graphs and
create a dashboard.

38. Services should
publish their health
status.

39. Monitor metrics and
health status with
simple checks to
raise alerts on
failure.

40. Build your services
for production.

41. Contact and Links
Félix-Étienne Trépanier
Twitter - @felixtrepanier
Github - @coderunner
Links and References: https://github.
com/coderunner/monitoring-references