This document discusses fullstack monitoring and outlines the key components needed: collecting metrics from servers and applications, alerting when issues arise, and visualizing metrics. Metrics can be collected using agents, scripts, logs, and from application code. They are sent to storage locations like Zabbix Server, Elasticsearch, and InfluxDB. Alerting involves notifying teams when things go down or abnormal behavior is detected using various channels. Visualization of metrics over time is best done with Grafana, which allows viewing metrics from multiple sources together on responsive dashboards.

1. Fullstack monitoring
How I monitor our apps and infra

2. _______________________________________
Julien Maîtrehenry
DevTools at PetalMD
Co-leader Hacking Health Québec
_______________________________________
Julien.maitrehenry@me.com
@jmaitrehenry
CONTACT

3. • We need to collect metrics (servers and apps)
• We need some alerting when something go down
• We need to visualize our metrics
What we need?
SUMMARY

4. • Basic metrics: CPU, Memory, Filesystem, system uptime, …
• Security metrics: /etc/passwd changed, firewall change, connections, …
• Software metrics: service is running?, custom service stats
Metrics
COLLECTING METRICS

5. • Zabbix Agent
• Custom scripts
• Logs
• Application code
How can we catch them all?
COLLECTING METRICS

6. • Direct connections
• Using a forwarder:
• Logstash
• Rsyslog
• StatsD
How to send them?
COLLECTING METRICS

7. • Zabbix Server
• Elasticsearch
• InfluxDB
Where we can store them?
COLLECTING METRICS

8. • Something go down => alert
• Something may have a problem => alert
• Something strange happened => alert
Alerting on what?
ALERTING

9. • By email
• By SMS
• By push notification
• By Slack / other instant messaging
• Do not use all channels for all alert!
• Use alert level (Info, Warning, Critical, …)
How?
ALERTING

10. Example – Zabbix alerts / triggers
ALERTING

11. Example – Zabbix triggers history
ALERTING

12. Example – Zabbix trigger actions - Slack
ALERTING

13. • Zabbix can expose metrics via graph, dashboard, etc
Metrics exposure
VIZUALISING

14. • Zabbix can expose metrics via single stats history, graph, dashboard, …
Metrics exposure
VIZUALISING
Single numeric metrics
Multi numeric metrics

15. • Zabbix can expose metrics via single stats history, graph, dashboard, …
Metrics exposure
VIZUALISING
Dashboard alias multi graph in one panel

16. • But it’s really not user friendly
• Really slow to build some multi metrics graph and dashboard
• Can’t connect to other storage for aggregate data
Zabbix can expose metrics…
VIZUALISING

17. • Kibana with only Elasticsearch as storage
• Graphite, but ugly too
• Grafana
But some tools exist only for this goal!
VIZUALISING

18. • Nice UI with dark and white theme
• Multisource (Elasticsearch, Zabbix, influxDB, Graphite and more)
• Inside a dashboard
• Inside a graph
• Dashboard / graph templating
• Responsive
• Multi auth backend: Password base, Github, Google Auth, …
• Really active project
Grafana
VIZUALISING

19. Grafana - Dashboard
VIZUALISING

20. Grafana - Dashboard
VIZUALISING
Zabbix metrics

21. Grafana - Dashboard
VIZUALISING
Zabbix metrics
Elasticsearch metrics

22. Grafana - Editor
VIZUALISING

23. Grafana - Templating
VIZUALISING

24. Grafana - Templating
VIZUALISING

25. Questions?